Data storage method and device, data query method and device, equipment and storage medium

Abstract

The invention provides a data storage method and device, a data query method and device, equipment and a storage medium, and the method comprises the steps: grouping all to-be-stored data in a data set, and obtaining a grouping identifier and an intra-group identifier corresponding to each to-be-stored data; for each piece of to-be-stored data, determining an identifier of the to-be-stored data based on the identifier of the data set, the grouping identifier and the intra-group identifier; determining meta information of the to-be-stored data and feature information of the to-be-stored data; storing by taking the identifier of the to-be-stored data as a key and the meta-information of the to-be-stored data as a value; taking the identifier of the to-be-stored data as a key, taking the original data of the to-be-stored data as a value, and storing based on the grouping identifier and the intra-group identifier corresponding to the to-be-stored data; and taking the identifier of the to-be-stored data as a key, taking the feature information of the to-be-stored data as a value, and storing based on the grouping identifier and the intra-group identifier corresponding to the to-be-stored data. According to the method, large-scale data can be efficiently stored, and subsequent quick data searching is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data storage method, a data query method, an apparatus, a device, and a storage medium. Background Art

[0002] With the advent of the big data era, data storage and management technologies are facing unprecedented challenges. In various application scenarios, especially in the management of large-scale graphic and text datasets, how to efficiently store, update, and query data has become an urgent problem to be solved.

[0003] In the management of large-scale graphic and text datasets, due to the huge amount of data and diverse types, how to efficiently store these large-scale data and facilitate subsequent rapid data retrieval is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The present invention provides a data storage method, a data query method, an apparatus, a device and a storage medium to address the defects in the prior art that large-scale data cannot be efficiently stored and are not convenient for subsequent rapid data retrieval, thereby achieving efficient storage of large-scale data and facilitating subsequent rapid data retrieval.

[0005] The present invention provides a data storage method, comprising: Group all the data to be stored in the data set to obtain the group identifier and the intra-group identifier corresponding to each data to be stored; For each of the data to be stored, determining an identifier of the data to be stored based on the identifier of the data set, the group identifier, and the intra-group identifier; Determining metadata of the data to be stored and feature information of the data to be stored; Using the identifier of the data to be stored as a key and the meta-information of the data to be stored as a value to store the meta-information; Using the identifier of the data to be stored as a key and the original data of the data to be stored as a value, the original data is stored based on the group identifier and the intra-group identifier corresponding to the data to be stored; The identifier of the data to be stored is used as a key, and the characteristic information of the data to be stored is used as a value. The characteristic information is stored based on the group identifier and the intra-group identifier corresponding to the data to be stored.

[0006] According to a data storage method provided by the present invention, the method further includes: Respectively obtaining a storage path of the meta information, a storage path of the original data, and a storage path of the feature information; Obtaining version information of the meta information, version information of the original data, and version information of the feature information; Adding the version information of the meta information to the storage path of the meta information; Adding the version information of the original data to the storage path of the original data; The version information of the feature information is added to the storage path of the feature information.

[0007] According to a data storage method provided by the present invention, the method further includes: Adding a version field to the meta information; The version information of the original data and the version information of the characteristic information are determined as the field value of the version field.

[0008] According to a data storage method provided by the present invention, the method further includes: Receive data update request; When the data update request includes updated content of the meta information of the target data to be stored, performing a full update on the meta information of all the data to be stored; In a case where the data update request includes update content of the original data of the target data to be stored, incrementally updating the original data of the target data to be stored based on the update content of the original data; In a case where the data update request includes the updated content of the characteristic information of the target data to be stored, the characteristic information of the target data to be stored is incrementally updated based on the updated content of the characteristic information.

[0009] According to a data storage method provided by the present invention, the method further includes: After all the metadata of the data to be stored is fully updated, the version information of the metadata of each data to be stored is updated; After incrementally updating the original data of the target data to be stored, updating the version information of the original data of the target data to be stored; After incrementally updating the characteristic information of the target data to be stored, updating the version information of the characteristic information of the target data to be stored; The updated version information of the meta information, the updated version information of the original data, and the updated version information of the feature information are stored in a version update file.

[0010] According to a data storage method provided by the present invention, the method further includes: receiving a data processing request, wherein the data processing request includes a data processing type; Based on the number of nodes in the distributed execution engine and the maximum processing capacity of each node, all the data to be stored are divided into tasks to obtain at least one task corresponding to each node; For each of the tasks, the node corresponding to the task processes the metadata, original data and feature information in the data to be stored corresponding to the task based on the data processing type to obtain updated content of the metadata, updated content of the original data and updated content of the feature information, wherein the data to be processed corresponding to the tasks processed simultaneously by different nodes are located in different groups.

[0011] The present invention also provides a data query method, comprising: Obtaining a data query request, wherein the data query request includes an identifier of the data to be queried, and the identifier of the data to be queried includes a target identifier of a data set, a target group identifier, and an identifier within a target group; Based on the target identifier of the data set, obtaining a storage path of target meta information of the data to be queried, a storage path of target original data of the data to be queried, and a storage path of target feature information of the data to be queried, wherein the target meta information, the target original data, and the target feature information are stored based on the data storage method described in any of the above embodiments; Querying the target meta information in the storage path of the target meta information; In the storage path of the target original data, based on the target group identifier and the target intra-group identifier, query the target original data, and in the storage path of the target feature information, based on the target group identifier and the target intra-group identifier, query the target feature information; The target meta information, the target original data and the target feature information are determined as the query result of the data to be queried.

[0012] According to a data query method provided by the present invention, querying the target meta information in the storage path of the target meta information includes: Obtaining version information of the target meta information from a version update file; The target meta information is queried based on the storage path of the target meta information and the version information of the target meta information.

[0013] According to a data query method provided by the present invention, the target original data is queried based on the target group identifier and the target intra-group identifier in the storage path of the target original data, and the target feature information is queried based on the target group identifier and the target intra-group identifier in the storage path of the target feature information, including: Parsing the target meta information to obtain version information of the target original data and version information of the target feature information in the version field of the target meta information; querying the target original data based on the storage path of the target original data, the version information of the target original data, the target group identifier, and the target intra-group identifier; The target feature information is queried based on the storage path of the target feature information, the version information of the target feature information, the target group identifier, and the target intra-group identifier.

[0014] The present invention also provides a data storage device, comprising the following modules: A grouping module is used to group all the data to be stored in the data set and obtain the group identifier and the intra-group identifier corresponding to each data to be stored; a determination module, configured to determine, for each of the data to be stored, an identifier of the data to be stored based on the identifier of the data set, the group identifier, and the intra-group identifier; The determining module is further configured to determine the metadata of the data to be stored and the feature information of the data to be stored; a storage module, configured to store the metadata of the data to be stored using the identifier of the data to be stored as a key and the metadata of the data to be stored as a value; The storage module is further configured to use the identifier of the data to be stored as a key and the original data of the data to be stored as a value, and store the original data based on the group identifier and the intra-group identifier corresponding to the data to be stored; The storage module is further configured to use the identifier of the data to be stored as a key and the characteristic information of the data to be stored as a value, and store the characteristic information based on the group identifier and the intra-group identifier corresponding to the data to be stored.

[0015] The present invention also provides a data query device, comprising: An acquisition module, configured to acquire a data query request, wherein the data query request includes an identifier of the data to be queried, and the identifier of the data to be queried includes a target identifier of a data set, a target group identifier, and an identifier within a target group; The acquisition module is configured to acquire, based on the target identifier of the data set, a storage path of target meta information of the data to be queried, a storage path of target original data of the data to be queried, and a storage path of target feature information of the data to be queried, wherein the target meta information, the target original data, and the target feature information are stored based on the data storage method described in any one of the above embodiments; A query module, configured to query the target meta information under the storage path of the target meta information; The query module is configured to query the target original data based on the target group identifier and the target intra-group identifier in the storage path of the target original data, and query the target feature information based on the target group identifier and the target intra-group identifier in the storage path of the target feature information; The determination module is used to determine the target meta information, the target original data and the target feature information as the query result of the data to be queried.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the computer program, the processor implements any of the data storage methods described above, or implements any of the data query methods described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements any of the data storage methods described above, or implements any of the data query methods described above.

[0018] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned data storage methods, or implements any of the above-mentioned data query methods.

[0019] The data storage method, data query method, device, equipment, and storage medium provided by the present invention group all data to be stored in a data set to obtain a group identifier and an intra-group identifier corresponding to each data to be stored. For each data to be stored, based on the identifier of the data set, the group identifier, and the intra-group identifier, the identifier of the data to be stored is determined. After determining the metadata of the data to be stored and the characteristic information of the data to be stored, the metadata is stored using the identifier of the data to be stored as a key and the metadata of the data to be stored as a value. The original data is stored using the identifier of the data to be stored as a key and the original data of the data to be stored as a value based on the group identifier and the intra-group identifier corresponding to the data to be stored. The characteristic information is stored using the identifier of the data to be stored as a key and the characteristic information of the data to be stored as a value based on the group identifier and the intra-group identifier corresponding to the data to be stored. Due to the group storage and the unique identifier of the data to be stored, a large amount of data to be stored can be effectively organized and managed, improving the access efficiency of the data to be stored. In addition, because the metadata, original data, and characteristic information of the data to be stored are stored separately, the required data parts can be flexibly accessed according to different application scenarios, avoiding unnecessary data loading, improving system performance, and improving the query and reading efficiency of subsequent data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of the processing flow of graphic and text data provided by an embodiment of the present invention.

[0022] Figure 2 A flowchart of a data storage method provided by an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of the architecture of the data update process system provided in an embodiment of this aspect.

[0024] Figure 4 A schematic diagram of the execution flow within the distributed execution engine provided by an embodiment of the present invention.

[0025] Figure 5 A schematic diagram of a separate storage architecture for data to be stored provided by an embodiment of the present invention.

[0026] Figure 6 A flowchart of a data query method provided by an embodiment of the present invention.

[0027] Figure 7 A schematic structural diagram of a data storage device provided in an embodiment of the present invention.

[0028] Figure 8 A schematic diagram of the structure of a data query device provided by an embodiment of the present invention.

[0029] Figure 9 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0031] With the rapid development of artificial intelligence (AI), large multimodal models have become a research hotspot in academia and industry, owing to their exceptional performance in areas such as image and text understanding, image generation, and cross-modal dialogue. However, the performance of such models is highly dependent on massive amounts of high-quality training data. The key bottleneck in the development of large multimodal models is the ability to construct and efficiently manage large-scale image and text datasets. Current mainstream solutions rely primarily on publicly available internet data and open-source datasets, but these solutions suffer from a widespread lack of data management systems, resulting in inefficient data storage. Furthermore, existing data storage methods can slow subsequent data queries.

[0032] To address the above-mentioned issues, embodiments of the present invention provide a data storage method. Embodiments of the present invention are applicable to scenarios involving the storage of large-scale data, and in particular, to scenarios involving the storage of large-scale graphic and text data. The method may be implemented by an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed data storage device, or by a data storage device within the electronic device. The data storage device may be implemented using software, hardware, or a combination of both.

[0033] Figure 1 A schematic diagram of the processing flow of graphic data provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown in the figure, the training dataset for multimodal large models can be processed in the following ways: first, the user collects the original data through data crawling, downloading, etc.; second, the original data is cleaned and normalized to generate an initial version of the dataset that meets the storage specifications; then, through multiple rounds of iterative updates, the dataset is optimized and improved, and finally a high-quality dataset version is formed; finally, a training set is constructed through sampling for model training.

[0034] In the embodiment of the present invention, the image and text dataset management system will be described in detail from three dimensions: dataset storage, dataset iteration, and dataset use.

[0035] Figure 2 A flow chart of a data storage method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes: Step 201: Group all the data to be stored in the data set to obtain the group identifier and the intra-group identifier corresponding to each data to be stored.

[0036] In this step, all the data to be stored in the data set can be grouped, wherein the group identifier is used to identify the group to which each data to be stored belongs, and the intra-group identifier is used to identify the position of each data to be stored within the group to which it belongs.

[0037] For example, assuming the data to be stored is image data, assuming there is a dataset containing 1,000 images, these images can be grouped into groups of 100, resulting in 10 groups. Each group is assigned a unique group identifier, such as group 1, group 2, ..., group 10. At the same time, the image data within each group is also assigned an intra-group identifier, such as 1, 2, ..., 100, which identifies the position of the image data within the group.

[0038] Step 202: For each data to be stored, determine the identifier of the data to be stored based on the identifier of the data set, the group identifier and the intra-group identifier.

[0039] In this step, a unique identifier is determined for each piece of data to be stored based on the dataset identifier, group identifier, and intra-group identifier. For example, if the data to be stored is image data, the image_id identifier is a unique identifier for each image in the dataset. It uses a three-part structure, with each part separated by a #. The first part is the dataset identifier, such as the dataset name. Legal values ​​are any string not containing a #. Dataset names must be globally unique. The second part is the image's part number within the dataset. This format is a fixed-length string of 6, with zeros padded to the left. The third part is the image's internal part number. This format is a fixed-length string of 8, with zeros padded to the left. For example, if the image_id of a piece of data to be stored is Test#000001#00000001, then Test is the dataset identifier, and 000001 indicates that the data to be stored is within group 1 and is located at position 00000001 within group 1.

[0040] The identification of the data to be stored is determined by the dataset identification, group identification and intra-group identification. This identification method can ensure that each piece of data to be stored is unique in the entire system, facilitating subsequent storage and query operations.

[0041] Step 203: Determine the metadata of the data to be stored and the feature information of the data to be stored.

[0042] In this step, the metadata image_meta typically contains basic attributes of the data to be stored, such as label information, data source, data creation time, data size or dimensions, and data type. Label information can include, for example, text descriptions (captions), optical character recognition (OCR) results, model confidence scores, and entity detection results. Feature information image_feature extracts and describes the data content of the data to be stored, such as image_encode features extracted by the Vision Transformer (ViT), such as keywords in text data, or feature representations or feature vectors extracted from image data.

[0043] Step 204: The metadata is stored using the identifier of the data to be stored as a key and the metadata of the data to be stored as a value.

[0044] In this step, to balance scalability and access efficiency, the Lightning Memory-Mapped Database (LMDB) can be used as the storage engine. Its core advantages include: (1) key-value storage based on memory mapping, eliminating file system addressing overhead; (2) single-file storage structure, significantly reducing the IO time of massive small files; (3) structured storage, which can convert different types of raw data into a unified key-value storage; (4) support for concurrent reading, with a throughput of up to millions of QPS.

[0045] In the LMDB file for the metadata image_meta of the data to be stored, the key is the image_id identifier of the data to be stored, and the value is the metadata of the data to be stored. This metadata can be JSON-serialized data. Using JSON as the value format supports dynamic field expansion, facilitating the subsequent expansion of metadata types such as label information and image dimensions. Using LMDB for storage in the outer layer maintains a certain level of read and write efficiency. Furthermore, since metadata primarily consists of string text or floating-point scores, the space occupied is manageable. Therefore, each dataset only requires one global LMDB to store the metadata image_meta, avoiding storage fragmentation.

[0046] Step 205: Using the identifier of the data to be stored as a key and the original data of the data to be stored as a value, the original data is stored based on the group identifier and the intra-group identifier corresponding to the data to be stored.

[0047] In this step, the original data of the data to be stored can be understood as the original binary data of the data to be stored. Taking the data to be stored as image data as an example, in order to meet the efficient storage requirements of ultra-large-scale image data (100,000-100 million level), in this embodiment, a group storage strategy can be adopted to group all the images to be stored in the data set, and each group stores a fixed number of images to be stored to control the size of the LMDB in a single group. In the specific implementation process, the size of the LMDB storing 20,000 images is approximately 20GB. In the LMDB file of the original data image_data of the data to be stored, the key key is the identifier image_id of the data to be stored, and the value value is the compressed image binary data. When performing data query, the group in which the corresponding data is located and the position within the group can be directly obtained through the identifier image_id of the data to be stored, thereby facilitating data reading.

[0048] Compared with single LMDB storage, the above method of storing the original data of the data to be stored in groups has the characteristics of low random read latency, excellent parallel read and write capabilities, and strong scalability.

[0049] Step 206: Using the identifier of the data to be stored as a key and the characteristic information of the data to be stored as a value, the characteristic information is stored based on the group identifier and the intra-group identifier corresponding to the data to be stored.

[0050] In this step, in order to balance storage efficiency and query performance, a group storage strategy consistent with the original data image_data is adopted to store the characteristic information of the data to be stored. That is, the key key is also the identifier image_id of the data to be stored, and the value value is the characteristic information of the data to be stored.

[0051] In practical applications, because there may be multiple image features, feature information, similar to metadata, uses JSON format. The key in JSON is the feature name, and the value is the byte-formatted data obtained by decoding the feature vector through NumPy conversion.

[0052] When storing in the form of key-value pairs, the feature information may be stored in a corresponding position of the corresponding group based on the group identifier and the intra-group identifier in the identifier image_id of the data to be stored.

[0053] The data storage method provided by an embodiment of the present invention groups all data to be stored in a data set to obtain a group identifier and an intra-group identifier corresponding to each data to be stored. For each data to be stored, the identifier of the data to be stored is determined based on the identifier of the data set, the group identifier, and the intra-group identifier. After determining the metadata of the data to be stored and the characteristic information of the data to be stored, the metadata is stored using the identifier of the data to be stored as a key and the metadata of the data to be stored as a value. The original data is stored using the identifier of the data to be stored as a key and the original data of the data to be stored as a value based on the group identifier and the intra-group identifier corresponding to the data to be stored. The characteristic information is stored using the identifier of the data to be stored as a key and the characteristic information of the data to be stored as a value based on the group identifier and the intra-group identifier corresponding to the data to be stored. Due to the group storage and the unique identifier of the data to be stored, a large amount of data to be stored can be effectively organized and managed, improving the access efficiency of the data to be stored. In addition, because the metadata, original data, and characteristic information of the data to be stored are stored separately, the required data parts can be flexibly accessed according to different application scenarios, avoiding unnecessary data loading, improving system performance, and improving the efficiency of subsequent data query and reading.

[0054] Furthermore, based on the above embodiment, in order to support version management of the data to be stored, it is necessary to set storage paths and version information for the metadata, raw data, and feature information, respectively. For example, the metadata storage path, raw data storage path, and feature information storage path can be obtained, respectively, along with the version information of the metadata, raw data, and feature information. The metadata version information is added to the metadata storage path, the raw data version information is added to the raw data storage path, and the feature information version information is added to the feature information storage path.

[0055] Specifically, during the use of a dataset, the data in the dataset is often updated. Assuming that the data to be stored is image data, common update operations mainly include the following three categories: (1) original image processing, including cropping, watermark removal, coding and other modification operations; (2) supplementation and updating of annotated data; (3) quality screening of the dataset, such as eliminating low-quality image samples. For the same data to be stored, there will be different versions. Therefore, in order to ensure the consistency of the data after the data to be stored is updated and to achieve systematic version control management, a version identification layer can be introduced in the storage path of the metadata image_meta, the original data image_data and the feature information image_feature, and the physical isolation of different versions of data can be achieved through an independent directory structure.

[0056] For example, assume the metadata storage path is " / data / meta / ", the raw data storage path is " / data / raw / ", and the feature information storage path is " / data / feature / ". The metadata version is "v2.0", the raw data version is "v1.0", and the feature information version is "v1.0". Add the version information to the storage path to form a complete storage path. For example, the metadata storage path is " / data / meta / v2.0 / ", the raw data storage path is " / data / raw / v1.0 / ", and the feature information storage path is " / data / feature / v1.0 / ".

[0057] In addition, a version description document can be added for each data set to synchronously record version change information each time the data is updated to ensure the traceability of version changes.

[0058] In this embodiment, version information can be added to the storage path so that the storage path contains version information, achieving the purpose of simultaneously maintaining multiple versions of metadata, original data and feature information, facilitating data backtracking and historical query, and providing a basis for data updating and iteration.

[0059] Exemplarily, a version field may be added to the meta information, and the version information of the original data and the version information of the feature information may be determined as the field value of the version field.

[0060] Specifically, the metadata image_meta is updated relatively frequently, while the original data image_data and feature information image_feature are usually modified less frequently and kept synchronized. Therefore, based on the different iterative characteristics of the above three types of data, a version field image_version can be added to the metadata image_meta. This version field serves as a version association identifier, which can represent the versions of the original data image_data and feature information image_feature corresponding to the current image_meta version, thereby ensuring data consistency.

[0061] In this embodiment, by adding a version field to the metadata and querying the value of the version field, the version information of the original data and feature information associated with the metadata can be directly obtained without the need for additional query operations, thereby improving the query efficiency of the system.

[0062] Exemplarily, on the basis of the above-mentioned embodiments, considering that the original data image_data and feature information image_feature of the data to be stored occupy a large storage space and have a low update frequency, in an embodiment of the present invention, a differentiated storage update strategy can be set to reduce redundant data and optimize storage efficiency. Specifically, after receiving a data update request, if the data update request includes the updated content of the metadata of the target data to be stored, the metadata of all the data to be stored is fully updated. If the data update request includes the updated content of the original data of the target data to be stored, the original data of the target data to be stored is incrementally updated based on the updated content of the original data. If the data update request includes the updated content of the feature information of the target data to be stored, the feature information of the target data to be stored is incrementally updated based on the updated content of the feature information.

[0063] Specifically, assuming that the data to be stored is image data, since during the data iteration process, usually only part of the image will be modified, therefore, when the data update request includes the updated content of the original data of the target data to be stored, and / or, includes the updated content of the feature information of the target data to be stored, an incremental update method can be used to only re-store the original data and feature information of the image that has actually changed, and the unchanged image will still be retained in the old version directory. This can effectively avoid repeated storage of unchanged data and significantly reduce storage overhead.

[0064] Furthermore, because image_meta metadata updates frequently and the storage overhead of a single data entry is low, a full update approach can be used. Complete metadata is stored with each version iteration to ensure data version consistency. This approach simplifies the management of image_meta metadata, avoids cross-version dependencies, and improves query efficiency.

[0065] In this embodiment, the meta information is fully updated, and the original data and feature information are incrementally updated, which not only ensures data consistency but also improves system performance and efficiency.

[0066] Exemplarily, based on the above embodiments, after the metadata of all data to be stored is fully updated, the version information of the metadata of each data to be stored is updated; after the original data of the target data to be stored is incrementally updated, the version information of the original data of the target data to be stored is updated; after the feature information of the target data to be stored is incrementally updated, the version information of the feature information of the target data to be stored is updated; the version information of the updated metadata, the version information of the updated original data and the version information of the updated feature information are stored in the version update file.

[0067] After completing the data update, the corresponding version information needs to be updated to reflect the latest status of the data. Specifically, after completing the full update of the metadata, the version information of the metadata of all the data to be stored is updated, such as from "v1.0" to "v1.1". After completing the incremental update of the original data of the target data to be stored, the version information of the original data of the target data to be stored is updated, such as from "v1.0" to "v1.1". After completing the incremental update of the feature information of the target data to be stored, the version information of the feature information of the target data to be stored is updated, such as from "v1.0" to "v1.1".

[0068] Furthermore, the updated version information needs to be updated not only to the storage path, but also to a dedicated version update file so that the system can quickly obtain the latest version information of various types of data. The version update file can use JSON or XML format to record the latest version information of each data type, such as: { "meta_version": "v1.1", "raw_version": "v1.1", "feature_version": "v1.1" } In this embodiment, by updating the version information, data under each version can be easily and clearly traced. In addition, the updated version information of the meta information, the updated version information of the original data, and the updated version information of the feature information are stored in the version update file. By maintaining this version update file, the latest version information of each type of data can be quickly obtained without traversing all data, thereby improving the system's response speed.

[0069] Furthermore, a data processing request can also be received, which includes a data processing type. Based on the number of nodes in the distributed execution engine and the maximum processing capacity of each node, all data to be stored are divided into tasks to obtain at least one task corresponding to each node. For each task, the node corresponding to the task processes the metadata, original data and feature information in the data to be stored corresponding to the task based on the data processing type to obtain updated content of the metadata, updated content of the original data and updated content of the feature information, wherein the data to be processed corresponding to the tasks processed simultaneously by different nodes are located in different groups.

[0070] Specifically, as data scale continues to grow, the cost of fully updating datasets increases significantly. In particular, when using multimodal models for data augmentation, the computational process relying on artificial intelligence (AI) accelerators like graphics processing units (GPUs) further increases resource consumption. To improve data update efficiency, embodiments of the present invention provide an efficient and stable distributed data processing system. Figure 3 This is a schematic diagram of the architecture of the data update process system provided by the embodiment of this aspect, such as Figure 3 As shown, the registrar provides basic built-in data processing modules, such as aesthetic score assessment or facial recognition, and uses a standardized interface design to unify input and output formats. Users can use the registrar to register and manage modular modules and add custom data processing modules. The process building framework uses YAML configuration files to build data processing processes from built-in and user-defined data processing modules and preliminarily verifies the correctness, resource utilization, and efficiency of the data processing processes. The distributed execution engine inputs the original dataset and the constructed data processing process, automatically dispatches resources from multiple machines and graphics cards to perform distributed computing, and outputs the updated dataset.

[0071] Figure 4 The following is a schematic diagram of the execution flow inside the distributed execution engine provided in the embodiment of the present invention, as shown in FIG. Figure 4 As shown, a new data processing module is injected through the registrar. This data processing module is a module that can handle the data processing type included in the data processing request. First, based on the number of available nodes in the distributed execution engine and the maximum processing capacity of each node, the image_id set of all data to be stored is divided into tasks, resulting in at least one task corresponding to each node. For example, assuming there are two processing nodes, each node can process up to 200 data items, and there are a total of 800 data items to be processed, these data can be divided into four tasks, each including 200 data items. Node 1 processes tasks 1 and 2, and node 2 processes tasks 3 and 4.

[0072] During task allocation, to facilitate subsequent parallel strategies, avoid data conflicts, and improve processing efficiency, the image_ids of the data to be stored, processed by different nodes in the same loop, are ensured to be grouped separately. This means that the data corresponding to tasks being processed simultaneously by different nodes is grouped separately. Because the data in different groups is independent, different nodes can process data in parallel without worrying about data dependencies or conflicts, thus maximizing system resource utilization and improving processing efficiency.

[0073] During the data processing phase, each node loads the metadata image_meta, original data image_data, and feature information image_feature corresponding to the image_id of the data to be stored in parallel, and completely executes the data processing flow corresponding to the data processing type within the node to process the metadata image_meta, original data image_data, and feature information image_feature to obtain their corresponding updated content.

[0074] When the number of tasks is greater than the number of available nodes, multiple rounds of data processing will be performed in a cyclic manner, and global synchronization and writing of updated data will be performed at the end of each round of processing. For example, Figure 4 As shown, node 1 corresponds to both Task 1 and Task 2. Node 1 first processes all data to be stored in Task 1, writes updated data after processing is complete, and then processes all data to be stored in Task 2. Furthermore, when processing data to be stored in Task 1 or Task 2, the data in each group can be processed sequentially.

[0075] During the data update phase, the original data image_data and feature information image_feature are stored in groups, and the task allocation ensures that the data processed by each node is in different groups. Therefore, the updated original data image_data and feature information image_feature will be directly written to the corresponding storage by each node. The metadata image_meta will be merged into the master node during each round of data processing synchronization. The master node is responsible for updating the metadata image_meta to avoid conflicts caused by simultaneous writing by multiple nodes. Figure 4 In the example, assuming that the master node is node 1, node 1 updates the metadata image_meta during each round of data processing synchronization.

[0076] In this embodiment, the distributed execution engine is used to process the data to be stored, thereby improving the data processing efficiency in large-scale data processing scenarios and meeting the performance requirements in actual applications.

[0077] The following is an overall description of the data storage method provided in the embodiment of the present invention. Figure 5 A schematic diagram of a separate storage architecture for data to be stored provided in an embodiment of the present invention, such as Figure 5As shown, for each piece of data to be stored in a dataset, the identifier (image_id) for the data to be stored includes the dataset identifier, group identifier, and intra-group identifier. Metadata (image_meta) does not need to be stored in groups; each dataset only requires a global LMDB to store image_meta. Version information can be added to the storage path of the metadata (image_meta). This means that versions 1, 2, 3, and 4 are physically isolated and stored in different locations, enabling rapid version tracing.

[0078] The original data image_data and feature information image_feature can be stored in groups. Similarly, version information can be added to the storage path to store different versions of data in different locations.

[0079] In addition, a description document can be added for each dataset. This document includes key information such as the data source, scale, and annotation method, helping users quickly understand the characteristics of the dataset. Furthermore, a statistical data document is included, which contains version information and various other statistical content, such as label information distribution statistics. Users can visually analyze the label distribution of different datasets and dynamically adjust filtering conditions (such as clarity threshold or resolution threshold settings) to optimize the data selection process.

[0080] The data storage method provided by the present invention stores the metadata, original data and feature information of the data to be stored separately, and can flexibly access the required data parts according to different application scenarios, avoid unnecessary data loading, improve system performance, and improve the efficiency of subsequent data query and reading.

[0081] The following describes in detail the data query method corresponding to the data storage method described in the above embodiments. Figure 6 A flow chart of a data query method provided by an embodiment of the present invention is shown as follows: Figure 6 As shown, the method includes: Step 601: Obtain a data query request, the data query request includes an identifier of the data to be queried, and the identifier of the data to be queried includes a target identifier, a target group identifier, and an identifier within a target group of a data set.

[0082] In this step, some data may be queried or filtered out from the dataset based on actual needs. For example, if the user needs to filter out images with a resolution greater than a preset resolution, or filter out images whose label information meets preset requirements, etc., in order to build a final dataset that meets model training or other application requirements.

[0083] In actual applications, users can query the required data based on the explanatory documents or visualized statistical data corresponding to each data set. When the user determines the data to be filtered based on the explanatory documents or statistical data, a data query request is triggered. The data query request includes the identifier of the data to be queried, which includes the target identifier of the data set, the target group identifier, and the identifier within the target group. For example, the identifier of the data to be queried included in the data query request is "DS001#3#45", where DS001 represents the target identifier of the data set, that is, the data set to which the queried data belongs, 3 represents the group to which the queried data belongs, and 45 represents the position of the queried data within the group.

[0084] Step 602: Based on the target identifier of the data set, obtain the storage path of the target meta information of the data to be queried, the storage path of the target original data of the data to be queried, and the storage path of the target feature information of the data to be queried.

[0085] The target meta-information, target original data and target feature information are stored based on the data storage method described in any of the above embodiments.

[0086] Specifically, for each data set, a global configuration file corresponding to the data set will be added. The global configuration file can centrally manage the relevant information of the data set, including the identification of the data set, the storage path of the metadata of all data in the data set, the storage path of the original data, and the storage path of the feature information.

[0087] Based on the target identifier of the data set, the storage path of the target meta information of the data to be queried, the storage path of the target original data of the data to be queried, and the storage path of the target feature information of the data to be queried can be obtained from the global configuration file.

[0088] Step 603: Query the target meta-information in the storage path of the target meta-information.

[0089] In this step, the target meta information can be searched under the storage path of the target meta information. Since the meta information is stored globally, the corresponding meta information can be directly searched only according to the identifier of the data to be searched.

[0090] Step 604: In the storage path of the target original data, based on the target group identifier and the target intra-group identifier, query the target original data, and in the storage path of the target feature information, based on the target group identifier and the target intra-group identifier, query the target feature information.

[0091] In this step, the target raw data storage path can be accurately located based on the target group identifier and target intra-group identifier to which the target raw data belongs and its storage location within the group. Similarly, the target feature information storage path can be accurately located based on the target group identifier and target intra-group identifier to which the target feature information belongs and its storage location within the group. Since the raw data and feature information are stored according to the group and intra-group identifier, this information is needed to locate the specific data location.

[0092] Step 605: The target meta information, target original data and target feature information are determined as the query result of the data to be queried.

[0093] In this step, the queried target meta information, target original data and target feature information are combined together and returned to the user as the query result of the data to be queried.

[0094] The data query method provided by an embodiment of the present invention obtains a data query request, which includes an identifier of the data to be queried. The identifier of the data to be queried includes a target identifier, a target group identifier, and a target intra-group identifier of a data set. Based on the target identifier of the data set, the method obtains the storage path of the target metadata of the data to be queried, the storage path of the target raw data of the data to be queried, and the storage path of the target feature information of the data to be queried. The method then queries the target metadata in the storage path of the target metadata. Based on the target group identifier and the target intra-group identifier, the method queries the target raw data in the storage path of the target raw data. Based on the target group identifier and the target intra-group identifier, the method also queries the target feature information in the storage path of the target feature information. The method then determines the target metadata, target raw data, and target feature information as the query result for the data to be queried. Because the metadata, raw data, and feature information can be stored separately, efficient querying can be achieved during querying. Furthermore, the method accurately and quickly locates the target metadata, target raw data, and target feature information of the data to be queried using only the identifier of the data to be queried, enabling random querying of single data and on-demand loading. Furthermore, when querying single data, I / O overhead is low, saving system resources.

[0095] Exemplarily, based on the above embodiment, when querying the target meta information under the storage path of the target meta information, it can be done in the following way: obtain the version information of the target meta information from the version update file, and query the target meta information based on the storage path of the target meta information and the version information of the target meta information.

[0096] Specifically, the version update file stores the version information of the metadata for each data item in the dataset. After determining the dataset identifier, the version information of the target metadata for the data item to be queried is retrieved from the corresponding version update file based on the dataset identifier. For example, the latest version of the metadata retrieved from the version update file is "v1.1."

[0097] Furthermore, a complete query path, such as " / data / meta / v1.1", can be constructed based on the storage path of the target meta information and the obtained version information, and the target meta information can be queried under the query path.

[0098] In this embodiment, by obtaining the version information of the target meta information from the version update file, it is possible to ensure that the version information of the target meta information is the latest version of the data. In addition, historical versions can also be obtained from the version update file, thereby improving the flexibility of data version management.

[0099] Exemplarily, based on the above embodiments, in the storage path of the target original data, based on the target group identifier and the target intra-group identifier, the target original data is queried, and in the storage path of the target feature information, based on the target group identifier and the target intra-group identifier, the target feature information is queried. This can be done in the following manner: Parse the target meta-information to obtain the version information of the target original data and the version information of the target feature information in the version field of the target meta-information; query the target original data based on the storage path of the target original data, the version information of the target original data, the target group identifier and the target intra-group identifier; query the target feature information based on the storage path of the target feature information, the version information of the target feature information, the target group identifier and the target intra-group identifier.

[0100] Specifically, since the target meta-information includes a version field, after obtaining the target meta-information, the target meta-information is parsed to obtain the value of the version field, which is the version information of the target raw data and the target feature information. For example, the target raw data version is "v1.1" and the target feature information version is "v1.0" parsed from the meta-information.

[0101] Furthermore, since different versions of data are stored in different locations, the storage path corresponding to the version information of the target original data can be constructed based on the storage path of the target original data and the version information of the target original data. For example, the address of the target original data obtained by query is / data / raw / , and the version information of the target original data obtained by parsing is v1.1, then the storage path corresponding to the constructed version information is / data / raw / v1.1. In addition, since the v1.1 version of the original data of all data in the same data set is in this storage path, it is necessary to further determine the group to which the target original data belongs in this storage path based on the target group identifier, and determine the storage location of the target original data in the group to which it belongs based on the target group identifier, so that the target original data can be quickly located.

[0102] Similarly, the storage path corresponding to the version information of the target feature information can be constructed based on the storage path of the target feature information and the version information of the target feature information. For example, the address of the target feature information obtained by query is / data / feature / , and the version information of the target feature information obtained by parsing is v1.1. Then the storage path corresponding to the constructed version information is / data / feature / v1.1. In addition, since the feature information of version v1.1 of all data in the same data set is in this storage path, it is necessary to further determine the group to which the target feature information belongs in this storage path based on the target group identifier, and determine the storage location of the target feature information in the group to which it belongs based on the target group identifier, so that the target feature information can be quickly located.

[0103] In this embodiment, the method of querying the target original data and the target feature information based on the version information of the target original data and the version information of the target feature information in the version field in the metadata can ensure that the versions of the queried target original data and the target feature information are consistent with the versions recorded in the metadata, thereby ensuring the consistency and integrity of the data.

[0104] The data storage device provided by the present invention is described below. The data storage device described below and the data storage method described above can be referenced to each other.

[0105] Figure 7 A structural diagram of a data storage device provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, the data storage device 700 includes: A grouping module 11 is used to group all the data to be stored in the data set and obtain a group identifier and an intra-group identifier corresponding to each data to be stored; a determination module 12, configured to determine, for each of the data to be stored, an identifier of the data to be stored based on the identifier of the data set, the group identifier, and the intra-group identifier; The determining module 12 is further configured to determine the metadata of the data to be stored and the feature information of the data to be stored; The storage module 13 is configured to store the metadata of the data to be stored by using the identifier of the data to be stored as a key and the metadata of the data to be stored as a value; The storage module 13 is further configured to use the identifier of the data to be stored as a key and the original data of the data to be stored as a value, and store the original data based on the group identifier and the intra-group identifier corresponding to the data to be stored; The storage module 13 is further configured to use the identifier of the data to be stored as a key and the characteristic information of the data to be stored as a value, and store the characteristic information based on the group identifier and the intra-group identifier corresponding to the data to be stored.

[0106] In an exemplary embodiment, the apparatus further includes: an acquisition module and an addition module, wherein: an acquisition module, configured to respectively acquire the storage path of the meta information, the storage path of the original data, and the storage path of the feature information; The acquisition module is further used to obtain the version information of the meta information, the version information of the original data and the version information of the feature information; An adding module, configured to add the version information of the meta information to the storage path of the meta information; The adding module is further used to add the version information of the original data to the storage path of the original data; The adding module is further configured to add the version information of the feature information to the storage path of the feature information.

[0107] In an exemplary embodiment, the adding module is further configured to add a version field to the meta information; The determining module 12 is configured to determine the version information of the original data and the version information of the feature information as the field value of the version field.

[0108] In an exemplary embodiment, the apparatus further includes: a receiving module and an updating module, wherein: A receiving module, configured to receive a data update request; An updating module, configured to update all metadata of all target data to be stored when the data update request includes updated metadata of the target data to be stored; The updating module is further configured to, when the data update request includes updated content of the original data of the target data to be stored, incrementally update the original data of the target data to be stored based on the updated content of the original data; The updating module is further configured to incrementally update the characteristic information of the target data to be stored based on the updated content of the characteristic information when the data update request includes the updated content of the characteristic information of the target data to be stored.

[0109] In an exemplary embodiment, the updating module is further configured to update the version information of each piece of metadata of the data to be stored after the metadata of all the data to be stored is fully updated; The updating module is further configured to update the version information of the original data of the target data to be stored after incrementally updating the original data of the target data to be stored; The updating module is further configured to update the version information of the characteristic information of the target data to be stored after incrementally updating the characteristic information of the target data to be stored; The storage module 13 is further configured to store the updated version information of the meta information, the updated version information of the original data, and the updated version information of the feature information into a version update file.

[0110] In an exemplary embodiment, the apparatus further comprises: a partitioning module and a processing module, wherein: The receiving module is further configured to receive a data processing request, wherein the data processing request includes a data processing type; A partitioning module, configured to partition all the data to be stored into tasks based on the number of nodes in the distributed execution engine and the maximum processing capacity of each node, to obtain at least one task corresponding to each node; A processing module is used to process the metadata, original data and feature information in the data to be stored corresponding to each of the tasks through the node corresponding to the task based on the data processing type, to obtain updated content of the metadata, updated content of the original data and updated content of the feature information, wherein the data to be processed corresponding to the tasks processed simultaneously by different nodes are located in different groups.

[0111] The device of this embodiment can be used to execute the method of any embodiment in the data storage method embodiment. Its specific implementation process and technical effects are similar to those in the data storage method embodiment. For details, please refer to the detailed description in the data storage method embodiment, which will not be repeated here.

[0112] Figure 8 A schematic diagram of the structure of a data query device provided by an embodiment of the present invention, such as Figure 8 As shown, the data query device 800 includes: An acquisition module 21 is configured to acquire a data query request, wherein the data query request includes an identifier of the data to be queried, and the identifier of the data to be queried includes a target identifier of a data set, a target group identifier, and an identifier within a target group; The acquisition module 21 is configured to acquire, based on the target identifier of the data set, a storage path of the target metadata of the data to be queried, a storage path of the target original data of the data to be queried, and a storage path of the target feature information of the data to be queried, wherein the target metadata, the target original data, and the target feature information are stored based on the data storage method described in any of the above embodiments; A query module 22, configured to query the target meta-information in the storage path of the target meta-information; The query module 22 is configured to query the target original data based on the target group identifier and the target intra-group identifier in the storage path of the target original data, and query the target feature information based on the target group identifier and the target intra-group identifier in the storage path of the target feature information; The determination module 23 is configured to determine the target meta information, the target original data, and the target feature information as the query result of the data to be queried.

[0113] In an exemplary embodiment, the query module 22 is specifically configured to: Obtaining version information of the target meta information from a version update file; The target meta information is queried based on the storage path of the target meta information and the version information of the target meta information.

[0114] In an exemplary embodiment, the query module 22 is specifically configured to: Parsing the target meta information to obtain version information of the target original data and version information of the target feature information in the version field of the target meta information; querying the target original data based on the storage path of the target original data, the version information of the target original data, the target group identifier, and the target intra-group identifier; The target feature information is queried based on the storage path of the target feature information, the version information of the target feature information, the target group identifier, and the target intra-group identifier.

[0115] The device of this embodiment can be used to execute the method of any embodiment in the data query method embodiment. Its specific implementation process and technical effects are similar to those in the data query method embodiment. For details, please refer to the detailed description in the data query method embodiment, which will not be repeated here.

[0116] Figure 9 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 9 As shown, the electronic device may include: a processor 910 , a communication interface 920 , a memory 930 and a communication bus 940 , wherein the processor 910 , the communication interface 920 and the memory 930 communicate with each other via the communication bus 940 . The processor 910 can call the logical instructions in the memory 930 to execute the data storage method, which includes: grouping all the data to be stored in the data set to obtain the group identifier and the intra-group identifier corresponding to each data to be stored; for each data to be stored, based on the identifier of the data set, the group identifier and the intra-group identifier, determining the identifier of the data to be stored; determining the metadata of the data to be stored and the characteristic information of the data to be stored; using the identifier of the data to be stored as a key and the metadata of the data to be stored as a value to store the metadata; using the identifier of the data to be stored as a key and the original data of the data to be stored as a value, storing the original data based on the group identifier and the intra-group identifier corresponding to the data to be stored; using the identifier of the data to be stored as a key and the characteristic information of the data to be stored as a value, storing the characteristic information based on the group identifier and the intra-group identifier corresponding to the data to be stored.

[0117] The processor 910 can call the logic instructions in the memory 930 to execute the data query method, which includes: obtaining a data query request, the data query request including the identifier of the data to be queried, and the identifier of the data to be queried including the target identifier, target group identifier and target group identifier of the data set; based on the target identifier of the data set, obtaining the storage path of the target metadata of the data to be queried, the storage path of the target original data of the data to be queried and the storage path of the target feature information of the data to be queried, the target metadata, the target original data and the target feature information are stored based on the data storage method described in any of the above embodiments; querying the target metadata under the storage path of the target metadata; querying the target original data based on the target group identifier and the target group identifier under the storage path of the target original data, and querying the target feature information based on the target group identifier and the target group identifier under the storage path of the target feature information; determining the target metadata, the target original data and the target feature information as the query result of the data to be queried.

[0118] Furthermore, the logic instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0119] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data storage method provided by the above methods, which method includes: grouping all the data to be stored in the data set to obtain the group identifier and the intra-group identifier corresponding to each data to be stored; for each data to be stored, based on the identifier of the data set, the group identifier and the intra-group identifier, determining the identifier of the data to be stored; determining the metadata of the data to be stored and the characteristic information of the data to be stored; using the identifier of the data to be stored as a key and the metadata of the data to be stored as a value to store the metadata; using the identifier of the data to be stored as a key and the original data of the data to be stored as a value, storing the original data based on the group identifier and the intra-group identifier corresponding to the data to be stored; using the identifier of the data to be stored as a key and the characteristic information of the data to be stored as a value, storing the characteristic information based on the group identifier and the intra-group identifier corresponding to the data to be stored.

[0120] A computer is capable of executing the data query method provided by each of the above methods, which includes: obtaining a data query request, the data query request including an identifier of the data to be queried, the identifier of the data to be queried including a target identifier, a target group identifier, and a target intra-group identifier of a data set; based on the target identifier of the data set, obtaining a storage path of target metadata of the data to be queried, a storage path of target original data of the data to be queried, and a storage path of target feature information of the data to be queried, the target metadata, the target original data, and the target feature information being stored based on the data storage method described in any of the above embodiments; querying the target metadata under the storage path of the target metadata; querying the target original data based on the target group identifier and the target intra-group identifier under the storage path of the target original data, and querying the target feature information based on the target group identifier and the target intra-group identifier under the storage path of the target feature information; determining the target metadata, the target original data, and the target feature information as the query result of the data to be queried.

[0121] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data storage method provided by the above-mentioned methods, the method comprising: grouping all the data to be stored in the data set to obtain the group identifier and the intra-group identifier corresponding to each data to be stored; for each data to be stored, determining the identifier of the data to be stored based on the identifier of the data set, the group identifier and the intra-group identifier; determining the metadata of the data to be stored and the characteristic information of the data to be stored; using the identifier of the data to be stored as a key and the metadata of the data to be stored as a value to store the metadata; using the identifier of the data to be stored as a key and the original data of the data to be stored as a value, storing the original data based on the group identifier and the intra-group identifier corresponding to the data to be stored; using the identifier of the data to be stored as a key and the characteristic information of the data to be stored as a value, storing the characteristic information based on the group identifier and the intra-group identifier corresponding to the data to be stored.

[0122] When the computer program is executed by a processor, it is implemented to execute the data query method provided by the above methods, which method includes: obtaining a data query request, the data query request includes an identifier of the data to be queried, and the identifier of the data to be queried includes a target identifier, a target group identifier, and a target intra-group identifier of a data set; based on the target identifier of the data set, obtaining a storage path of target metadata of the data to be queried, a storage path of target original data of the data to be queried, and a storage path of target feature information of the data to be queried, the target metadata, the target original data, and the target feature information being stored based on the data storage method described in any of the above embodiments; querying the target metadata under the storage path of the target metadata; querying the target original data based on the target group identifier and the target intra-group identifier under the storage path of the target original data, and querying the target feature information based on the target group identifier and the target intra-group identifier under the storage path of the target feature information; determining the target metadata, the target original data, and the target feature information as a query result of the data to be queried.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0124] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data storage method, characterized in that: include: Group all the data to be stored in the data set to obtain the group identifier and the intra-group identifier corresponding to each data to be stored; For each of the data to be stored, determining an identifier of the data to be stored based on the identifier of the data set, the group identifier, and the intra-group identifier; Determining metadata of the data to be stored and feature information of the data to be stored; Using the identifier of the data to be stored as a key and the meta-information of the data to be stored as a value to store the meta-information; Using the identifier of the data to be stored as a key and the original data of the data to be stored as a value, the original data is stored based on the group identifier and the intra-group identifier corresponding to the data to be stored; The identifier of the data to be stored is used as a key, and the characteristic information of the data to be stored is used as a value. The characteristic information is stored based on the group identifier and the intra-group identifier corresponding to the data to be stored.

2. The data storage method according to claim 1, wherein: The method further comprises: Respectively obtaining a storage path of the meta information, a storage path of the original data, and a storage path of the feature information; Obtaining version information of the meta information, version information of the original data, and version information of the feature information; Adding the version information of the meta information to the storage path of the meta information; Adding the version information of the original data to the storage path of the original data; The version information of the feature information is added to the storage path of the feature information.

3. The data storage method according to claim 2, wherein: The method further comprises: Adding a version field to the meta information; The version information of the original data and the version information of the characteristic information are determined as the field value of the version field.

4. The data storage method according to claim 2, wherein: The method further comprises: Receive data update request; When the data update request includes updated content of the meta information of the target data to be stored, performing a full update on the meta information of all the data to be stored; In a case where the data update request includes update content of the original data of the target data to be stored, incrementally updating the original data of the target data to be stored based on the update content of the original data; In a case where the data update request includes the updated content of the characteristic information of the target data to be stored, the characteristic information of the target data to be stored is incrementally updated based on the updated content of the characteristic information.

5. The data storage method according to claim 4, characterized in that: The method further comprises: After all the metadata of the data to be stored is fully updated, the version information of the metadata of each data to be stored is updated; After incrementally updating the original data of the target data to be stored, updating the version information of the original data of the target data to be stored; After incrementally updating the characteristic information of the target data to be stored, updating the version information of the characteristic information of the target data to be stored; The updated version information of the meta information, the updated version information of the original data, and the updated version information of the feature information are stored in a version update file.

6. The data storage method according to claim 4, characterized in that: The method further comprises: receiving a data processing request, wherein the data processing request includes a data processing type; Based on the number of nodes in the distributed execution engine and the maximum processing capacity of each node, all the data to be stored are divided into tasks to obtain at least one task corresponding to each node; For each of the tasks, the node corresponding to the task processes the metadata, original data and feature information in the data to be stored corresponding to the task based on the data processing type to obtain updated content of the metadata, updated content of the original data and updated content of the feature information, wherein the data to be processed corresponding to the tasks processed simultaneously by different nodes are located in different groups.

7. A data query method, characterized in that: include: Obtaining a data query request, wherein the data query request includes an identifier of the data to be queried, and the identifier of the data to be queried includes a target identifier of a data set, a target group identifier, and an identifier within a target group; Based on the target identifier of the data set, obtaining a storage path of target meta information of the data to be queried, a storage path of target original data of the data to be queried, and a storage path of target feature information of the data to be queried, wherein the target meta information, the target original data, and the target feature information are stored based on the data storage method according to any one of claims 1 to 6; Querying the target meta information in the storage path of the target meta information; In the storage path of the target original data, based on the target group identifier and the target intra-group identifier, query the target original data, and in the storage path of the target feature information, based on the target group identifier and the target intra-group identifier, query the target feature information; The target meta information, the target original data and the target feature information are determined as the query result of the data to be queried.

8. The data query method according to claim 7, characterized in that: The step of querying the target meta information in the storage path of the target meta information includes: Obtaining version information of the target meta information from a version update file; The target meta information is queried based on the storage path of the target meta information and the version information of the target meta information.

9. The data query method according to claim 7, characterized in that: The step of querying the target original data based on the target group identifier and the target intra-group identifier in the storage path of the target original data, and querying the target feature information based on the target group identifier and the target intra-group identifier in the storage path of the target feature information includes: Parsing the target meta information to obtain version information of the target original data and version information of the target feature information in the version field of the target meta information; querying the target original data based on the storage path of the target original data, the version information of the target original data, the target group identifier, and the target intra-group identifier; The target feature information is queried based on the storage path of the target feature information, the version information of the target feature information, the target group identifier, and the target intra-group identifier.

10. A data storage device, characterized in that: include: A grouping module is used to group all the data to be stored in the data set and obtain the group identifier and the intra-group identifier corresponding to each data to be stored; a determination module, configured to determine, for each of the data to be stored, an identifier of the data to be stored based on the identifier of the data set, the group identifier, and the intra-group identifier; The determining module is further configured to determine the metadata of the data to be stored and the feature information of the data to be stored; a storage module, configured to store the metadata of the data to be stored using the identifier of the data to be stored as a key and the metadata of the data to be stored as a value; The storage module is further configured to use the identifier of the data to be stored as a key and the original data of the data to be stored as a value, and store the original data based on the group identifier and the intra-group identifier corresponding to the data to be stored; The storage module is further configured to use the identifier of the data to be stored as a key and the characteristic information of the data to be stored as a value, and store the characteristic information based on the group identifier and the intra-group identifier corresponding to the data to be stored.

11. A data query device, characterized in that: include: An acquisition module, configured to acquire a data query request, wherein the data query request includes an identifier of the data to be queried, and the identifier of the data to be queried includes a target identifier of a data set, a target group identifier, and an identifier within a target group; The acquisition module is configured to acquire, based on the target identifier of the data set, a storage path of target metadata of the data to be queried, a storage path of target original data of the data to be queried, and a storage path of target feature information of the data to be queried, wherein the target metadata, the target original data, and the target feature information are stored based on the data storage method according to any one of claims 1 to 6; A query module, configured to query the target meta information under the storage path of the target meta information; The query module is configured to query the target original data based on the target group identifier and the target intra-group identifier in the storage path of the target original data, and query the target feature information based on the target group identifier and the target intra-group identifier in the storage path of the target feature information; The determination module is used to determine the target meta information, the target original data and the target feature information as the query result of the data to be queried.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the processor implements the data storage method according to any one of claims 1 to 6, or implements the data query method according to any one of claims 7 to 9.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the data storage method according to any one of claims 1 to 6, or implements the data query method according to any one of claims 7 to 9.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data storage method according to any one of claims 1 to 6, or implements the data query method according to any one of claims 7 to 9.

Images