Distributed data set management system based on versioned metadata

Abstract

The embodiment of the invention provides a distributed data set management system based on versioned metadata, and the system comprises a data management unit which receives a data set, creates versioned metadata for the data set, and distributes the data set corresponding to version information to a calculation cluster according to the version information of a training task; wherein the versioned metadata comprises version information and metadata; the data storage unit is used for carrying out mirroring processing on the data set to obtain a data set mirror image and storing the data set mirror image in an object storage cluster; and the computing cluster downloads the data set according to the data set distributed by the data management unit and executes the training task.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a distributed dataset management system based on versioned metadata. Background Technology

[0002] When training deep learning models using medical AI platforms, a large amount of data is typically required, including various types of data, most of which are small, fragmented files. In a cluster network environment, the efficiency of accessing these files directly affects the training efficiency.

[0003] In existing technologies, a centralized Network Attached Storage (NAS) or a simple distributed file system, such as a Network File System (NFS), is typically used to manage training data. All image data is usually stored in a large shared folder, such as ` / datasets / autonomous_driving / 2024 / 05 / `. New images are continuously added to this folder, or existing images are directly overwritten or modified. When a training task starts, the training script reads all the images in this shared folder. Meanwhile, data engineers are constantly adding new data and modifying problematic old data. This leads to two problems: firstly, the dataset itself is constantly being updated during training, causing version inconsistencies and resulting in unreproducible training results; secondly, when multiple training tasks start simultaneously, they all need to read millions of images from the same NAS server, causing the NAS's network I / O and disk I / O to quickly reach performance bottlenecks. This results in training tasks spending most of their time waiting for data, leaving the computing GPU idle and wasting significant resources. On the other hand, if the original training task relies on an older version of a certain image and the data accidentally overwrites it, it can lead to errors in the original training task that are difficult to detect, thus causing collaboration conflicts.

[0004] Chengda provides an optimized clustered storage system to improve storage speed and response time, thereby improving the platform's performance and efficiency. It can also help the platform better manage and utilize storage space and resolve collaboration conflicts when multiple training tasks are performed simultaneously. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a distributed dataset management system based on versioned metadata. To achieve the above objectives, this invention provides a distributed dataset management system based on versioned metadata, comprising: The data management unit receives a dataset, creates versioned metadata for the dataset, and distributes the first dataset corresponding to the version information to the computing cluster according to the version information of the training task; wherein, the versioned metadata includes version information and metadata; The data storage unit performs mirroring processing on the dataset to obtain a dataset image, and stores the dataset image in the object storage cluster; The computing cluster downloads the dataset based on the first dataset distributed by the data management unit and completes the training task.

[0006] Preferably, the dataset includes an original dataset and an updated dataset, and receiving the dataset and creating versioned metadata for the dataset specifically includes: Receive the original dataset and create a first versioned metadata for the original dataset; Receive the updated dataset and create a second versioned metadata for the updated dataset.

[0007] Preferably, the dataset mirror uses a block device format file.

[0008] Preferably, storing the dataset in a mirrored form within an object storage cluster specifically includes: The dataset is then segmented to obtain segmented data, which is then stored in an object storage cluster.

[0009] More preferably, the dataset includes multiple first files, and the step of splitting the dataset to obtain split data and storing the split data in an object storage cluster specifically includes: The first file in the dataset is segmented to obtain multiple first segmented data; wherein, the first file includes a globally unique identifier; Multiple first-splitter data are stored on different storage nodes in the object storage cluster; wherein, the first-splitter data includes a globally unique identifier.

[0010] More preferably, the globally unique identifier is specifically a hash value generated based on the first file.

[0011] Preferably, distributing the first dataset corresponding to the version information to the computing cluster according to the version information of the training task specifically includes: Obtain metadata based on the version information; The availability of the first dataset corresponding to the version information is broadcast to the computing cluster to trigger each computing node in the computing cluster to download the first dataset.

[0012] More preferably, after broadcasting the availability of the first dataset corresponding to the version information to the computing cluster, the system further includes: The compute node is connected to each storage node in the object storage cluster; The computing node simultaneously obtains multiple segments of data corresponding to the first dataset from each storage node.

[0013] More preferably, after the computing node simultaneously obtains multiple partitioned data corresponding to the first dataset from each storage node, the computing node mounts the image corresponding to the first dataset as the computing node's file system by mounting.

[0014] This invention provides a distributed dataset management system based on versioned metadata. It achieves efficient data management through incremental version control via a data management unit, storing only newly added or corrected datasets on multiple storage nodes in an object storage cluster to save space and recording detailed metadata information. It employs a content hash-based object storage cluster for deduplication optimization and distributed storage, ensuring that duplicate files across versions are stored as a single instance. Training tasks can specify a particular dataset version, and the system dynamically schedules computing nodes to pull the required data blocks from multiple storage nodes in parallel, avoiding centralized bottlenecks. It also supports decoupling operations for training task reproduction and data iteration for new training tasks, forming a technical architecture system with efficient collaboration between storage, management, and computing resources, effectively improving the efficiency of dataset storage and distribution. Attached Figure Description

[0015] Figure 1 A system block diagram of a distributed dataset management system based on versioned metadata provided in this embodiment of the invention; Figure 2 This is a flowchart of the training task dataset distribution process of a distributed dataset management system based on versioned metadata provided by this invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0018] The versioned metadata-based distributed dataset management system provided in this invention applies versioned management technology to distributed dataset management, changing the problems of slow dataset storage and unreproducible updates in existing distributed dataset management technologies. This improves the storage speed and response time of the cluster storage system, enabling more efficient management and utilization of datasets.

[0019] Figure 1 A system block diagram of a distributed dataset management system based on versioned metadata provided in this embodiment of the invention is shown below. Figure 1 The technical solution of the present invention will be described with reference to specific embodiments.

[0020] like Figure 1 As shown, the distributed dataset management system 10000 based on versioned metadata provided by this invention includes a data management unit 10001, a data storage unit 10002, and a computing cluster 10003. Each component of the system will be described in detail below.

[0021] Data management unit 10001 receives the dataset, creates versioned metadata for the dataset, and distributes the first dataset corresponding to the version information to the computing cluster according to the version information of the training task.

[0022] Versioned metadata includes version information and metadata.

[0023] Specifically, in this embodiment of the invention, the data management unit 10001 is used to receive externally input datasets and create versioned metadata for the received datasets to complete version management of the data. At the same time, it receives externally input training task requirements, obtains the metadata corresponding to the version based on the version information in the training task requirements, and realizes the distribution of the dataset.

[0024] In this embodiment of the invention, the dataset includes an original dataset and an updated dataset. Receiving the dataset and creating versioned metadata for the dataset specifically includes: receiving the original dataset and creating a first versioned metadata for the original dataset; receiving the updated dataset and creating a second versioned metadata for the updated dataset.

[0025] In a specific example of this invention, the original dataset refers to the dataset initially received by the data management unit from external input, while the updated dataset refers to updated or expanded data received from external input after the original dataset has been received. For example, if a data engineer packages 5 million images collected in the first week of May and inputs them into data management unit 10001, the data management unit will identify the received packaged 5 million images as the original dataset. Upon receiving the original dataset, a first versioned metadata is generated for it, which includes at least a file list, a version number, and a creation time. Of course, the first versioned metadata may also include other information about the dataset, which cannot all be listed here. In this embodiment, the first versioned metadata can be named, for example, autonomous_driving:v1.0. After inputting the original dataset, some of the 5 million images input to the data management unit need to be updated, and sometimes the dataset needs to be supplemented. For example, a week later, 1 million new images were added to the dataset of 5 million images, and 10,000 images from the original dataset were corrected, such as by adding annotations, and are recorded as 10,000 corrected images. At this time, instead of overwriting the original dataset, a new versioned metadata, namely the second versioned metadata, is generated by the data management unit and named autonomous_driving:v1.1. Thus, autonomous_driving:v1.1 shares the unmodified data with autonomous_driving:v1.0, storing only the incremental 1 million new images and 10,000 corrected images.

[0026] In a preferred embodiment of the present invention, the data management unit 10001 is further configured to receive training task requirements from external input, obtain the metadata corresponding to the version information in the training task requirements, and distribute the dataset. Specific steps include: the data management unit obtains metadata based on the received version information; and broadcasts the availability of the first dataset corresponding to the version information to the computing cluster to trigger the computing nodes to download the first dataset. For example, in a specific example of the present invention, task A is to reproduce a previously performed training task, submitting a training task Train_v1.0 and specifying the required dataset version, i.e., the version information is autonomous_driving:v1.0. The dataset management unit, based on the training task requirements, finds the versioned metadata autonomous_driving:v1.0 and distributes the metadata of this version to the computing cluster. Task B is to train based on the latest data, submitting a training task Train_v1.1 and specifying the required dataset version, i.e., the version information is autonomous_driving:v1.1. Then, based on the training task requirements, the dataset management unit finds the versioned metadata autonomous_driving:v1.1 and distributes and schedules the metadata corresponding to autonomous_driving:v1.1 to the computing cluster.

[0027] Data storage unit 10002 performs mirroring processing on the dataset to obtain a dataset image, and stores the dataset image in the object storage cluster.

[0028] Specifically, in this embodiment of the invention, the data storage unit performs mirroring processing on the received dataset to obtain a dataset image, and then stores the dataset image in the object storage cluster.

[0029] In a preferred embodiment of the present invention, the dataset mirror uses a block device format file, which is a type of block device model file. Block device format files are device-level files that can be loaded and used as devices. Therefore, when accessed at higher layers, the original access method can be maintained without modifying the file access interface. Furthermore, this file access method is supported by different deep learning frameworks, exhibiting excellent compatibility. Secondly, the metadata of fragmented files is recorded in the file, and the file metadata required for training only needs to depend on this dataset mirror, which can greatly improve access efficiency.

[0030] In a preferred embodiment of the present invention, storing the dataset mirror in an object storage cluster specifically includes: splitting the dataset to obtain split data, and storing the split data in the object storage cluster. In this embodiment, the dataset includes multiple first files; each first file in the dataset is split to obtain multiple first split data; wherein each first file includes a globally unique identifier; and the multiple first split data are stored on different storage nodes in the object storage cluster; wherein each first split data includes a globally unique identifier. In this embodiment, the globally unique identifier is an identifier that can uniquely identify the first file; for example, a hash value generated based on the first file can be used as the globally unique identifier.

[0031] In a specific example of this invention, the dataset includes multiple images, each image being a first file. During dataset mirroring, each image is segmented into multiple first-segment data. These segmented first-segment data are then stored on storage nodes within an object storage cluster. This avoids the situation where the bandwidth of a single point limits data throughput due to the large data volume of a single image, resulting in low image data transmission efficiency. Furthermore, the segmented data storage of this invention enables multi-point parallel data transmission, fully utilizing cluster bandwidth and improving network bandwidth utilization efficiency.

[0032] Computing cluster 10003 distributes the first dataset corresponding to the version information of the training task to the computing cluster.

[0033] Specifically, in this embodiment of the invention, the computer cluster 10003 includes multiple computing nodes. The computing nodes serve as the devices for the actual execution of the training task and process the distributed images separately according to the image size.

[0034] In a preferred embodiment of the present invention, after the data management unit broadcasts the availability of the first dataset corresponding to the version information to the computing cluster, the computing nodes connect to each storage node in the object storage cluster; the computing nodes simultaneously obtain multiple segments of data corresponding to the first dataset from each storage node. For example, after the computer cluster starts the training task, the program on the computing node does not pull data from a central server, but instead pulls image patches corresponding to the version information simultaneously from multiple server nodes in the object storage cluster.

[0035] In a preferred embodiment of the present invention, after the computing node simultaneously obtains multiple partitioned data corresponding to the first dataset from each storage node, the computing node mounts the image corresponding to the first dataset as its file system. After the model files are distributed, the computing node mounts the image as the device's file system. At this point, the training task can access the dataset content through a common file interface, and the mature file system caching mechanism of the kernel can be used to improve file system access efficiency. Furthermore, metadata access to small files during training becomes a local operation that no longer relies on the network. Local operations can leverage the acceleration mechanism of the mature local file system to reduce latency and significantly improve access efficiency.

[0036] In a specific example of an embodiment of the present invention, based on a distributed dataset management system with versioned metadata provided by the present invention, user B submits the latest model training, and the overall process of the system receiving the training task and distributing the dataset is as follows: Figure 2 As shown, it includes the following steps: S110, Task Trigger: The data management unit inputs the training task corresponding to Train_v1.1 from an external source, where the version information of this training task is specified as autonomous_driving:v1.1. S120, Scheduling Response: The data management unit queries the metadata database of the DM system to obtain the incremental list corresponding to autonomous_driving:v1.1, including unmodified images shared by autonomous_driving:v1.1 and autonomous_driving:v1.0, 1 million newly added images, and 10,000 corrected images.

[0037] S130, Resource Location: The data management unit determines which storage nodes in the object storage cluster, i.e., physical storage nodes, these incremental data are distributed on based on the file hash mapping table.

[0038] S140, Concurrent Read: Compute instances on each compute node in a computer cluster simultaneously send "HTTP Range Request" requests to multiple storage nodes to achieve high-speed data loading.

[0039] S150, Hot Data Cache: Compute nodes will automatically cache frequently accessed hot data to the local SSD cache layer.

[0040] The distributed dataset management system based on versioned metadata provided in this embodiment of the invention can also support multi-task concurrency. That is, it can simultaneously receive multiple training tasks from external input, find datasets for multiple training tasks, and distribute the datasets corresponding to the version information specified by the training tasks to each training task, thereby providing efficient computing nodes with datasets to execute multiple training tasks. The overall process of distributing the dataset for each training task among multiple training tasks is the same as the example above, and will not be repeated here.

[0041] This invention provides a distributed dataset management system based on versioned metadata. It achieves efficient data management through incremental version control via a data management unit, storing only newly added or corrected datasets on multiple storage nodes in an object storage cluster to save space and recording detailed metadata information. It employs a content hash-based object storage cluster for deduplication optimization and distributed storage, ensuring that duplicate files across versions are stored as a single instance. Training tasks can specify a particular dataset version, and the system dynamically schedules computing nodes to pull the required data blocks from multiple storage nodes in parallel, avoiding centralized bottlenecks. It also supports decoupling operations for training task reproduction and data iteration for new training tasks, forming a technical architecture system with efficient collaboration between storage, management, and computing resources, effectively improving the efficiency of dataset storage and distribution.

[0042] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0043] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0044] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed dataset management system based on versioned metadata, characterized in that, The distributed dataset management system includes, The data management unit receives a dataset, creates versioned metadata for the dataset, and distributes the first dataset corresponding to the version information to the computing cluster according to the version information of the training task; wherein, the versioned metadata includes version information and metadata; The data storage unit performs mirroring processing on the dataset to obtain a dataset image, and stores the dataset image in the object storage cluster; The computing cluster downloads the dataset based on the first dataset distributed by the data management unit and executes the training task.

2. The distributed dataset management system according to claim 1, characterized in that, The dataset includes an original dataset and an updated dataset. Receiving the dataset and creating versioned metadata for the dataset specifically includes: Receive the original dataset and create a first versioned metadata for the original dataset; Receive the updated dataset and create a second versioned metadata for the updated dataset.

3. The distributed dataset management system according to claim 1, characterized in that, The dataset mirror uses files in block device format.

4. The distributed dataset management system according to claim 1, characterized in that, The specific steps of storing the dataset in a mirror image within an object storage cluster include: The dataset is then segmented to obtain segmented data, which is then stored in an object storage cluster.

5. The distributed dataset management system according to claim 4, characterized in that, The dataset includes multiple first files. The process of splitting the dataset to obtain split data and storing the split data in an object storage cluster specifically includes: The first file in the dataset is segmented to obtain multiple first segmented data; wherein, the first file includes a globally unique identifier; Multiple first-splitter data are stored on different storage nodes in the object storage cluster; wherein, the first-splitter data includes a globally unique identifier.

6. The distributed dataset management system according to claim 5, characterized in that, The globally unique identifier is specifically a hash value generated based on the first file.

7. The distributed dataset management system according to claim 1, characterized in that, Distributing the first dataset corresponding to the version information of the training task to the computing cluster specifically includes: Obtain metadata based on the version information; The availability of the first dataset corresponding to the version information is broadcast to the computing cluster to trigger each computing node in the computing cluster to download the first dataset.

8. The distributed dataset management system according to claim 7, characterized in that, After broadcasting the availability of the first dataset corresponding to the version information to the computing cluster, the system further includes: The compute node is connected to each storage node in the object storage cluster; The computing node simultaneously obtains multiple segments of data corresponding to the first dataset from each storage node.

9. The distributed dataset management system according to claim 8, characterized in that, After the computing node simultaneously obtains multiple partitioned data corresponding to the first dataset from each storage node, the computing node mounts the image corresponding to the first dataset as the computing node's file system by mounting.

Images