Training data set version management method and system

By generating unique identifiers for the dataset using UUIDs, generating version identifiers using V = D + T + S, creating static snapshots and performing hash value verification, and introducing a data signature mechanism, the problem of lax version control of training datasets is solved, achieving efficient and secure dataset management and improving the reliability and efficiency of model training.

CN118964335AActive Publication Date: 2024-11-15SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411441169.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-16
Publication Date
2024-11-15
Estimated Expiration
2044-10-16

AI Technical Summary

Technical Problem

In existing technologies, training datasets suffer from lax version control, missing version information, low update efficiency, and difficulty in backtracking, leading to frequent data consistency issues that affect model training results and reliability. Furthermore, data is easily tampered with, lost, or damaged during transmission, storage, and processing.

Method used

UUIDs are used to generate unique identifiers for datasets, and the formula V = D + T + S is used to generate version identifiers. Static snapshots are created and hash value verification is performed. A data signature mechanism is introduced to ensure the integrity of the dataset. Through multi-level version control and intelligent management, efficient tracking and management of dataset versions are achieved.

Benefits of technology

It improves the accuracy and efficiency of data management, ensures clear and traceable dataset versions, reduces manual intervention and error rates, guarantees data integrity and security, and facilitates the debugging and optimization of machine learning models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964335A_ABST
    Figure CN118964335A_ABST
Patent Text Reader

Abstract

The invention discloses a training data set version management method and system, and relates to the technical field of data management. Comprising the steps of 1, establishing a data set for model training and generating a unique identifier for the data set, 2, managing a data set version, 21, establishing the data set version, 22, generating a data set version identifier by using a formula V = D + T + S, and 23, creating a static snapshot of the selected data set according to the data set version; step 24, according to the static snapshot, checking whether a file object of which the content is changed in the data set is contained in the static snapshot, if so, copying the current version of the file object to a corresponding storage of the static snapshot, updating an object index of metadata of the static snapshot, and then carrying out change operation on the data set, and step 25, checking the integrity of the data set; the data management complexity is reduced, and the quality and reliability of model training are guaranteed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention discloses a training data set version management method and system, and relates to the technical field of data management. Background Art

[0002] With the rapid development of artificial intelligence technology and the advent of the big data era, the scale of training data sets, as the cornerstone of machine learning models, has become increasingly large and complex. However, there are currently problems such as lax version control of training data sets, missing version information, low update efficiency, and difficulty in backtracking. These problems not only increase the complexity of data management, but may also lead to frequent data consistency problems, which in turn affects the results and reliability of model training. At the same time, due to the lack of an effective version control mechanism, data differences between different versions are difficult to accurately track, and historical version data is often difficult to restore. More seriously, during the process of data transmission, storage, and processing, the integrity of the data may be damaged, such as data tampering, loss, or damage, which also directly affects the results and reliability of model training. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides a training dataset version management method and system, which significantly reduces the complexity of data management, improves the efficiency and accuracy of version management, and thus ensures the quality and reliability of model training.

[0004] The specific scheme proposed by the present invention is: The present invention provides a training data set version management method, comprising: Step 1: Create a data set for model training and generate a unique identifier for the data set. The unique identifier uses the universal unique identifier UUID. Step 2: Manage dataset versions: Step 21: Create a dataset version. Step 22: Use the formula V = D + T + S to generate a dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp; Step 23: Create a static snapshot of the selected dataset according to the dataset version. The metadata saved in the static snapshot includes the dataset version identifier V, the creation time, the list of file objects included in the dataset, the object index, and the version identifier of each file object; Step 24: Based on the static snapshot, check whether the file object in the data set that will undergo content changes is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content changes, and update the object index of the metadata of the static snapshot before performing the data set change operation. Step 25: Verify the integrity of the dataset: According to the dataset version, clarify the boundaries of the dataset, determine which file objects are included in the current dataset version, use the hash algorithm to calculate the hash value of each file object in the dataset and record the hash value. Before the dataset is scheduled to execute the training task, traverse all file objects in the dataset, use the same hash algorithm to recalculate the hash value of each file object, and compare the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0005] Furthermore, in step 1 of the training dataset version management method, the file objects in the dataset are also classified and managed, including: identifying the file object type according to the extension type of the file contained in the current dataset, the file object type includes pictures, videos, audio and text, and saving the file object type and the save path of the file object in the dataset as metadata in the database.

[0006] Furthermore, there are two ways to establish a dataset version in step 21 of the training dataset version management method. One way is to capture a static snapshot of the data based on the original file object content in the dataset, establish and publish the dataset version; the other way is to change the file object content based on the file object content of the existing dataset version based on the published dataset version, generate a new dataset version and publish it.

[0007] Furthermore, in step 25 of the training dataset version management method, verifying the dataset integrity includes: Introduce a data signature mechanism, use a hash value record file to record the hash value of each file object, use a private key to encrypt the hash value record file, and generate a digital signature. Before the data set is scheduled to execute a training task, the digital signature is decrypted using the corresponding public key to obtain the original hash value record file, and all file objects in the data set are traversed. The hash value of each file object is recalculated using the same hash algorithm, and the newly calculated hash value is compared with the hash value recorded in the hash value record file. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0008] The present invention also provides a training data set version management system, including a data set management module, a version management module and an integrity verification module. The dataset management module establishes a dataset for model training and generates a unique identifier for the dataset. The unique identifier uses the universal unique identifier UUID. The version management module manages dataset versions: Create a dataset version, The formula V = D + T + S is used to generate the dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp. According to the dataset version, a static snapshot of the selected dataset is created. The metadata saved in the static snapshot includes the dataset version identifier V, the creation time, the list of file objects contained in the dataset, the object index, and the version identifier of each file object; According to the static snapshot, check whether the file object whose content will be changed in the data set is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content is changed, update the object index of the metadata of the static snapshot, and then perform the data set change operation; The integrity verification module verifies the integrity of the data set: according to the data set version, it clarifies the boundaries of the data set, determines which file objects are included in the current data set version, uses a hash algorithm to calculate the hash value of each file object in the data set and records the hash value. Before the data set is scheduled to execute a training task, it traverses all file objects in the data set, uses the same hash algorithm to recalculate the hash value of each file object, and compares the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0009] Furthermore, the data set management module of the training data set version management system also classifies and manages the file objects in the data set, including: identifying the file object type according to the extension type of the file contained in the current data set, the file object type includes pictures, videos, audio and text, and saving the file object type and the save path of the file object in the data set as metadata in the database.

[0010] Furthermore, the version management module of the training dataset version management system establishes a dataset version in two ways. One way is to capture a static snapshot of the data based on the original file object content in the dataset, establish and publish the dataset version; the other way is to change the file object content based on the file object content of the existing dataset version based on the published dataset version, generate a new dataset version and publish it.

[0011] Furthermore, the integrity verification module of the training dataset version management system verifies the integrity of the dataset, including: Introduce a data signature mechanism, use a hash value record file to record the hash value of each file object, use a private key to encrypt the hash value record file, and generate a digital signature. Before the data set is scheduled to execute a training task, the digital signature is decrypted using the corresponding public key to obtain the original hash value record file, and all file objects in the data set are traversed. The hash value of each file object is recalculated using the same hash algorithm, and the newly calculated hash value is compared with the hash value recorded in the hash value record file. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0012] The benefits of the method of the present invention are: Through version control and intelligent data processing mechanisms, efficient tracking and management of training data set versions are achieved. This not only solves many problems faced in data set management, such as version confusion, data loss, and tampering risks, but also provides strong support for the training and iteration of machine learning models.

[0013] The use of multi-level version control strategies and refined management can accurately capture every change in the data set and generate a unique version identifier, ensuring that the version of the data set is clear and traceable, while reducing manual intervention and error rates, greatly improving the accuracy and efficiency of data set management. Users can easily query, compare and roll back to any historical version, thereby accelerating the debugging and optimization process of machine learning models.

[0014] The process automatically manages and monitors the entire process of dataset creation, modification, publication and use, which improves the efficiency of dataset management, ensures data integrity and security, and promotes team collaboration and data sharing. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0016] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0017] Embodiment 1: The present invention provides a training data set version management method, comprising: Step 1: Create a data set for model training and generate a unique identifier for the data set. The unique identifier can be a 32-bit universal unique identifier UUID.

[0018] Wherein, step 1 also classifies and manages the file objects in the dataset, including: identifying the file object type according to the extension type of the file contained in the current dataset, the file object type includes pictures, videos, audio and text, etc., and saving the file object type and the save path of the file object in the dataset as metadata in the database.

[0019] Step 2: Manage dataset versions: Step 21: Establish a dataset version. There are two ways to establish a dataset version. One way is to capture a static snapshot of the data based on the original file object content in the dataset, establish a dataset version and publish it; the other way is to change the file object content based on the existing dataset version, generate a new dataset version and publish it.

[0020] Step 22: Use the formula V = D + T + S to generate a dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp.

[0021] Step 23: Create a static snapshot of the selected dataset based on the dataset version. The metadata saved in the static snapshot includes the dataset version identifier V, the creation time, the list of file objects contained in the dataset, the object index, and the version identifier of each file object. Each dataset version is an accurate snapshot of the original dataset at a specific point in time. Therefore, the dataset version not only captures the static snapshot of the data, but also contains the complete history and evolution trajectory of the dataset. The dataset version identifier of the present invention can ensure the high traceability of the dataset, so that any questions or backtracking requirements about the dataset can be quickly queried, responded to and resolved.

[0022] Step 24: In the management process of static snapshots, the logic of copy-on-write is adopted. When the content of the file in the data set changes, according to the static snapshot, check whether the file object in the data set whose content will change is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content changes, and update the object index of the metadata of the static snapshot before performing the data set change operation. This process minimizes the storage space occupied, and there is no need to copy the entire data set for each static snapshot. Static snapshots save metadata, not the data files themselves. A large number of data files are still actually stored in the initial object storage or file storage. Therefore, only when the data file of the published data set version changes, the file is copied to the storage of the static snapshot for backup to ensure the integrity and consistency of the data, while reducing costs and avoiding unnecessary resource consumption.

[0023] Step 25: Verify the integrity of the data set: According to the data set version, clarify the boundaries of the data set, determine which file objects are included in the current data set version, use a hash algorithm to calculate the hash value of each file object in the data set and record the hash value. Hash algorithms such as MD5, SHA-1, and preferably SHA-256, etc. Before the data set is scheduled to execute a training task, traverse all file objects in the data set, use the same hash algorithm to recalculate the hash value of each file object, and compare the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0024] Verifying the integrity of the data set in step 25 may further include: A data signature mechanism is introduced, and the hash value record file is used to record the hash value of each file object. The hash value record file can be integrated into the static snapshot metadata of the dataset version. The hash value record file is encrypted with a private key to generate a digital signature. Before the data set is scheduled to execute a training task, the digital signature is decrypted using the corresponding public key to obtain the original hash value record file, and all file objects in the data set are traversed. The hash value of each file object is recalculated using the same hash algorithm, and the newly calculated hash value is compared with the hash value recorded in the hash value record file. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0025] Through this series of technical measures, not only can the integrity of the data set be strictly protected throughout the entire process of creation, transmission and use, but potential security threats can also be discovered and responded to in a timely manner, providing higher reliability and security guarantees for the machine learning training process, and helping to improve the overall quality and efficiency of data science work.

[0026] Embodiment 2: The present invention also provides a training data set version management system, including a data set management module, a version management module and an integrity verification module. The dataset management module establishes a dataset for model training and generates a unique identifier for the dataset. The unique identifier uses the universal unique identifier UUID. The version management module manages dataset versions: Create a dataset version, The formula V = D + T + S is used to generate the dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp. According to the dataset version, create a static snapshot of the selected dataset. The metadata of the static snapshot includes the dataset version identifier V, the creation time, the list of file objects contained in the dataset, the object index, and the version identifier of each file object. According to the static snapshot, check whether the file object whose content will be changed in the data set is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content is changed, update the object index of the metadata of the static snapshot, and then perform the data set change operation; The integrity verification module verifies the integrity of the data set: according to the data set version, it clarifies the boundaries of the data set, determines which file objects are included in the current data set version, uses a hash algorithm to calculate the hash value of each file object in the data set and records the hash value. Before the data set is scheduled to execute a training task, it traverses all file objects in the data set, uses the same hash algorithm to recalculate the hash value of each file object, and compares the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0027] As the information interaction and execution process between the modules in the above-mentioned system are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0028] Likewise, the benefits of the system of the present invention are: Through version control and intelligent data processing mechanisms, efficient tracking and management of training data set versions are achieved. This not only solves many problems faced in data set management, such as version confusion, data loss, and tampering risks, but also provides strong support for the training and iteration of machine learning models.

[0029] The use of multi-level version control strategies and refined management can accurately capture every change in the data set and generate a unique version identifier, ensuring that the version of the data set is clear and traceable, while reducing manual intervention and error rates, greatly improving the accuracy and efficiency of data set management. Users can easily query, compare and roll back to any historical version, thereby accelerating the debugging and optimization process of machine learning models.

[0030] The process automatically manages and monitors the entire process of dataset creation, modification, publication and use, which improves the efficiency of dataset management, ensures data integrity and security, and promotes team collaboration and data sharing.

[0031] It should be noted that not all steps and modules in the above-mentioned processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0032] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A training dataset version management method, Its characteristics are include: Step 1: Create a data set for model training and generate a unique identifier for the data set. The unique identifier uses the universal unique identifier UUID. Step 2: Manage dataset versions: Step 21: Create a dataset version. Step 22: Use the formula V = D + T + S to generate a dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp; Step 23: Create a static snapshot of the selected dataset according to the dataset version. The metadata saved in the static snapshot includes the dataset version identifier V, the creation time, the list of file objects included in the dataset, the object index, and the version identifier of each file object; Step 24: Based on the static snapshot, check whether the file object in the data set that will undergo content changes is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content changes, and update the object index of the metadata of the static snapshot before performing the data set change operation. Step 25: Verify the integrity of the dataset: According to the dataset version, clarify the boundaries of the dataset, determine which file objects are included in the current dataset version, use the hash algorithm to calculate the hash value of each file object in the dataset and record the hash value. Before the dataset is scheduled to execute the training task, traverse all file objects in the dataset, use the same hash algorithm to recalculate the hash value of each file object, and compare the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

2. A training data set version management method according to claim 1, Its characteristics are Step 1 also classifies and manages the file objects in the dataset, including: identifying the file object type according to the extension type of the file contained in the current dataset, the file object type includes pictures, videos, audio and text, and saving the file object type and the save path of the file object in the dataset as metadata in the database.

3. A training data set version management method according to claim 1, Its characteristics are There are two ways to establish a dataset version in step 21. One way is to capture a static snapshot of the data based on the original file object content in the dataset, establish and publish the dataset version; the other way is to change the file object content based on the file object content of the existing dataset version based on the published dataset version, generate a new dataset version and publish it.

4. A training data set version management method according to claim 1, Its characteristics are Step 25 verifies the integrity of the data set, including: Introduce a data signature mechanism, use a hash value record file to record the hash value of each file object, use a private key to encrypt the hash value record file, and generate a digital signature. Before the data set is scheduled to execute a training task, the digital signature is decrypted using the corresponding public key to obtain the original hash value record file, and all file objects in the data set are traversed. The hash value of each file object is recalculated using the same hash algorithm, and the newly calculated hash value is compared with the hash value recorded in the hash value record file. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

5. A training dataset version management system, Its characteristics are Including data set management module, version management module and integrity verification module, The dataset management module establishes a dataset for model training and generates a unique identifier for the dataset. The unique identifier uses the universal unique identifier UUID. The version management module manages dataset versions: Create a dataset version, The formula V = D + T + S is used to generate the dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp. According to the dataset version, a static snapshot of the selected dataset is created. The metadata saved in the static snapshot includes the dataset version identifier V, the creation time, the list of file objects contained in the dataset, the object index, and the version identifier of each file object; According to the static snapshot, check whether the file object whose content will be changed in the data set is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content is changed, update the object index of the metadata of the static snapshot, and then perform the data set change operation; The integrity verification module verifies the integrity of the data set: according to the data set version, it clarifies the boundaries of the data set, determines which file objects are included in the current data set version, uses a hash algorithm to calculate the hash value of each file object in the data set and records the hash value. Before the data set is scheduled to execute a training task, it traverses all file objects in the data set, uses the same hash algorithm to recalculate the hash value of each file object, and compares the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

6. A training data set version management system according to claim 5, Its characteristics are The dataset management module also classifies and manages the file objects in the dataset, including: identifying the file object type based on the extension type of the file contained in the current dataset, the file object type includes pictures, videos, audio and text, and saving the file object type and the save path of the file object in the dataset as metadata in the database.

7. A training data set version management system according to claim 5, Its characteristics are The version management module establishes a dataset version in two ways. One way is to capture a static snapshot of the data based on the original file object content in the dataset, establish a dataset version and publish it. The other way is to change the file object content based on the file object content of the existing dataset version, generate a new dataset version and publish it.

8. A training data set version management system according to claim 5, Its characteristics are The integrity check module verifies the integrity of the data set, including: Introduce a data signature mechanism, use a hash value record file to record the hash value of each file object, use a private key to encrypt the hash value record file, and generate a digital signature. Before the data set is scheduled to execute a training task, the digital signature is decrypted using the corresponding public key to obtain the original hash value record file, and all file objects in the data set are traversed. The hash value of each file object is recalculated using the same hash algorithm, and the newly calculated hash value is compared with the hash value recorded in the hash value record file. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

Citation Information

Patent Citations

  • Distributed collaborative design data version control method and device

    CN110674114A

  • Method, system and equipment for managing platform data set and storage medium

    CN114138719A

  • Data set version management method and device based on Git, equipment and storage medium

    CN114911768A

  • Versioned relational data set management

    CN116569161A

  • Machine learning method and device based on lake and cabin integration

    CN117787432A

Cited By

  • Data set version management method and device based on computing power of intelligent computing center

    CN119988351A

  • Machine learning training sample-oriented efficient data version management platform

    CN120104595A

  • Efficient data version management platform for machine learning training samples

    CN120104595B