A training data set version management method and system

Abstract

The present invention discloses a training data set version management method and system, which relates to the technical field of data management; the method and system comprise: step 1: establishing a data set for model training, generating a unique identifier for the data set, step 2: managing the data set version; step 21: establishing the data set version, step 22: generating a data set version identifier by using the formula V=D+T+S, step 23: creating a static snapshot of the selected data set according to the data set version; step 24: checking whether a file object in the data set whose content will change is contained in the static snapshot according to the static snapshot, if so, copying the current version of the file object to the corresponding storage of the static snapshot, updating the object index of the metadata of the static snapshot, and then performing the data set change operation, step 25: verifying the integrity of the data set; the present invention reduces the complexity of data management and ensures the quality and reliability of model training.

Description

Technical Field

[0001] The invention discloses a training data set version management method and system, and relates to the technical field of data management. Background Art

[0002] With the rapid development of artificial intelligence technology and the advent of the big data era, the scale of training data sets, as the cornerstone of machine learning models, has become increasingly large and complex. However, there are currently problems such as lax version control of training data sets, missing version information, low update efficiency, and difficulty in backtracking. These problems not only increase the complexity of data management, but may also lead to frequent data consistency problems, which in turn affects the results and reliability of model training. At the same time, due to the lack of an effective version control mechanism, data differences between different versions are difficult to accurately track, and historical version data is often difficult to restore. More seriously, during the process of data transmission, storage, and processing, the integrity of the data may be damaged, such as data tampering, loss, or damage, which also directly affects the results and reliability of model training. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides a training dataset version management method and system, which significantly reduces the complexity of data management, improves the efficiency and accuracy of version management, and thus ensures the quality and reliability of model training.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a training data set version management method, comprising:

[0006] Step 1: Create a data set for model training and generate a unique identifier for the data set. The unique identifier uses the universal unique identifier UUID.

[0007] Step 2: Manage dataset versions:

[0008] Step 21: Create a dataset version.

[0009] Step 22: Use the formula V = D + T + S to generate a dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp;

[0010] Step 23: Create a static snapshot of the selected dataset according to the dataset version. The metadata saved in the static snapshot includes the dataset version identifier V, the creation time, the list of file objects included in the dataset, the object index, and the version identifier of each file object;

[0011] Step 24: Based on the static snapshot, check whether the file object in the data set that will undergo content changes is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content changes, and update the object index of the metadata of the static snapshot before performing the data set change operation.

[0012] Step 25: Verify the integrity of the dataset: According to the dataset version, clarify the boundaries of the dataset, determine which file objects are included in the current dataset version, use the hash algorithm to calculate the hash value of each file object in the dataset and record the hash value. Before the dataset is scheduled to execute the training task, traverse all file objects in the dataset, use the same hash algorithm to recalculate the hash value of each file object, and compare the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0013] Furthermore, in step 1 of the training dataset version management method, the file objects in the dataset are also classified and managed, including: identifying the file object type according to the extension type of the file contained in the current dataset, the file object type includes pictures, videos, audio and text, and saving the file object type and the save path of the file object in the dataset as metadata in the database.

[0014] Furthermore, there are two ways to establish a dataset version in step 21 of the training dataset version management method. One way is to capture a static snapshot of the data based on the original file object content in the dataset, establish and publish the dataset version; the other way is to change the file object content based on the file object content of the existing dataset version based on the published dataset version, generate a new dataset version and publish it.

[0015] Furthermore, in step 25 of the training dataset version management method, verifying the dataset integrity includes:

[0016] Introduce a data signature mechanism, use a hash value record file to record the hash value of each file object, use a private key to encrypt the hash value record file, and generate a digital signature.

[0017] Before the data set is scheduled to execute a training task, the digital signature is decrypted using the corresponding public key to obtain the original hash value record file, and all file objects in the data set are traversed. The hash value of each file object is recalculated using the same hash algorithm, and the newly calculated hash value is compared with the hash value recorded in the hash value record file. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0018] The present invention also provides a training data set version management system, including a data set management module, a version management module and an integrity verification module.

[0019] The dataset management module establishes a dataset for model training and generates a unique identifier for the dataset. The unique identifier uses the universal unique identifier UUID.

[0020] The version management module manages dataset versions:

[0021] Create a dataset version,

[0022] The formula V = D + T + S is used to generate the dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp.

[0023] According to the dataset version, a static snapshot of the selected dataset is created. The metadata saved in the static snapshot includes the dataset version identifier V, the creation time, the list of file objects contained in the dataset, the object index, and the version identifier of each file object;

[0024] According to the static snapshot, check whether the file object whose content will be changed in the data set is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content is changed, update the object index of the metadata of the static snapshot, and then perform the data set change operation;

[0025] The integrity verification module verifies the integrity of the data set: according to the data set version, it clarifies the boundaries of the data set, determines which file objects are included in the current data set version, uses a hash algorithm to calculate the hash value of each file object in the data set and records the hash value. Before the data set is scheduled to execute a training task, it traverses all file objects in the data set, uses the same hash algorithm to recalculate the hash value of each file object, and compares the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0026] Furthermore, the data set management module of the training data set version management system also classifies and manages the file objects in the data set, including: identifying the file object type according to the extension type of the file contained in the current data set, the file object type includes pictures, videos, audio and text, and saving the file object type and the save path of the file object in the data set as metadata in the database.

[0027] Furthermore, the version management module of the training dataset version management system establishes a dataset version in two ways. One way is to capture a static snapshot of the data based on the original file object content in the dataset, establish and publish the dataset version; the other way is to change the file object content based on the file object content of the existing dataset version based on the published dataset version, generate a new dataset version and publish it.

[0028] Furthermore, the integrity verification module of the training dataset version management system verifies the integrity of the dataset, including:

[0029] Introduce a data signature mechanism, use a hash value record file to record the hash value of each file object, use a private key to encrypt the hash value record file, and generate a digital signature.

[0030] Before the data set is scheduled to execute a training task, the digital signature is decrypted using the corresponding public key to obtain the original hash value record file, and all file objects in the data set are traversed. The hash value of each file object is recalculated using the same hash algorithm, and the newly calculated hash value is compared with the hash value recorded in the hash value record file. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0031] The benefits of the method of the present invention are:

[0032] Through version control and intelligent data processing mechanisms, efficient tracking and management of training data set versions are achieved. This not only solves many problems faced in data set management, such as version confusion, data loss, and tampering risks, but also provides strong support for the training and iteration of machine learning models.

[0033] The use of multi-level version control strategies and refined management can accurately capture every change in the data set and generate a unique version identifier, ensuring that the version of the data set is clear and traceable, while reducing manual intervention and error rates, greatly improving the accuracy and efficiency of data set management. Users can easily query, compare and roll back to any historical version, thereby accelerating the debugging and optimization process of machine learning models.

[0034] The process automatically manages and monitors the entire process of dataset creation, modification, publication and use, which improves the efficiency of dataset management, ensures data integrity and security, and promotes team collaboration and data sharing. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0036] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0037] Embodiment 1: The present invention provides a training data set version management method, comprising:

[0038] Step 1: Create a data set for model training and generate a unique identifier for the data set. The unique identifier can be a 32-bit universal unique identifier UUID.

[0039] Wherein, step 1 also classifies and manages the file objects in the dataset, including: identifying the file object type according to the extension type of the file contained in the current dataset, the file object type includes pictures, videos, audio and text, etc., and saving the file object type and the save path of the file object in the dataset as metadata in the database.

[0040] Step 2: Manage dataset versions:

[0041] Step 21: Establish a dataset version. There are two ways to establish a dataset version. One way is to capture a static snapshot of the data based on the original file object content in the dataset, establish a dataset version and publish it; the other way is to change the file object content based on the existing dataset version, generate a new dataset version and publish it.

[0042] Step 22: Use the formula V = D + T + S to generate a dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp.

[0043] Step 23: Create a static snapshot of the selected dataset based on the dataset version. The metadata saved in the static snapshot includes the dataset version identifier V, the creation time, the list of file objects contained in the dataset, the object index, and the version identifier of each file object. Each dataset version is an accurate snapshot of the original dataset at a specific point in time. Therefore, the dataset version not only captures the static snapshot of the data, but also contains the complete history and evolution trajectory of the dataset. The dataset version identifier of the present invention can ensure the high traceability of the dataset, so that any questions or backtracking requirements about the dataset can be quickly queried, responded to and resolved.

[0044] Step 24: In the management process of static snapshots, the logic of copy-on-write is adopted. When the content of the file in the data set changes, according to the static snapshot, check whether the file object in the data set whose content will change is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content changes, and update the object index of the metadata of the static snapshot before performing the data set change operation. This process minimizes the storage space occupied, and there is no need to copy the entire data set for each static snapshot. Static snapshots save metadata, not the data files themselves. A large number of data files are still actually stored in the initial object storage or file storage. Therefore, only when the data file of the published data set version changes, the file is copied to the storage of the static snapshot for backup to ensure the integrity and consistency of the data, while reducing costs and avoiding unnecessary resource consumption.

[0045] Step 25: Verify the integrity of the data set: According to the data set version, clarify the boundaries of the data set, determine which file objects are included in the current data set version, use a hash algorithm to calculate the hash value of each file object in the data set and record the hash value. Hash algorithms such as MD5, SHA-1, and preferably SHA-256, etc. Before the data set is scheduled to execute a training task, traverse all file objects in the data set, use the same hash algorithm to recalculate the hash value of each file object, and compare the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0046] Verifying the integrity of the data set in step 25 may further include:

[0047] A data signature mechanism is introduced, and the hash value record file is used to record the hash value of each file object. The hash value record file can be integrated into the static snapshot metadata of the dataset version. The hash value record file is encrypted with a private key to generate a digital signature.

[0048] Before the data set is scheduled to execute a training task, the digital signature is decrypted using the corresponding public key to obtain the original hash value record file, and all file objects in the data set are traversed. The hash value of each file object is recalculated using the same hash algorithm, and the newly calculated hash value is compared with the hash value recorded in the hash value record file. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0049] Through this series of technical measures, not only can the integrity of the data set be strictly protected throughout the entire process of creation, transmission and use, but potential security threats can also be discovered and responded to in a timely manner, providing higher reliability and security guarantees for the machine learning training process, and helping to improve the overall quality and efficiency of data science work.

[0050] Embodiment 2: The present invention also provides a training data set version management system, including a data set management module, a version management module and an integrity verification module.

[0051] The dataset management module establishes a dataset for model training and generates a unique identifier for the dataset. The unique identifier uses the universal unique identifier UUID.

[0052] The version management module manages dataset versions:

[0053] Create a dataset version,

[0054] The formula V = D + T + S is used to generate the dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp.

[0055] According to the dataset version, create a static snapshot of the selected dataset. The metadata of the static snapshot includes the dataset version identifier V, the creation time, the list of file objects contained in the dataset, the object index, and the version identifier of each file object.

[0056] According to the static snapshot, check whether the file object whose content will be changed in the data set is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content is changed, update the object index of the metadata of the static snapshot, and then perform the data set change operation;

[0057] The integrity verification module verifies the integrity of the data set: according to the data set version, it clarifies the boundaries of the data set, determines which file objects are included in the current data set version, uses a hash algorithm to calculate the hash value of each file object in the data set and records the hash value. Before the data set is scheduled to execute a training task, it traverses all file objects in the data set, uses the same hash algorithm to recalculate the hash value of each file object, and compares the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

[0058] As the information interaction and execution process between the modules in the above-mentioned system are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0059] Likewise, the benefits of the system of the present invention are:

[0060] Through version control and intelligent data processing mechanisms, efficient tracking and management of training data set versions are achieved. This not only solves many problems faced in data set management, such as version confusion, data loss, and tampering risks, but also provides strong support for the training and iteration of machine learning models.

[0061] The use of multi-level version control strategies and refined management can accurately capture every change in the data set and generate a unique version identifier, ensuring that the version of the data set is clear and traceable, while reducing manual intervention and error rates, greatly improving the accuracy and efficiency of data set management. Users can easily query, compare and roll back to any historical version, thereby accelerating the debugging and optimization process of machine learning models.

[0062] The process automatically manages and monitors the entire process of dataset creation, modification, publication and use, which improves the efficiency of dataset management, ensures data integrity and security, and promotes team collaboration and data sharing.

[0063] It should be noted that not all steps and modules in the above-mentioned processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0064] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A training data set version management method, characterized by include: Step 1: Create a data set for model training and generate a unique identifier for the data set. The unique identifier uses the universal unique identifier UUID. Step 2: Manage dataset versions: Step 21: Create a dataset version. Step 22: Use the formula V = D + T + S to generate a dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp; Step 23: Create a static snapshot of the selected dataset according to the dataset version. The metadata saved in the static snapshot includes the dataset version identifier V, the creation time, the list of file objects included in the dataset, the object index, and the version identifier of each file object; Step 24: Based on the static snapshot, check whether the file object in the data set that will undergo content changes is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content changes, and update the object index of the metadata of the static snapshot before performing the data set change operation. Step 25: Verify the integrity of the dataset: According to the dataset version, clarify the boundaries of the dataset, determine which file objects are included in the current dataset version, use the hash algorithm to calculate the hash value of each file object in the dataset and record the hash value. Before the dataset is scheduled to execute the training task, traverse all file objects in the dataset, use the same hash algorithm to recalculate the hash value of each file object, and compare the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

2. A training data set version management method according to claim 1, characterized in that In step 1, the file objects in the data set are also classified and managed, including: identifying the file object type according to the extension type of the file contained in the current data set, the file object type includes pictures, videos, audio and text, and saving the file object type and the save path of the file object in the data set as metadata in the database.

3. A training data set version management method according to claim 1, characterized in that There are two ways to establish a dataset version in step 21. One way is to capture a static snapshot of the data based on the original file object content in the dataset, establish a dataset version and publish it. The other way is to change the file object content based on the file object content of the existing dataset version based on the published dataset version, generate a new dataset version and publish it.

4. A training data set version management method according to claim 1, characterized in that In step 25, the integrity of the data set is verified, including: Introduce a data signature mechanism, use a hash value record file to record the hash value of each file object, use a private key to encrypt the hash value record file, and generate a digital signature. Before the data set is scheduled to execute a training task, the digital signature is decrypted using the corresponding public key to obtain the original hash value record file, and all file objects in the data set are traversed. The hash value of each file object is recalculated using the same hash algorithm, and the newly calculated hash value is compared with the hash value recorded in the hash value record file. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

5. A training data set version management system, characterized by Including data set management module, version management module and integrity verification module, The dataset management module establishes a dataset for model training and generates a unique identifier for the dataset. The unique identifier uses the universal unique identifier UUID. The version management module manages dataset versions: Create a dataset version, The formula V = D + T + S is used to generate the dataset version identifier, where V is the dataset version identifier, D is the dataset unique identifier, T is the timestamp, and S is the sequential number used to distinguish multiple dataset versions under the same timestamp. According to the dataset version, create a static snapshot of the selected dataset. The metadata saved in the static snapshot includes the dataset version identifier V, the creation time, the list of file objects contained in the dataset, the object index, and the version identifier of each file object; According to the static snapshot, check whether the file object whose content will be changed in the data set is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot before the content is changed, update the object index of the metadata of the static snapshot, and then perform the data set change operation; The integrity verification module verifies the integrity of the data set: according to the data set version, it clarifies the boundaries of the data set, determines which file objects are included in the current data set version, uses a hash algorithm to calculate the hash value of each file object in the data set and records the hash value. Before the data set is scheduled to execute a training task, it traverses all file objects in the data set, uses the same hash algorithm to recalculate the hash value of each file object, and compares the newly calculated hash value with the recorded hash value. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

6. A training data set version management system according to claim 5, characterized in that The dataset management module also classifies and manages the file objects in the dataset, including: identifying the file object type based on the extension type of the file contained in the current dataset, the file object type includes pictures, videos, audio and text, and saving the file object type and the save path of the file object in the dataset as metadata in the database.

7. A training data set version management system according to claim 5, characterized in that The version management module establishes a dataset version in two ways. One way is to capture a static snapshot of the data based on the original file object content in the dataset, establish a dataset version and publish it. The other way is to change the file object content based on the file object content of the existing dataset version, generate a new dataset version and publish it.

8. A training data set version management system according to claim 5, characterized in that The integrity check module verifies the integrity of the data set, including: Introduce a data signature mechanism, use a hash value record file to record the hash value of each file object, use a private key to encrypt the hash value record file, and generate a digital signature. Before the data set is scheduled to execute a training task, the digital signature is decrypted using the corresponding public key to obtain the original hash value record file, and all file objects in the data set are traversed. The hash value of each file object is recalculated using the same hash algorithm, and the newly calculated hash value is compared with the hash value recorded in the hash value record file. If all hash values ​​match, the verification passes, otherwise an alarm is triggered.

Images