Efficient data version management platform for machine learning training samples

Abstract

The present application relates to the field of data management, and more particularly to a kind of efficient data version management platform for machine learning training sample, the present application sets data warehouse module, feature analysis module, label arrangement module and retrieval processing module, the data features of corresponding each version data set are extracted and analyzed sample data set by feature analysis module, the difference factors between data features are extracted, version change representation coefficient is calculated correspondingly, to represent the change of data itself after data set is processed multiple times, version change label is set to sample data set by label arrangement module, subsequent retrieval in data warehouse module according to the adaptability of version change label, to match the retrieval demand information of user end, improve the analysis efficiency when facing mass data set analysis, match each version of data set that can meet the demand of user end, improve data set utilization rate.

Description

Technical Field

[0001] This invention relates to the field of data management, and more particularly to an efficient data version management platform for machine learning training samples. Background Technology

[0002] In the field of machine learning, with the continuous expansion of data scale and the increasing complexity of data sources, traditional data management methods are struggling to meet the needs of modern machine learning projects. First, data version control has become a major challenge. Machine learning projects often require multiple rounds of processing, including cleaning, labeling, and augmentation, resulting in numerous versions. The lack of an effective version management mechanism makes data traceability difficult, affecting the reproducibility of model results. Second, data sharing and reuse are inefficient. Data silos are prevalent between different teams or projects, leading to a significant amount of valuable data not being fully utilized and resulting in resource waste. Furthermore, data synchronization and consistency issues are becoming increasingly prominent. In a distributed environment, ensuring real-time synchronization and consistency of data between different machine learning task nodes becomes extremely challenging.

[0003] For example, Chinese Patent Publication No. CN118964335A discloses a training dataset version management method and system, relating to the field of data management technology; it includes: Step 1: Establishing a dataset for model training and generating a unique identifier for the dataset; Step 2: Managing dataset versions: Step 21: Establishing a dataset version; Step 22: Generating a dataset version identifier using the formula V=D+T+S; Step 23: Creating a static snapshot of the selected dataset based on the dataset version; Step 24: Checking whether file objects in the dataset whose content will change are included in the static snapshot based on the static snapshot. If so, copying the current version of the file object to the corresponding storage of the static snapshot, updating the object index of the metadata of the static snapshot, and then performing the dataset change operation; Step 25: Verifying the integrity of the dataset; This invention reduces the complexity of data management and ensures the quality and reliability of model training.

[0004] However, the following problems still exist in the existing technology:

[0005] The utilization efficiency of several versions of the dataset after multiple rounds of processing is low, and when the data volume is large, the computational power consumed by searching and sharing the dataset is high, resulting in low efficiency. Summary of the Invention

[0006] To address this, the present invention provides an efficient data version management platform for machine learning training samples, which overcomes the problems of low utilization efficiency of multiple versions of datasets after multiple rounds of processing in the prior art, and high computing power consumption and low efficiency when searching and sharing datasets when the data volume is large.

[0007] To achieve the above objectives, the present invention provides an efficient data version management platform for machine learning training samples, comprising:

[0008] The data warehouse module is used to store several sample datasets and corresponding version datasets of each sample dataset after processing.

[0009] The feature parsing module, which is connected to the data storage module, is used to parse the data features of the sample dataset and the corresponding version datasets, extract the difference factors between the data features, and calculate the version variation characterization coefficient based on the difference factors.

[0010] The label placement module is connected to the data storage module and the feature parsing module respectively, and is used to set version variation labels for each sample dataset based on the version variation characterization coefficient.

[0011] A retrieval processing module, connected to the data storage module and the tag placement module, is used to obtain retrieval requirement information and perform retrieval in the data storage module based on version variation tags, including...

[0012] Only the data from the sample dataset is extracted and matched with the search requirement information. Based on the matching results, the sample dataset and version dataset to be extracted are called.

[0013] Alternatively, clustering can be performed based on the differences in data features between the sample dataset and the corresponding version dataset to obtain several cluster sets. Each cluster set can be filtered and parsed, and the cluster set can be called based on the filtering and parsing results.

[0014] The filtering and parsing process includes matching data from any dataset within the cluster with the information required for retrieval.

[0015] Furthermore, the feature parsing module is used to parse the data features of the sample dataset and the corresponding versions of the dataset, including:

[0016] Extract data features from each image data in the sample dataset and the version dataset, including resolution, area of ​​contour features, saturation, contrast, brightness, and signal-to-noise ratio;

[0017] Calculate the differences in data features among the image data in the sample dataset and the version dataset, including the average difference ratio of resolution, the average difference ratio of area, the average difference ratio of contrast, the average difference ratio of brightness, and the average difference ratio of signal-to-noise ratio.

[0018] Furthermore, the feature parsing module calculates the version variation characterization coefficient for the sample dataset based on the difference factors, including:

[0019] This is used to sum the weighted differences to obtain the version variation characterization coefficient.

[0020] Furthermore, the arrangement module is used to set version variation labels for each sample dataset based on the version variation characterization coefficient, including:

[0021] Used to determine the version variation characterization coefficients corresponding to each sample dataset;

[0022] If the version variation characterization coefficient is greater than or equal to the preset version variation characterization threshold, then the sample dataset is determined to have a strong variation label.

[0023] If the version variation characterization coefficient is less than the preset version variation characterization threshold, it is determined that the sample dataset is set with a weak variation label.

[0024] Furthermore, the retrieval processing module acquires retrieval request information including,

[0025] Used to obtain retrieval request information sent by the user, including the required reference data samples.

[0026] Furthermore, based on version variation tags, searches are performed in the data warehouse module, including...

[0027] If the sample dataset has a weakly variable label, then only the data in the sample dataset is extracted and matched with the search requirement information, and the sample dataset and version dataset to be extracted are called according to the matching result.

[0028] If the sample dataset has a strongly variable label, then clustering is performed based on the differences in data features between the sample dataset and the corresponding version dataset to obtain several cluster sets. Each cluster set is then filtered and analyzed, and the cluster set is called based on the filtering and analysis results.

[0029] Furthermore, the retrieval processing module extracts data from the sample dataset and matches it with the retrieval requirement information, including:

[0030] Used to extract data from the sample dataset;

[0031] Used to calculate the goodness of fit between the data and the reference data sample.

[0032] Furthermore, the retrieval processing module, based on the matching results, calls the sample dataset and version dataset to be extracted, including...

[0033] If the goodness of fit of the sample dataset is greater than a predetermined goodness of fit threshold, it is determined that the sample dataset and several versions of the sample dataset need to be called.

[0034] Furthermore, the retrieval processing module is used to cluster the differences in data features between the sample dataset and the corresponding version dataset, resulting in several cluster sets, including...

[0035] Cluster the sample dataset and the corresponding version dataset according to the clustering criteria;

[0036] The clustering condition is that the difference factors between any datasets in the cluster set are all less than a predetermined clustering factor threshold.

[0037] Furthermore, the cluster set is invoked based on the filtered parsing results, including:

[0038] It is used to retrieve any data from a cluster, compare the retrieved data with the reference sample data, and solve for the goodness of fit;

[0039] If the fit is greater than a predetermined fit threshold, it is determined that the cluster set needs to be invoked.

[0040] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention sets up a data storage module, a feature parsing module, a label placement module, and a retrieval processing module. The feature parsing module extracts data features from the parsed sample dataset and corresponding versions of the dataset, extracts the difference factors between data features, and calculates the version variation characterization coefficient to characterize the variation of the dataset itself after multiple processing. The label placement module sets version variation labels on the sample dataset, and subsequently, based on the version variation labels, it adaptively retrieves data in the data storage module to match the user's retrieval needs, improving the analysis efficiency when analyzing massive datasets, matching datasets of various versions that can meet the user's needs, and improving dataset utilization.

[0041] In particular, the feature parsing module of this invention extracts the difference factors between data features and calculates the version variation characterization coefficient. In practice, sample datasets undergo multiple rounds of processing to meet training requirements, resulting in a large number of version datasets. Due to differences in processing methods, the differences between these sample datasets and the original dataset in terms of data feature dimensions are relatively discrete. In some cases, the differences are small, and the original dataset and its derived version datasets tend to meet the same training requirements. Therefore, several difference factors are identified, and the version variation characterization coefficient is calculated. The difference factors are the basic features of the data. Basic features can be obtained quickly with low computational cost and can characterize the differences between datasets. Therefore, calculating the version variation characterization coefficient provides data support for setting version variation labels for the version datasets, which facilitates the subsequent adaptive retrieval of the data warehouse module. Under the premise of ensuring reliability, it improves the retrieval efficiency of massive data and increases the utilization rate of the version datasets.

[0042] In particular, the version variation tag retrieval module of this invention retrieves data from the data warehouse module. In practice, after the user uploads the retrieval request information, a deep search and deep matching are required for the data warehouse module. However, the data volume is massive. For sample datasets with weak version variation tags, the version variation is small. Therefore, when performing deep matching, it is considered to extract data from the sample dataset and match it with the retrieval request information, instead of matching the version dataset derived from the sample dataset. This reduces the amount of data processing while ensuring reliability. For strong version variation tags, clustering is prioritized. Each cluster is filtered and parsed, and data is extracted from the clusters and matched with the retrieval request information. Then, different retrieval methods are adaptively used to improve retrieval efficiency under massive data while ensuring reliability and increasing the utilization rate of the version dataset. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the structure of an efficient data version management platform for machine learning training samples according to an embodiment of the present invention.

[0044] Figure 2 A logical block diagram for setting version variation labels for each sample dataset in an embodiment of the present invention;

[0045] Figure 3 This is a logic block diagram of a data warehouse module for searching based on version variation tags according to an embodiment of the present invention.

[0046] Figure 4 In this embodiment of the invention, the sample dataset and version dataset to be extracted are called based on the matching results. Detailed Implementation

[0047] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0048] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0049] It should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0050] Please see Figures 1 to 4 As shown, Figure 1 This is a schematic diagram of the structure of an efficient data version management platform for machine learning training samples according to an embodiment of the present invention. Figure 2 This is a block diagram illustrating the logic for setting version variation labels for each sample dataset in an embodiment of the present invention. Figure 3 This is a logic block diagram illustrating the retrieval process in the data warehouse module based on version variation tags, according to an embodiment of the present invention. Figure 4 To retrieve the required sample dataset and version dataset based on the matching results in this embodiment of the invention, this embodiment provides a big data-based precise advertising delivery system, which includes:

[0051] The data warehouse module is used to store several sample datasets and corresponding version datasets of each sample dataset after processing.

[0052] The feature parsing module, which is connected to the data storage module, is used to parse the data features of the sample dataset and the corresponding version datasets, extract the difference factors between the data features, and calculate the version variation characterization coefficient based on the difference factors.

[0053] The label placement module is connected to the data storage module and the feature parsing module respectively, and is used to set version variation labels for each sample dataset based on the version variation characterization coefficient.

[0054] A retrieval processing module, connected to the data storage module and the tag placement module, is used to obtain retrieval requirement information and perform retrieval in the data storage module based on version variation tags, including...

[0055] Only the data from the sample dataset is extracted and matched with the search requirement information. Based on the matching results, the sample dataset and version dataset to be extracted are called.

[0056] Alternatively, clustering can be performed based on the differences in data features between the sample dataset and the corresponding version dataset to obtain several cluster sets. Each cluster set can be filtered and parsed, and the cluster set can be called based on the filtering and parsing results.

[0057] The filtering and parsing process includes matching data from any dataset within the cluster with the information required for retrieval.

[0058] Specifically, there are no restrictions on the specific structure of the data storage module. It can be a virtual database, or other forms, as long as it can store data. This will not be elaborated further.

[0059] Specifically, there are no restrictions on the specific structure of the feature parsing module, the tag placement module, and the retrieval processing module. They can all be composed of logical components or combinations of logical components. Logical components include field-programmable processors, computers, or microprocessors in computers.

[0060] Understandably, after a single processing of a dataset, a new dataset is formed, which is called a version dataset. After several processings, several version datasets can be formed. There is no limitation on the processing method. In practice, the dataset may be subjected to, for example, color dithering (adjusting the brightness, contrast, and saturation of the image), Gaussian modules (blurring the image to enhance the model's robustness to noise), noise addition (adding Gaussian noise or salt and pepper noise to the image), and affine transformations (translation, rotation, scaling, and cropping). After processing, the data characteristics of the image will be changed to meet the corresponding training requirements. Of course, there are other processing methods as well, which will not be elaborated here.

[0061] Specifically, the feature parsing module is used to parse the data features of the sample dataset and the corresponding versions of the dataset, including:

[0062] Extract data features from each image data in the sample dataset and the version dataset, including resolution, area of ​​contour features, saturation, contrast, brightness, and signal-to-noise ratio;

[0063] Calculate the differences in data features among the image data in the sample dataset and the version dataset, including the average difference ratio of resolution, the average difference ratio of area, the average difference ratio of contrast, the average difference ratio of brightness, and the average difference ratio of signal-to-noise ratio.

[0064] It is understandable that the difference ratio is the ratio of the difference between two values ​​to the mean of the two values.

[0065] It is understandable that the sample dataset needs to be compared with multiple versions of the dataset. The resolution difference ratio, area difference ratio, contrast difference ratio, brightness difference ratio, and signal-to-noise ratio difference ratio can be calculated one by one. Then, the average resolution difference ratio, average area difference ratio, average contrast difference ratio, average brightness difference ratio, and average signal-to-noise ratio difference ratio can be calculated. This will not be elaborated further.

[0066] Specifically, the feature parsing module calculates the version variation characterization coefficient for the sample dataset based on the difference factors, including:

[0067] This is used to sum the weighted differences to obtain the version variation characterization coefficient.

[0068] In practice, the weights for the average difference ratio of resolution, average difference ratio of area, average difference ratio of contrast, average difference ratio of brightness, and average difference ratio of signal-to-noise ratio are 0.25, 0.15, 0.15, 0.15, and 0.3, respectively.

[0069] This invention's feature parsing module extracts the difference factors between data features and calculates the version variation characterization coefficient. In practice, sample datasets undergo multiple rounds of processing to meet training requirements, resulting in numerous version datasets. Due to differences in processing methods, the differences between these sample datasets and the original dataset in terms of data feature dimensions are relatively discrete. In some cases, the differences are small, and the original dataset and its derived version datasets tend to meet the same training requirements. Therefore, several difference factors are identified, and the version variation characterization coefficient is calculated. The difference factors are the basic features of the data. Basic features can be obtained quickly with low computational cost and can characterize the differences between datasets. Therefore, calculating the version variation characterization coefficient provides data support for setting version variation labels for the version datasets, facilitating the subsequent adaptive retrieval of the data warehouse module. This improves the retrieval efficiency of massive amounts of data and increases the utilization rate of the version datasets while ensuring reliability.

[0070] Specifically, the arrangement module is used to set version variation labels for each sample dataset based on the version variation characterization coefficient, including:

[0071] Used to determine the version variation characterization coefficients corresponding to each sample dataset;

[0072] If the version variation characterization coefficient is greater than or equal to the preset version variation characterization threshold, then the sample dataset is determined to have a strong variation label.

[0073] If the version variation characterization coefficient is less than the preset version variation characterization threshold, it is determined that the sample dataset is set with a weak variation label.

[0074] This invention uses version variation tags in the retrieval processing module to search the data warehouse module. In practice, after a user uploads their search request information, a deep search and deep matching process is required for the data warehouse module. However, the data volume is massive. For sample datasets with weak version variation tags, the version variation is relatively small. Therefore, during deep matching, the system considers extracting data from the sample dataset and matching it with the search request information, instead of matching the version dataset derived from the sample dataset. This reduces the amount of data processing while ensuring reliability. For strong version variation tags, clustering is prioritized. Each cluster is then filtered and analyzed, and data is extracted from the clusters and matched with the search request information. Furthermore, different search methods are adaptively used to improve search efficiency under massive data volumes while ensuring reliability and increasing the utilization rate of the version dataset.

[0075] Specifically, in practice, the threshold for version variation is selected within the range [0.25, 0.3].

[0076] Specifically, the retrieval processing module obtains retrieval request information including,

[0077] Used to obtain retrieval request information sent by the user, including the required reference data samples.

[0078] It is understood that the reference data sample is image data, and those skilled in the art can select image data that meets the training requirements and use the image data as the reference data sample.

[0079] Specifically, retrieval is performed in the data warehouse module based on version variation tags, including:

[0080] If the sample dataset has a weakly variable label, then only the data in the sample dataset is extracted and matched with the search requirement information, and the sample dataset and version dataset to be extracted are called according to the matching result.

[0081] If the sample dataset has a strongly variable label, then clustering is performed based on the differences in data features between the sample dataset and the corresponding version dataset to obtain several cluster sets. Each cluster set is then filtered and analyzed, and the cluster set is called based on the filtering and analysis results.

[0082] Specifically, the retrieval processing module extracts data from the sample dataset and matches it with the retrieval requirement information, including:

[0083] Used to extract data from the sample dataset;

[0084] Used to calculate the goodness of fit between the data and the reference data sample.

[0085] Specifically, the sample dataset consists of image data. The goodness of fit is calculated by solving the goodness of fit between image data. The purpose is to characterize the similarity between images. For example, the goodness of fit can be obtained by calculating the structural similarity index corresponding to the image data. Of course, other methods can also be used, which will not be elaborated here.

[0086] Specifically, the retrieval processing module, based on the matching results, calls the sample dataset and version dataset to be extracted, including:

[0087] If the goodness of fit of the sample dataset is greater than a predetermined goodness of fit threshold, it is determined that the sample dataset and several versions of the sample dataset need to be called.

[0088] Specifically, the goodness-of-fit threshold is pre-calculated, the dataset used in the training process is recorded, the mean goodness-of-fit between the data in the dataset is determined, the mean goodness-of-fit of the corresponding mean goodness-of-fit in several training processes is calculated, and the goodness-of-fit threshold is set as the product of the mean goodness-of-fit and the accuracy coefficient, with the accuracy coefficient selected in the interval [0.85, 0.95].

[0089] Specifically, the retrieval processing module is used to cluster the differences in data features between the sample dataset and the corresponding version dataset, resulting in several cluster sets, including...

[0090] Cluster the sample dataset and the corresponding version dataset according to the clustering criteria;

[0091] The clustering condition is that the difference factors between any datasets in the cluster set are all less than a predetermined clustering factor threshold.

[0092] The clustering factor threshold is set based on the version variation characterization threshold, and is set to 0.85 times the version variation characterization threshold.

[0093] Specifically, calling the cluster set based on the filtering and parsing results includes:

[0094] It is used to retrieve any data from a cluster, compare the retrieved data with the reference sample data, and solve for the goodness of fit;

[0095] If the fit is greater than a predetermined fit threshold, it is determined that the cluster set needs to be invoked.

[0096] Understandably, after retrieving data, the index of the retrieved dataset can be sent to the user's end so that the user can find the corresponding dataset and retrieve it.

[0097] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A high-efficiency data version management platform for machine learning training samples, characterized in that, include: The data storage module is used to store several sample datasets and corresponding version datasets of each sample dataset after processing. The samples in the sample datasets are image data. The feature parsing module, which is connected to the data storage module, is used to parse the data features of the sample dataset and the corresponding version datasets, extract the difference factors between the data features, and calculate the version variation characterization coefficient based on the difference factors. The label placement module is connected to the data storage module and the feature parsing module respectively, and is used to set version variation labels for each sample dataset based on the version variation characterization coefficient. A retrieval processing module, connected to the data storage module and the tag placement module, is used to obtain retrieval requirement information and perform retrieval in the data storage module based on version variation tags, including... Only the data from the sample dataset is extracted and matched with the search requirement information. Based on the matching results, the sample dataset and version dataset to be extracted are called. Alternatively, clustering can be performed based on the differences in data characteristics between the sample dataset and the corresponding version dataset to obtain several cluster sets. Each cluster set can be filtered and parsed, and the cluster set can be called based on the filtering and parsing results. The filtering and parsing process includes matching data from any dataset in the cluster with the retrieval requirements. Retrieval based on version variation tags in the data warehouse module includes, If the sample dataset has a weakly variable label, then only the data in the sample dataset is extracted and matched with the search requirement information, and the sample dataset and version dataset to be extracted are called according to the matching result. If the sample dataset has a strongly variable label, then clustering is performed based on the differences in data features between the sample dataset and the corresponding version dataset to obtain several cluster sets. Each cluster set is then filtered and analyzed, and the cluster set is called based on the filtering and analysis results.

2. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that, The feature parsing module is used to parse the data features of the sample dataset and the corresponding versions of the dataset, including: Extract data features from each image data in the sample dataset and the version dataset, including resolution, area of ​​contour features, saturation, contrast, brightness, and signal-to-noise ratio; Calculate the differences in data features among the image data in the sample dataset and the version dataset, including the average difference ratio of resolution, the average difference ratio of area, the average difference ratio of contrast, the average difference ratio of brightness, and the average difference ratio of signal-to-noise ratio.

3. The efficient data version management platform for machine learning training samples according to claim 2, characterized in that, The feature parsing module calculates the version variation characterization coefficients for the sample dataset based on the difference factors, including: This is used to sum the weighted differences to obtain the version variation characterization coefficient.

4. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that, The arrangement module is used to set version variation labels for each sample dataset based on the version variation characterization coefficients, including: Used to determine the version variation characterization coefficients corresponding to each sample dataset; If the version variation characterization coefficient is greater than or equal to the preset version variation characterization threshold, then the sample dataset is determined to have a strong variation label. If the version variation characterization coefficient is less than the preset version variation characterization threshold, it is determined that the sample dataset is set with a weak variation label.

5. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that, The retrieval processing module obtains retrieval request information including... Used to obtain retrieval request information sent by the user, including the required reference data samples.

6. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that, The retrieval processing module extracts data from the sample dataset and matches it with the retrieval requirement information, including... Used to extract data from the sample dataset; This is used to calculate the goodness of fit between the data in the sample dataset and the reference data sample.

7. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that, The retrieval processing module, based on the matching results, calls the sample dataset and version dataset to be extracted, including... If the goodness of fit of the sample dataset is greater than a predetermined goodness of fit threshold, it is determined that the sample dataset and several versions of the sample dataset need to be called.

8. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that, The retrieval processing module is used to cluster the differences in data features between the sample dataset and the corresponding version dataset, resulting in several cluster sets, including: Cluster the sample dataset and the corresponding version dataset according to the clustering criteria; The clustering condition is that the difference factors between any datasets in the cluster set are all less than a predetermined clustering factor threshold.

9. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that, The cluster set called based on the filtering and parsing results includes: It is used to retrieve any data from a cluster, compare the retrieved data with the reference sample data, and solve for the goodness of fit; If the fit is greater than a predetermined fit threshold, it is determined that the cluster set needs to be invoked.

Images