Machine learning training sample-oriented efficient data version management platform

Abstract

The invention relates to the field of data management, in particular to a machine learning training sample-oriented efficient data version management platform, which is provided with a data storage module, a feature analysis module, a label arrangement module and a retrieval processing module, the method comprises the following steps: extracting and analyzing a sample data set and data features corresponding to each version data set through a feature analysis module, extracting difference factors among the data features, correspondingly calculating a version variation characterization coefficient to characterize the variation condition of data per se after the data sets are processed for multiple times, setting a version variation tag for the sample data set through a tag arrangement module, and displaying the version variation tag. Subsequently, retrieval is adaptively performed in the data storage module according to the version variation label so as to match the retrieval demand information of the user side, the analysis efficiency when mass data sets are analyzed is improved, the data sets of various versions which can meet the demands of the user side are matched, and the utilization rate of the data sets is improved.

Description

Technical Field

[0001] The present invention relates to the field of data management, and in particular to an efficient data version management platform for machine learning training samples. Background Art

[0002] In the field of machine learning, with the continuous expansion of data scale and the increasing complexity of data sources, traditional data management methods can no longer meet the needs of modern machine learning projects. First of all, data version control has become a major problem. In machine learning projects, data sets often need to undergo multiple rounds of processing such as cleaning, labeling, and enhancement, resulting in a large number of versions. The lack of an effective version management mechanism makes data traceability difficult, affecting the reproducibility of model results. Secondly, data sharing and reuse are inefficient. The phenomenon of data islands between different teams or projects is prevalent, resulting in a large amount of valuable data not being fully utilized, resulting in a waste of resources. In addition, data synchronization and consistency issues are becoming increasingly prominent. In a distributed environment, it becomes extremely challenging to ensure real-time synchronization and consistency of data between different machine learning task nodes.

[0003] For example, Chinese patent publication number: CN118964335A, discloses a training data set version management method and system, which relates to the field of data management technology; including: step 1: establish a data set for model training and generate a unique identifier for the data set, step 2: manage the data set version: step 21: establish the data set version, step 22: use the formula V=D+T+S to generate a data set version identifier, step 23: create a static snapshot of the selected data set according to the data set version; step 24: according to the static snapshot, check whether the file object in the data set whose content will change is included in the static snapshot. If so, copy the current version of the file object to the corresponding storage of the static snapshot, and update the object index of the metadata of the static snapshot, and then perform the data set change operation, step 25: verify the integrity of the data set; the present invention reduces the complexity of data management and ensures the quality and reliability of model training.

[0004] However, the prior art still has the following problems:

[0005] The utilization efficiency of several versions of data sets after multiple rounds of processing is low, and when the amount of data is large, the computing power consumed in searching and sharing the data sets is high and the efficiency is low. Summary of the invention

[0006] To this end, the present invention provides an efficient data version management platform for machine learning training samples, so as to overcome the problems in the prior art of low efficiency in utilizing several versions of data sets after multiple rounds of processing, and high computing power consumed in searching and sharing data sets when the data volume is large, resulting in low efficiency.

[0007] To achieve the above objectives, the present invention provides an efficient data version management platform for machine learning training samples, which includes:

[0008] A data storage module, which is used to store a number of sample data sets and version data sets corresponding to different versions of each of the sample data sets after processing;

[0009] A feature analysis module, which is connected to the data storage module, is used to analyze the data features of the sample data set and the corresponding version data sets, extract the difference factors between the data features, and calculate the version variation representation coefficient based on the difference factors;

[0010] a label arrangement module, which is connected to the data storage module and the feature analysis module respectively, and is used to set a version variation label for each sample data set based on the version variation characterization coefficient;

[0011] A retrieval processing module, which is connected to the data storage module and the label arrangement module, is used to obtain retrieval requirement information and perform retrieval in the data storage module based on the version variation label, including:

[0012] Only extract the data of the sample data set to match the search requirement information, and call the sample data set and version data set to be extracted according to the matching result;

[0013] Or, clustering is performed based on the differences in data features between the sample data set and the corresponding version data set to obtain several cluster sets, each cluster set is screened and analyzed respectively, and the cluster set is called based on the screening and analysis results;

[0014] The screening and analysis includes screening the data of any data set in the cluster to match the search requirement information.

[0015] Furthermore, the feature analysis module is used to analyze the data features of the sample data set and the corresponding data sets of each version, including:

[0016] Extract data features of each image data in the sample data set and the version data set, including resolution, area of ​​contour features, saturation, contrast, brightness, and signal-to-noise ratio;

[0017] The difference factors between the data features of each image data in the sample data set and the version data set are calculated respectively, including the average difference ratio of resolution, the average difference ratio of area, the average difference ratio of contrast, the average difference ratio of brightness and the average difference ratio of signal-to-noise ratio.

[0018] Furthermore, the feature analysis module calculates the version variation representation coefficient for the sample data set based on the difference factor, including:

[0019] It is used to weight the difference factors and obtain the version variation representation coefficient.

[0020] Furthermore, the arrangement module is used to set the version variation label for each sample data set based on the version variation characterization coefficient, including:

[0021] Used to determine the version variation representation coefficient corresponding to each sample data set;

[0022] If the version variation characterization coefficient is greater than or equal to a preset version variation characterization threshold, it is determined that a strong variation label is set for the sample data set;

[0023] If the version variation characterization coefficient is less than a preset version variation characterization threshold, it is determined that a weak variation label is set for the sample data set.

[0024] Furthermore, the retrieval processing module obtains the retrieval requirement information including:

[0025] Used to obtain the retrieval requirement information sent by the user, including the required reference data samples.

[0026] Further, searching in the data storage module based on the version variation tag includes:

[0027] If the sample data set is a weak mutation label, only the data of the sample data set is extracted to match the search requirement information, and the sample data set and version data set to be extracted are called according to the matching result;

[0028] If the sample data set is a strong mutation label, clustering is performed based on the differences in data features between the sample data set and the corresponding version data set to obtain several cluster sets. Each cluster set is screened and analyzed, and the cluster set is called based on the screening and analysis results.

[0029] Furthermore, the retrieval processing module extracts data from the sample data set to match the retrieval requirement information, including:

[0030] Data used to extract sample data sets;

[0031] Used to calculate the fit between the data and the reference data sample.

[0032] Furthermore, the retrieval processing module calls the sample data set and the version data set to be extracted according to the matching result, including:

[0033] If the degree of fit corresponding to the sample data set is greater than a predetermined degree of fit threshold, it is determined that the sample data set and several version data sets corresponding to the sample data set need to be called.

[0034] Furthermore, the retrieval processing module is used to cluster the differences in data features between the sample data set and the corresponding version data set to obtain a number of cluster sets including:

[0035] Cluster the sample data set and the corresponding version data set according to the clustering conditions;

[0036] The clustering condition is that the difference factors between any data sets in the cluster set are less than a predetermined clustering factor threshold.

[0037] Further, calling the clustering set based on the screening and parsing results includes,

[0038] It is used to call any data from the cluster set, compare the called data with the reference sample data, and solve the degree of fit;

[0039] If the degree of fit is greater than a predetermined degree of fit threshold, it is determined that the cluster set needs to be called.

[0040] Compared with the prior art, the beneficial effect of the present invention lies in that the present invention sets a data storage module, a feature analysis module, a label arrangement module and a retrieval processing module, extracts and analyzes data features of a sample data set and data sets corresponding to each version through the feature analysis module, extracts difference factors between data features, and calculates version variation characterization coefficients accordingly to characterize the variation of the data itself after multiple processing of the data set, sets version variation labels for the sample data set through the label arrangement module, and subsequently adaptively searches in the data storage module based on the version variation labels to match the retrieval requirement information of the user end, thereby improving the analysis efficiency when analyzing massive data sets, matching data sets of various versions that can meet the needs of the user end, and improving the utilization rate of the data set.

[0041] In particular, the feature analysis module of the present invention extracts the difference factors between data features and calculates the version variation characterization coefficient. In actual situations, sample data sets undergo multiple rounds of processing in actual applications to meet training requirements, generating a large number of version data sets of several versions. Due to differences in processing methods, the differences in data feature dimensions of these sample data sets relative to the original data sets are relatively discrete. In some cases, the differences are small, and the original data set and several version data sets derived therefrom tend to meet the same training requirements. Therefore, it is considered to determine several difference factors and calculate the version variation characterization coefficient. The difference factors are the basic features of the data. The basic features can be acquired at a relatively fast speed, consume less computing power, and can characterize the differences between data sets. Therefore, the version variation characterization coefficient is calculated to provide data support for the subsequent setting of version variation labels for the version data sets, so as to facilitate the subsequent adaptive retrieval of the data storage module, improve the retrieval efficiency under massive data, and improve the utilization rate of the version data sets while ensuring reliability.

[0042] In particular, the retrieval processing module of the present invention searches the data storage module with the version variation label. In actual situations, after the user uploads the retrieval requirement information, it is necessary to use deep retrieval and deep matching on the data storage module. However, the data faced is massive. For the sample data set with a weak version variation label, the version variation is small. Therefore, when performing deep matching, consider only extracting data from the sample data set to match the retrieval requirement information, instead of matching the version data set derived from the sample data set. Under the premise of ensuring reliability, the amount of data processing is reduced. Under the strong version variation label, consider giving priority to clustering, screen and analyze each cluster set respectively, extract data from the cluster set to match the retrieval requirement information, and then adaptively use different retrieval methods to search. Under the premise of ensuring reliability, the retrieval efficiency under massive data is improved, and the utilization rate of the version data set is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the structure of an efficient data version management platform for machine learning training samples according to an embodiment of the present invention;

[0044] Figure 2 A logic block diagram of setting a version variation label for each sample data set according to an embodiment of the present invention;

[0045] Figure 3 A logical block diagram of searching in a data storage module based on a version variation tag according to an embodiment of the present invention;

[0046] Figure 4 The sample data set and the version data set to be extracted are called according to the matching results in the embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0049] It should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0050] See also Figures 1 to 4 As shown, Figure 1 Schematic diagram of the structure of an efficient data version management platform for machine learning training samples according to an embodiment of the present invention. Figure 2 A logic block diagram of setting a version variation label for each sample data set according to an embodiment of the present invention. Figure 3 The following is a logic diagram of searching in a data storage module based on a version change tag according to an embodiment of the present invention. Figure 4 In order to call the sample data set and version data set to be extracted according to the matching results in the embodiment of the present invention, the embodiment of the present invention provides an advertisement precision delivery system based on big data, which includes:

[0051] A data storage module, which is used to store a number of sample data sets and version data sets corresponding to different versions of each of the sample data sets after processing;

[0052] A feature analysis module, which is connected to the data storage module, is used to analyze the data features of the sample data set and the corresponding version data sets, extract the difference factors between the data features, and calculate the version variation representation coefficient based on the difference factors;

[0053] a label arrangement module, which is connected to the data storage module and the feature analysis module respectively, and is used to set a version variation label for each sample data set based on the version variation characterization coefficient;

[0054] A retrieval processing module, which is connected to the data storage module and the label arrangement module, is used to obtain retrieval requirement information and perform retrieval in the data storage module based on the version variation label, including:

[0055] Only extract the data of the sample data set to match the search requirement information, and call the sample data set and version data set to be extracted according to the matching result;

[0056] Or, clustering is performed based on the differences in data features between the sample data set and the corresponding version data set to obtain several cluster sets, each cluster set is screened and analyzed respectively, and the cluster set is called based on the screening and analysis results;

[0057] The screening and analysis includes screening the data of any data set in the cluster to match the search requirement information.

[0058] Specifically, there is no limitation on the specific structure of the data storage module, which can be a virtual database, or of course, other forms, as long as it can store data, which will not be elaborated here.

[0059] Specifically, there is no limitation on the specific structures of the feature analysis module, the label arrangement module and the retrieval processing module, and they can all be composed of logic components or a combination of logic components, and the logic components include a field programmable processor, a computer or a microprocessor in a computer.

[0060] It is understandable that after a data set completes a single processing, a new data set is formed, that is, a version data set. After several processings, several version data sets can be formed. There is no limitation on the processing method. In actual situations, the data set may usually be processed, for example, with color jitter (adjusting the brightness, contrast, and saturation of the image), Gaussian module (blurring the image to enhance the model's robustness to noise), noise addition (adding Gaussian noise or salt and pepper noise to the image), and affine transformation (translation, rotation, scaling, and shearing). After processing, the data features of the image will be changed to meet the corresponding training requirements. Of course, there are other processing methods, which will not be elaborated here.

[0061] Specifically, the feature analysis module is used to analyze the sample data set and the data features of the corresponding version data sets, including:

[0062] Extract data features of each image data in the sample data set and the version data set, including resolution, area of ​​contour features, saturation, contrast, brightness, and signal-to-noise ratio;

[0063] The difference factors between the data features of each image data in the sample data set and the version data set are calculated respectively, including the average difference ratio of resolution, the average difference ratio of area, the average difference ratio of contrast, the average difference ratio of brightness and the average difference ratio of signal-to-noise ratio.

[0064] It can be understood that the difference ratio is the ratio of the difference between two values ​​to the mean of the two values.

[0065] It is understandable that the sample data set needs to be compared with multiple versions of the data set, and the resolution difference ratio, area difference ratio, contrast difference ratio, brightness difference ratio and signal-to-noise ratio difference ratio can be solved one by one, and then the average resolution difference ratio, area average difference ratio, contrast average difference ratio, brightness average difference ratio and signal-to-noise ratio average difference ratio can be solved, which will not be repeated here.

[0066] Specifically, the feature analysis module calculates the version variation representation coefficient for the sample data set based on the difference factor, including:

[0067] It is used to weight the difference factors and obtain the version variation representation coefficient.

[0068] In implementation, the weights of the average difference ratio of resolution, the average difference ratio of area, the average difference ratio of contrast, the average difference ratio of brightness, and the average difference ratio of signal-to-noise ratio are 0.25, 0.15, 0.15, 0.15, and 0.3, respectively.

[0069] The feature analysis module of the present invention extracts the difference factors between data features and calculates the version variation characterization coefficient. In actual situations, sample data sets undergo multiple rounds of processing in actual applications to meet training requirements, generating a large number of version data sets of several versions. Due to differences in processing methods, the differences in data feature dimensions of these sample data sets relative to the original data sets are relatively discrete. In some cases, the differences are small, and the original data set and several version data sets derived therefrom tend to meet the same training requirements. Therefore, it is considered to determine several difference factors and calculate the version variation characterization coefficient. The difference factors are the basic features of the data. The basic features can be acquired at a relatively fast speed, consume less computing power, and can characterize the differences between data sets. Therefore, the version variation characterization coefficient is calculated to provide data support for subsequently setting version variation labels for the version data sets, so as to facilitate the subsequent adaptive retrieval of the data storage module, improve the retrieval efficiency under massive data and improve the utilization rate of the version data sets while ensuring reliability.

[0070] Specifically, the arrangement module is used to set the version variation label for each sample data set based on the version variation characterization coefficient, including:

[0071] Used to determine the version variation representation coefficient corresponding to each sample data set;

[0072] If the version variation characterization coefficient is greater than or equal to a preset version variation characterization threshold, it is determined that a strong variation label is set for the sample data set;

[0073] If the version variation characterization coefficient is less than a preset version variation characterization threshold, it is determined that a weak variation label is set for the sample data set.

[0074] The retrieval processing module of the present invention searches the data storage module with the version variation label. In actual situations, after the user uploads the retrieval requirement information, it is necessary to use deep retrieval to the data storage module for deep matching. However, the data faced is massive. For the sample data set with a weak version variation label, the version variation is small. Therefore, when performing deep matching, it is considered to extract only data from the sample data set to match the retrieval requirement information, replacing the matching result of the version data set derived from the sample data set. Under the premise of ensuring reliability, the data processing amount is reduced. Under the strong version variation label, it is considered to give priority to clustering, screen and analyze each cluster set respectively, extract data from the cluster set to match the retrieval requirement information, and then, adaptively use different retrieval methods to search. Under the premise of ensuring reliability, the retrieval efficiency under massive data is improved and the utilization rate of the version data set is improved.

[0075] Specifically, in the implementation, the version variation characterization threshold is selected within the interval [0.25, 0.3].

[0076] Specifically, the retrieval processing module obtains the retrieval requirement information including:

[0077] Used to obtain the retrieval requirement information sent by the user, including the required reference data samples.

[0078] It can be understood that the reference data sample is image data, and those skilled in the art can select image data that meets the training requirements and use the image data as the reference data sample.

[0079] Specifically, searching in the data storage module based on the version change tag includes:

[0080] If the sample data set is a weak mutation label, only the data of the sample data set is extracted to match the search requirement information, and the sample data set and version data set to be extracted are called according to the matching result;

[0081] If the sample data set is a strong mutation label, clustering is performed based on the differences in data features between the sample data set and the corresponding version data set to obtain several cluster sets. Each cluster set is screened and analyzed, and the cluster set is called based on the screening and analysis results.

[0082] Specifically, the retrieval processing module extracts data from the sample data set to match the retrieval requirement information, including:

[0083] Data used to extract sample data sets;

[0084] Used to calculate the fit between the data and the reference data sample.

[0085] Specifically, the data of the sample data set is image data, and the degree of fit is calculated by solving the degree of fit between image data, with the purpose of characterizing the similarity between images. For example, the degree of fit can be obtained by calculating the structural similarity index corresponding to the image data. Of course, other methods can also be used, which will not be elaborated here.

[0086] Specifically, the retrieval processing module calls the sample data set and version data set to be extracted according to the matching results, including:

[0087] If the degree of fit corresponding to the sample data set is greater than a predetermined degree of fit threshold, it is determined that the sample data set and several version data sets corresponding to the sample data set need to be called.

[0088] Specifically, the fitting threshold is calculated in advance, the data set used in the training process is recorded, the corresponding fitting mean between the data in the data set is determined, the mean of the corresponding fitting means in several training processes is solved, and the fitting threshold is set to the product of the mean and the precision coefficient, and the precision coefficient is selected in the interval [0.85, 0.95].

[0089] Specifically, the retrieval processing module is used to cluster the differences in data features between the sample data set and the corresponding version data set to obtain a number of cluster sets including:

[0090] Cluster the sample data set and the corresponding version data set according to the clustering conditions;

[0091] The clustering condition is that the difference factors between any data sets in the cluster set are less than a predetermined clustering factor threshold.

[0092] The clustering factor threshold is set based on the version variation representation threshold, and is set to 0.85 times the version variation representation threshold.

[0093] Specifically, clustering sets are called based on the screening and analysis results, including:

[0094] It is used to call any data from the cluster set, compare the called data with the reference sample data, and solve the degree of fit;

[0095] If the degree of fit is greater than a predetermined degree of fit threshold, it is determined that the cluster set needs to be called.

[0096] It is understandable that after calling the data, the index of the called data set can be sent to the user end, so that the user end can find the corresponding data set and call the data set.

[0097] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. An efficient data version management platform for machine learning training samples, characterized in that: include: A data storage module, which is used to store a number of sample data sets and version data sets corresponding to different versions of each of the sample data sets after processing; A feature analysis module, which is connected to the data storage module, is used to analyze the data features of the sample data set and the corresponding version data sets, extract the difference factors between the data features, and calculate the version variation representation coefficient based on the difference factors; a label arrangement module, which is connected to the data storage module and the feature analysis module respectively, and is used to set a version variation label for each sample data set based on the version variation characterization coefficient; A retrieval processing module, which is connected to the data storage module and the label arrangement module, is used to obtain retrieval requirement information and perform retrieval in the data storage module based on the version variation label, including: Only extract the data of the sample data set to match the search requirement information, and call the sample data set and version data set to be extracted according to the matching result; Or, clustering is performed based on the differences in data features between the sample data set and the corresponding version data set to obtain several cluster sets, each cluster set is screened and analyzed respectively, and the cluster set is called based on the screening and analysis results; The screening and analysis includes screening the data of any data set in the cluster to match the search requirement information.

2. The efficient data version management platform for machine learning training samples according to claim 1 is characterized in that: The feature analysis module is used to analyze the sample data set and the data features of the corresponding version data sets, including: Extract data features of each image data in the sample data set and the version data set, including resolution, area of ​​contour features, saturation, contrast, brightness, and signal-to-noise ratio; The difference factors between the data features of each image data in the sample data set and the version data set are calculated respectively, including the average difference ratio of resolution, the average difference ratio of area, the average difference ratio of contrast, the average difference ratio of brightness and the average difference ratio of signal-to-noise ratio.

3. The efficient data version management platform for machine learning training samples according to claim 2 is characterized in that: The feature analysis module calculates the version variation representation coefficient for the sample data set based on the difference factor, including: It is used to weight the difference factors and obtain the version variation representation coefficient.

4. The efficient data version management platform for machine learning training samples according to claim 1 is characterized in that: The arrangement module is used to set the version variation label for each sample data set based on the version variation characterization coefficient, including: Used to determine the version variation representation coefficient corresponding to each sample data set; If the version variation characterization coefficient is greater than or equal to a preset version variation characterization threshold, it is determined that a strong variation label is set for the sample data set; If the version variation characterization coefficient is less than a preset version variation characterization threshold, it is determined that a weak variation label is set for the sample data set.

5. The efficient data version management platform for machine learning training samples according to claim 1 is characterized in that: The retrieval processing module obtains the retrieval requirement information including: Used to obtain the retrieval requirement information sent by the user, including the required reference data samples.

6. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that: Search in the data storage module based on version change tags, including: If the sample data set is a weak mutation label, only the data of the sample data set is extracted to match the search requirement information, and the sample data set and version data set to be extracted are called according to the matching result; If the sample data set is a strong mutation label, clustering is performed based on the differences in data features between the sample data set and the corresponding version data set to obtain several cluster sets. Each cluster set is screened and analyzed, and the cluster set is called based on the screening and analysis results.

7. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that: The retrieval processing module extracts data from the sample data set and matches it with the retrieval requirement information, including: Data used to extract sample data sets; Used to calculate the fit between the data and the reference data sample.

8. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that: The retrieval processing module calls the sample data set and version data set to be extracted according to the matching results, including: If the degree of fit corresponding to the sample data set is greater than a predetermined degree of fit threshold, it is determined that the sample data set and several version data sets corresponding to the sample data set need to be called.

9. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that: The retrieval processing module is used to cluster the differences in data features between the sample data set and the corresponding version data set to obtain a number of cluster sets including: Cluster the sample data set and the corresponding version data set according to the clustering conditions; The clustering condition is that the difference factors between any data sets in the cluster set are less than a predetermined clustering factor threshold.

10. The efficient data version management platform for machine learning training samples according to claim 1, characterized in that: Based on the results of the screening analysis, the clustering set is called, including: It is used to call any data from the cluster set, compare the called data with the reference sample data, and solve the degree of fit; If the degree of fit is greater than a predetermined degree of fit threshold, it is determined that the cluster set needs to be called.

Images