Data cleaning method, device and system

Abstract

The present application provides a data cleaning method, device and system for solving the technical problems of low accuracy and inconsistent distribution of labeled sample data in the prior art. The method includes: performing model training on a first preset model based on a first sample data set to obtain training information for each sample data in the first sample data set; performing at least one data cleaning operation on the first sample data set based on the training information to obtain a target sample data set; wherein the data cleaning operation includes: deleting a target proportion of sample data in the first sample data set based on the training information to obtain a second sample data set; re-training the first preset model based on the second sample data set to obtain a second preset model, and performing model testing on the second preset model based on a test data set to obtain test parameters; in response to the test parameters being equal to preset indicators, determining the second sample data set as the target sample data set.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a data cleaning method, device and system. Background Art

[0002] In recent years, deep learning has developed rapidly in image processing fields such as target detection, target recognition, and action recognition based on its powerful ability to learn feature expressions. Deep learning obtains information that is of great help in interpreting data such as text, images, and sounds by learning the inherent laws and representation levels of sample data. Therefore, deep learning has very high requirements for the quality of sample data.

[0003] At present, for single-label sample data, sample data is often obtained by manually annotating the data. Due to the subjectivity of the annotators and the complexity of the scene, the annotation accuracy is low and the distribution of positive sample data and negative sample data is inconsistent, which affects the generalization of deep learning-based image processing models in actual application scenarios. If the annotated sample data is annotated again in order to improve the annotation accuracy, the annotation efficiency will be low.

[0004] It can be seen that the sample data labeled by existing technologies have problems such as low accuracy and inconsistent distribution, which leads to the poor generalization of deep learning-based image processing models in actual application scenarios. Summary of the Invention

[0005] The embodiments of the present application provide a data cleaning method, device and system for solving the technical problem in the prior art that the image processing model based on deep learning has poor generalization in actual application scenarios due to the low accuracy and inconsistent distribution of labeled sample data.

[0006] In a first aspect, to solve the above technical problems, an embodiment of the present application provides a data cleaning method, comprising:

[0007] Performing model training on a first preset model based on a first sample data set to obtain training information for each sample data in the first sample data set; wherein the training information is used to indicate a degree of similarity between a predicted value and a labeled value of the corresponding sample data, the predicted value being obtained by processing the corresponding sample data based on the first preset model;

[0008] Performing at least one data cleaning operation on the first sample data set based on the training information to obtain a target sample data set; wherein the data cleaning operation includes:

[0009] Deleting a target proportion of sample data from the first sample data set based on the training information to obtain a second sample data set;

[0010] Re-training the first preset model based on the second sample data set to obtain a second preset model, and performing model testing on the second preset model based on the test data set to obtain test parameters;

[0011] In response to the test parameter being equal to a preset indicator, the second sample data set is determined as the target sample data set.

[0012] In an embodiment of the present application, a first preset model can be trained based on a first sample data set to obtain training information for each sample data in the first sample data set, wherein the training information is used to indicate the degree of similarity between the predicted value and the labeled value of the corresponding sample data, the predicted value being obtained by processing the corresponding sample data based on the first preset model, and at least one data cleaning operation is performed on the first sample data set based on the training information to obtain a target sample data set, wherein the data cleaning operation includes: deleting a target proportion of sample data in the first sample data set based on the training information to obtain a second sample data set, retraining the first preset model based on the second sample data set to obtain a second preset model, and testing the second preset model based on a test data set to obtain a test parameter, and in response to the test parameter being equal to a preset indicator, determining the second sample data set as the target sample data set. Determining whether the sample data is noise by using the training information and the target proportion of the sample data, and deleting the sample data determined to be noise, compared to traditional secondary labeling, not only improves the labeling efficiency, but also avoids the problem of the preset model not being generalizable in actual application scenarios due to low labeling accuracy and inconsistent distribution of positive and negative sample data.

[0013] In an optional implementation, the method further includes: in response to the test parameter being not equal to a preset indicator, reselecting the target ratio and triggering the next data cleaning operation.

[0014] In an optional implementation, performing model training on a first preset model based on a first sample data set to obtain training information for each sample data in the first sample data set includes:

[0015] Performing model training on the first preset model based on the first sample data set, obtaining a first probability and a second probability of each sample data in the first sample data set in each training period; wherein the first probability is the probability that the predicted value of any sample data is the labeled value, and the second probability is the maximum probability that the predicted value is not the labeled value;

[0016] Obtain a difference between the first probability and the second probability, and use an average of the sum of the differences in each training period as training information of the corresponding sample data.

[0017] In an optional implementation, before deleting a target proportion of sample data from the first sample data set based on the training information to obtain the second sample data set, the method further includes:

[0018] A preset ratio set is obtained, the ratios in the preset ratio set are sorted from small to large, and a first ratio is selected as the target ratio; wherein the first ratio is the middle ratio in the preset ratio set.

[0019] In an optional implementation, deleting a target proportion of sample data from the first sample data set based on the training information to obtain a second sample data set includes:

[0020] The target proportion of sample data in the first sample data set is deleted in order based on the similarity levels indicated by the training information to obtain the second sample data set.

[0021] In an optional implementation, sequentially deleting the target proportion of sample data in the first sample data set based on the degree of similarity indicated by the training information to obtain the second sample data set includes:

[0022] Sort the negative sample data and the positive sample data in the first sample data set from small to large based on the degree of similarity indicated by the training information;

[0023] The target ratio of negative sample data and positive sample data is deleted in sequence starting from the largest ranked negative sample data and the smallest ranked positive sample data, respectively, to obtain the second sample data set.

[0024] In an embodiment of the present application, the negative sample data and positive sample data in the first sample data set can be sorted from small to large based on the degree of similarity indicated by the training information, and the negative sample data and positive sample data of the target ratio are deleted in order starting from the negative sample data with the largest sorting and the positive sample data with the smallest sorting, respectively, to obtain a second sample data set. By respectively deleting the number of simple negative sample data (negative sample data with a large degree of similarity, which is easily confused with the positive sample data) and difficult positive sample data (positive sample data with a small degree of similarity, which is more likely to be noise) corresponding to the target ratio, the gap between the positive and negative sample data is increased, and the overall quality of the sample data set is improved, thereby avoiding the problem of poor generalization of the preset model in actual application scenarios due to low annotation accuracy and inconsistent distribution of positive and negative sample data.

[0025] In an optional implementation, in response to the test parameter not being equal to the preset indicator, reselecting the target ratio includes:

[0026] In response to the test parameter not being equal to the preset indicator, grouping the proportions in the preset proportion set to obtain a first sub-set of preset proportions and a second sub-set of preset proportions; wherein the ranking of the proportions in the first sub-set of preset proportions is lower than the ranking of the proportions in the second sub-set of preset proportions;

[0027] If the test parameter is greater than the preset index, reselecting a second ratio as the target ratio; wherein the second ratio is the middle ratio in the first sub-preset ratio set;

[0028] If the test parameter is less than the preset indicator, a third ratio is reselected as the target ratio; wherein the third ratio is the middle ratio in the second sub-preset ratio set.

[0029] In an embodiment of the present application, in response to a test parameter not being equal to a preset index, the proportions in the preset proportion set can be grouped to obtain a first sub-preset proportion set and a second sub-preset proportion set, wherein the ranking of the proportions in the first sub-preset proportion set is lower than the ranking of the proportions in the second sub-preset proportion set, and if the test parameter is greater than the preset index, the second proportion is reselected as the target proportion, wherein the second proportion is the proportion ranked in the middle of the first sub-preset proportion set, and if the test parameter is less than the preset index, the third proportion is reselected as the target proportion, wherein the third proportion is the proportion ranked in the middle of the second sub-preset proportion set. By comparing the size of the test parameter and the preset index, a binary method is used to iteratively select the optimal target proportion in the preset proportion set, and then determine that the first sample data set for data cleaning based on the optimal target proportion is the target sample data set, and the first preset model for model training based on the target sample data set is the target model.

[0030] In a second aspect, an embodiment of the present application further provides a data cleaning device, comprising:

[0031] a training module, configured to perform model training on a first preset model based on a first sample data set, and obtain training information for each sample data in the first sample data set; wherein the training information is used to indicate a degree of similarity between a predicted value and a labeled value of the corresponding sample data, the predicted value being obtained by processing the corresponding sample data based on the first preset model;

[0032] A cleaning module is configured to perform at least one data cleaning operation on the first sample data set based on the training information to obtain a target sample data set; wherein the data cleaning operation includes:

[0033] Deleting a target proportion of sample data from the first sample data set based on the training information to obtain a second sample data set;

[0034] Re-training the first preset model based on the second sample data set to obtain a second preset model, and performing model testing on the second preset model based on the test data set to obtain test parameters;

[0035] In response to the test parameter being equal to a preset indicator, the second sample data set is determined as the target sample data set.

[0036] In an optional implementation, the method further includes: in response to the test parameter being not equal to a preset indicator, reselecting the target ratio and triggering the next data cleaning operation.

[0037] In an optional implementation manner, the training module is specifically used to:

[0038] Performing model training on the first preset model based on the first sample data set, obtaining a first probability and a second probability of each sample data in the first sample data set in each training period; wherein the first probability is the probability that the predicted value of any sample data is the labeled value, and the second probability is the maximum probability that the predicted value is not the labeled value;

[0039] Obtain a difference between the first probability and the second probability, and use an average of the sum of the differences in each training period as training information of the corresponding sample data.

[0040] In an optional implementation manner, the device further includes a selection module, configured to:

[0041] A preset ratio set is obtained, the ratios in the preset ratio set are sorted from small to large, and a first ratio is selected as the target ratio; wherein the first ratio is the middle ratio in the preset ratio set.

[0042] In an optional embodiment, the cleaning module is specifically used to:

[0043] The target proportion of sample data in the first sample data set is deleted in order based on the similarity levels indicated by the training information to obtain the second sample data set.

[0044] In an optional embodiment, the cleaning module is specifically used to:

[0045] Sort the negative sample data and the positive sample data in the first sample data set from small to large based on the degree of similarity indicated by the training information;

[0046] The target ratio of negative sample data and positive sample data is deleted in sequence starting from the largest ranked negative sample data and the smallest ranked positive sample data, respectively, to obtain the second sample data set.

[0047] In an optional embodiment, the cleaning module is specifically used to:

[0048] In response to the test parameter not being equal to the preset indicator, grouping the proportions in the preset proportion set to obtain a first sub-set of preset proportions and a second sub-set of preset proportions; wherein the ranking of the proportions in the first sub-set of preset proportions is lower than the ranking of the proportions in the second sub-set of preset proportions;

[0049] If the test parameter is greater than the preset index, reselecting a second ratio as the target ratio; wherein the second ratio is the middle ratio in the first sub-preset ratio set;

[0050] If the test parameter is less than the preset indicator, a third ratio is reselected as the target ratio; wherein the third ratio is the middle ratio in the second sub-preset ratio set.

[0051] In a third aspect, an embodiment of the present application further provides a data cleaning system, comprising:

[0052] a memory for storing program instructions;

[0053] The processor is configured to call the program instructions stored in the memory and execute the steps included in any one of the implementations of the first aspect according to the obtained program instructions.

[0054] In a fourth aspect, an embodiment of the present application provides a storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to enable a computer to execute the steps included in any one of the implementations in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application.

[0056] Figure 1 A schematic diagram of the structure of a data cleaning system provided in an embodiment of the present application;

[0057] Figure 2a A flowchart of a data cleaning method provided in an embodiment of the present application;

[0058] Figure 2b A schematic diagram of sequentially deleting a target ratio of negative sample data and positive sample data provided by an embodiment of the present application;

[0059] Figure 3 A schematic diagram of the structure of a data cleaning device provided in an embodiment of the present application;

[0060] Figure 4 A structural diagram of another data cleaning system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Unless there is a conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although a logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0062] The terms "first" and "second" in the specification and claims of this application and the accompanying drawings are used to distinguish different objects, not to describe a specific order. In addition, the terms "comprising" and "including" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to the process, method, product, or device.

[0063] In the embodiment of the present application, "at least one" may mean at least two, for example, two, three or more, and the embodiment of the present application does not limit this.

[0064] In addition, the term "and / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates an "or" relationship between the related objects.

[0065] Currently, single-label sample data is often obtained through manual annotation. Due to the subjectivity of the annotators and the complexity of the scenarios, the annotation accuracy is low and the distribution of positive and negative sample data is inconsistent, which affects the generalization of deep learning-based image processing models in actual application scenarios. If the annotated sample data is re-annotated to improve the annotation accuracy, the annotation efficiency will be low. Therefore, the sample data annotated by the existing technology has problems such as low accuracy and inconsistent distribution, resulting in the poor generalization of deep learning-based image processing models in actual application scenarios.

[0066] In view of this, an embodiment of the present application provides a data cleaning method. The method can train a first preset model based on a first sample data set to obtain training information for each sample data in the first sample data set, wherein the training information is used to indicate the degree of similarity between the predicted value and the labeled value of the corresponding sample data, and the predicted value is obtained by processing the corresponding sample data based on the first preset model. At least one data cleaning operation is performed on the first sample data set based on the training information to obtain a target sample data set. The data cleaning operation includes: deleting a target proportion of sample data in the first sample data set based on the training information to obtain a second sample data set; retraining the first preset model based on the second sample data set to obtain a second preset model; and testing the second preset model based on a test data set to obtain test parameters. In response to the test parameters being equal to a preset indicator, the second sample data set is determined as the target sample data set. The method determines whether the sample data is noise based on the training information and the target proportion of the sample data, and deletes the sample data determined to be noise. Compared with traditional secondary labeling, the method not only improves labeling efficiency, but also avoids the problem of poor generalization of the preset model in actual application scenarios due to low labeling accuracy and inconsistent distribution of positive and negative sample data.

[0067] In order to better understand the above technical solution, the technical solution of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0068] Figure 1 This is a structural diagram of a data cleaning system to which the method provided in the embodiment of the present application is applicable. Of course, the method provided in the embodiment of the present application can be applied to a variety of data cleaning systems. It should be understood that Figure 1The data cleaning system shown is a simple description of a data cleaning system that is applicable to the method provided in the embodiment of the present application, and is not a limitation of the data cleaning system that is applicable to the method provided in the embodiment of the present application.

[0069] Figure 1 The data cleaning system shown includes a memory 101, a processor 102, and a bus interface 103. The memory 101 and the processor 102 are connected via the bus interface 103. The memory 101 is used to store program instructions. The processor 102 is used to call the program instructions stored in the memory 101 and execute all steps included in the data cleaning method according to the obtained program instructions.

[0070] See Figure 2a , is a flow chart of a data cleaning method provided in an embodiment of the present application, which can be performed by the aforementioned Figure 1 The data cleaning system shown in FIG. 4 is executed. The specific process of the method is described as follows.

[0071] Step 201: Perform model training on a first preset model based on a first sample data set to obtain training information of each sample data in the first sample data set.

[0072] In an embodiment of the present application, for single-label sample data, a first sample data set can be obtained first, wherein the first sample data set includes the same number of positive sample data and negative sample data, and the positive sample data is obtained by cropping an image containing a target (target image), such as cropping a target area on the target image. The obtained positive sample data has low labeling accuracy and contains noise due to the subjectivity of the labeler when labeling the target and the complexity of the scene. The negative sample data is obtained by cropping an image not containing a target (background image), such as randomly cropping an area on the background image that is the same size as the target area. The obtained negative sample data does not contain noise because there is no need to label the target. Then, the first preset model is trained based on the first sample data set to obtain the first probability and the second probability of each sample data in the first sample data set in each training period, wherein the first probability is the probability that the predicted value of any sample data is the labeled value, and the second probability is the maximum probability that the predicted value of any sample data is not the labeled value, and the difference between the first probability and the second probability is obtained, and the mean of the sum of the differences in each training period is used as the training information of the corresponding sample data, wherein the first preset model is an image processing model based on deep learning, such as a target recognition model, a target detection model, an action recognition model, etc., and the training information is used to indicate the degree of similarity between the predicted value and the labeled value of the corresponding sample data, and the predicted value is obtained by processing the corresponding sample data based on the first preset model.

[0073] Specifically, the first preset model is trained based on the first sample data set, and the training information of each sample data in the first sample data set is obtained using the first formula, wherein the first formula is:

[0074]

[0075] AUM(x, y) is the area under the margin (AUM) between the predicted value output by the first preset model based on the sample data x and the labeled value y of the sample data x. The size of AUM can reflect the gap between the predicted value output by the first preset model based on the sample data x and the labeled value y of the sample data x. For example, the larger the AUM, the smaller the gap and the greater the similarity. On the contrary, the smaller the AUM, the larger the gap and the smaller the similarity. Therefore, AUM can be used as a criterion for judging whether the sample data is noise. For example, the smaller the AUM, the greater the probability that the positive sample data is noise. The larger the AUM, the easier it is for the negative sample data to be confused with the positive sample data. is the probability that the predicted value i output by the first preset model based on the sample data x in the tth training period (epoch) is the labeled value y of the sample data x, is the maximum probability that the predicted value i output by the first preset model based on the sample data x in the tth training period is not the labeled value y of the sample data x, T is the number of training periods. For example, the first preset model is a target recognition model, the labeled value y of the sample data x is a cup, and the probability that the predicted value i output by the first preset model based on the sample data x in the first training period is a cup is 60%, the probability of being a mobile phone is 20%, the probability of being a plate is 40%, and the probability of being a computer is 50%. Then, the probability that the predicted value i output by the first preset model based on the sample data x in the first training period is the labeled value y of the sample data x is 60%, and the maximum probability of not being the labeled value y of the sample data x is 50%.

[0076] Step 202: Perform at least one data cleaning operation on the first sample data set based on the training information to obtain a target sample data set.

[0077] In an embodiment of the present application, after obtaining training information for each sample data in the first sample data set indicating the degree of similarity between the predicted value output by the first preset model and the labeled value, at least one data cleaning operation is performed on the first sample data set based on the training information to obtain a target sample data set, the first preset model is retrained based on the second sample data set to obtain a second preset model, and the second preset model is tested based on the test data set to obtain a test parameter. In response to the test parameter being equal to the preset indicator, the second sample data set is determined as the target sample data set. In response to the test parameter not being equal to the preset indicator, the target ratio is reselected and the next data cleaning operation is triggered. Specifically, the steps of the data cleaning operation are as follows:

[0078] (1) Based on the degree of similarity indicated by the training information, sample data of a target proportion in the first sample data set is deleted in order to obtain a second sample data set.

[0079] First, obtain a preset ratio set, sort the ratios in the preset ratio set from small to large, and select the first ratio as the target ratio, where the first ratio is the ratio ranked in the middle in the preset ratio set. For example, the ratios in the preset ratio set obtained are 10%, 30%, and 50%, and 30% is selected as the target ratio. Based on the degree of similarity indicated by the training information, the negative sample data and the positive sample data in the first sample data set are sorted from small to large, and the negative sample data and the positive sample data of the target ratio are deleted in sequence starting from the negative sample data with the largest degree of similarity and the positive sample data with the smallest degree of similarity, respectively, to obtain a second sample data set. For example, X1(x, y) is the negative sample data remaining after deleting the number of negative sample data corresponding to the target ratio in sequence starting from the negative sample data with the largest degree of similarity, and X2(x, y) is the positive sample data remaining after deleting the number of positive sample data corresponding to the target ratio in sequence starting from the positive sample data with the smallest degree of similarity. D(x, y) is the second sample data set, then D(x, y) = X1(x, y) + X2(x, y). By deleting the corresponding number of simple negative sample data (negative sample data with a high degree of similarity, which is easily confused with positive sample data) and difficult positive sample data (positive sample data with a low degree of similarity, which is more likely to be noise) in the target ratio, the gap between positive and negative sample data is increased, and the overall quality of the sample data set is improved, thereby avoiding the problem of poor generalization of the preset model in actual application scenarios due to low labeling accuracy and inconsistent distribution of positive and negative sample data.

[0080] For example, Figure 2bAs shown, it is a schematic diagram of sequentially deleting a target proportion of negative sample data and positive sample data provided by an embodiment of the present application. Based on the size of the similarity, the positive sample data and negative sample data in the first sample data set are sorted from small to large, and the AUM distribution diagram of the positive sample data and the negative sample data is obtained, wherein the horizontal axis in the distribution diagram is the similarity (i.e., AUM), and the vertical axis is the number density value of the sample data (i.e., Density). Since negative sample data with a greater degree of similarity are easily confused with positive sample data, and positive sample data with a smaller degree of similarity are more likely to be noise, the target proportion c% of negative sample data is deleted starting from the negative sample data with the greatest similarity, and the target proportion c% of positive sample data is deleted starting from the positive sample data with the least similarity.

[0081] (2) Retraining the first preset model based on the second sample data set to obtain a second preset model, and performing model testing on the second preset model based on the test data set to obtain test parameters.

[0082] After deleting the target proportion of sample data in the first sample data set in order based on the degree of similarity indicated by the training information to obtain the second sample data set, the first preset model is re-trained based on the second sample data set to obtain the second preset model, and then the test data set is obtained, wherein the test data set is similar to the first sample data set and includes the same number of positive sample data and negative sample data. The second preset model is tested based on the test data set to obtain test parameters. For example, the preset model is a target recognition model, and the test parameters can be the recognition accuracy or recognition rate, etc. The embodiment of the present application does not make specific limitations on this.

[0083] (3) In response to the test parameter being equal to the preset index, the second sample data set is determined as the target sample data set; in response to the test parameter not being equal to the preset index, the target ratio is reselected and the next data cleaning operation is triggered.

[0084] After the second preset model is tested based on the test data set and the test parameters are obtained, it is determined whether the test parameters are equal to the preset indicators. If the test parameters are equal to the preset indicators, the first ratio is determined to be the optimal target ratio, the second sample data set is determined to be the target sample data set, and the second preset model is determined to be the target model. If the test parameters are not equal to the preset indicators, the target ratio is reselected from the preset ratio set to re-clean the first sample data set, and the first preset model is re-trained based on the re-cleaned first sample data set to obtain a new second preset model, until the test parameters obtained by model testing the new second preset model based on the test data set are equal to the preset indicators, and the re-selected target ratio is determined to be the optimal target ratio, the re-cleaned first sample data set is determined to be the target sample data set, and the new second preset model obtained by re-model training is determined to be the target model.

[0085] Specifically, if the test parameter is not equal to the preset index, the proportions in the preset proportion set are grouped to obtain a first sub-preset proportion set and a second sub-preset proportion set, wherein the proportions in the first sub-preset proportion set are ranked lower than the proportions in the second sub-preset proportion set. If the test parameter is greater than the preset index, the second proportion is selected as the target proportion to re-clean the first sample dataset, wherein the second proportion is the proportion ranked in the middle of the first sub-preset proportion set. If the test parameter is less than the preset index, the third proportion is selected as the target proportion to re-clean the first sample dataset, wherein the third proportion is the proportion ranked in the middle of the second sub-preset proportion set. The first preset model is re-trained based on the re-cleaned first sample dataset to obtain a new second preset model, until the test parameter obtained by model testing the new second preset model based on the test dataset equals the preset index. By comparing the test parameter with the preset index, a binary search method is used to iteratively select the optimal target proportion from the preset proportion set, and then the first sample dataset that has been re-cleaned based on the optimal target proportion is determined as the target sample dataset, and the first preset model that has been re-trained based on the target sample dataset is determined as the target model.

[0086] For example, the ratios in the preset ratio set are 10%, 15%, 20%, 30%, 40%, 45%, and 50%. For the first time, 30% is selected as the target ratio to perform data cleaning on the first sample data set to obtain a second sample data set. Based on the second sample data set, the first preset model is retrained to obtain a second preset model. Based on the test data set, the second preset model is tested to obtain a test parameter. If the test parameter corresponding to 30% is greater than the preset index, 15% is selected as the target ratio to re-clean the first sample data set for the second time. If the test parameter corresponding to 30% is less than the preset index, 45% is selected as the target ratio to re-clean the first sample data set for the second time. Similarly, if the test parameter corresponding to 15% is less than the preset index, 20% is selected as the target ratio to re-clean the first sample data set for the third time. If the test parameter corresponding to 45% is greater than the preset index, 40% is selected as the target ratio to re-clean the first sample data set for the third time, until the test parameter corresponding to the target ratio is equal to the preset index.

[0087] Based on the same inventive concept, the present application also provides a data cleaning device, which can be applied to the aforementioned Figure 1 The data cleaning system shown. This data cleaning device can implement the functions corresponding to the aforementioned data cleaning method. The data cleaning device can be a hardware structure, a software module, or a hardware structure and a software module. The data cleaning device can be implemented by a chip system, which can be composed of a chip or include a chip and other discrete devices. See Figure 3 , is a structural diagram of a data cleaning device provided in an embodiment of the present application, wherein the data cleaning device includes a training module 301 and a cleaning module 302.

[0088] A training module 301 is configured to perform model training on a first preset model based on a first sample data set to obtain training information for each sample data in the first sample data set; wherein the training information is used to indicate the degree of similarity between a predicted value and a labeled value of the corresponding sample data, the predicted value being obtained by processing the corresponding sample data based on the first preset model;

[0089] The cleaning module 302 is configured to perform at least one data cleaning operation on the first sample data set based on the training information to obtain a target sample data set; wherein the data cleaning operation includes:

[0090] Deleting a target proportion of sample data from the first sample data set based on the training information to obtain a second sample data set;

[0091] Re-training the first preset model based on the second sample data set to obtain a second preset model, and performing model testing on the second preset model based on the test data set to obtain test parameters;

[0092] In response to the test parameter being equal to a preset indicator, the second sample data set is determined as the target sample data set.

[0093] In an optional implementation, the method further includes: in response to the test parameter being not equal to a preset indicator, reselecting the target ratio and triggering the next data cleaning operation.

[0094] In an optional implementation manner, the training module 301 is specifically configured to:

[0095] Performing model training on the first preset model based on the first sample data set, obtaining a first probability and a second probability of each sample data in the first sample data set in each training period; wherein the first probability is the probability that the predicted value of any sample data is the labeled value, and the second probability is the maximum probability that the predicted value is not the labeled value;

[0096] Obtain a difference between the first probability and the second probability, and use an average of the sum of the differences in each training period as training information of the corresponding sample data.

[0097] In an optional implementation manner, the device further includes a selection module, configured to:

[0098] A preset ratio set is obtained, the ratios in the preset ratio set are sorted from small to large, and a first ratio is selected as the target ratio; wherein the first ratio is the middle ratio in the preset ratio set.

[0099] In an optional embodiment, the cleaning module 302 is specifically configured to:

[0100] The target proportion of sample data in the first sample data set is deleted in order based on the similarity levels indicated by the training information to obtain the second sample data set.

[0101] In an optional embodiment, the cleaning module 302 is specifically configured to:

[0102] Sort the negative sample data and the positive sample data in the first sample data set from small to large based on the degree of similarity indicated by the training information;

[0103] The target ratio of negative sample data and positive sample data is deleted in sequence starting from the largest ranked negative sample data and the smallest ranked positive sample data, respectively, to obtain the second sample data set.

[0104] In an optional embodiment, the cleaning module 302 is specifically configured to:

[0105] In response to the test parameter not being equal to the preset indicator, grouping the proportions in the preset proportion set to obtain a first sub-set of preset proportions and a second sub-set of preset proportions; wherein the ranking of the proportions in the first sub-set of preset proportions is lower than the ranking of the proportions in the second sub-set of preset proportions;

[0106] If the test parameter is greater than the preset index, reselecting a second ratio as the target ratio; wherein the second ratio is the middle ratio in the first sub-preset ratio set;

[0107] If the test parameter is less than the preset indicator, a third ratio is reselected as the target ratio; wherein the third ratio is the middle ratio in the second sub-preset ratio set.

[0108] Based on the same inventive concept, the present application also provides a data cleaning system. Figure 4 , is a structural diagram of a data cleaning system provided in an embodiment of the present application. The data cleaning system includes at least one processor 402 and a memory 401 connected to the at least one processor. The specific connection medium between the processor 402 and the memory 401 is not limited in the embodiment of the present application. Figure 4 For example, the processor 402 and the memory 401 are connected via a bus 400. Figure 4 The bus 400 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0109] In an embodiment of the present application, the memory 401 stores instructions that can be executed by at least one processor 402. At least one processor 402 can execute the steps included in the aforementioned data cleaning method by calling the instructions stored in the memory 401. Among them, the processor 402 is the control center of the data cleaning system, which can use various interfaces and lines to connect the various parts of the entire data cleaning system, and realize various functions of the data cleaning system by executing the instructions stored in the memory 401. Optionally, the processor 402 may include one or more processing units, and the processor 402 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 402. In some embodiments, the processor 402 and the memory 401 can be implemented on the same chip. In some embodiments, they can also be implemented separately on independent chips.

[0110] Memory 401 is a kind of non-volatile computer readable storage medium, which can be used for storing non-volatile software programs, non-volatile computer executable programs and modules.Memory 401 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic storage, disk, optical disk, etc. Memory 401 is any other medium that can be used to carry or store the desired program code with instruction or data structure form and can be accessed by a computer, but is not limited thereto. The memory 401 in the embodiment of the present application can also be a circuit or other arbitrarily capable of realizing a storage function, for storing program instructions and / or data.

[0111] In an embodiment of the present application, the processor 402 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the data cleaning method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0112] By designing and programming the processor 402, the code corresponding to the data cleaning method introduced in the aforementioned embodiment can be solidified into the chip, so that the chip can execute the steps of the aforementioned data cleaning method during operation. How to design and program the processor 402 is a technology well known to those skilled in the art and will not be repeated here.

[0113] Based on the same inventive concept, an embodiment of the present application further provides a storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer executes the steps of the aforementioned data cleaning method.

[0114] In some possible implementations, various aspects of the data cleaning method provided in the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a data cleaning system, the program code is used to enable the data cleaning system to perform the steps of the data cleaning method according to various exemplary embodiments of the present application described above in this specification.

[0115] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0116] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0119] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A data cleaning method, characterized in that: include: Performing model training on a first preset model based on a first sample data set to obtain training information for each sample data in the first sample data set; wherein the first preset model is an image processing model based on deep learning, and the training information is used to indicate the degree of similarity between a predicted value and a labeled value of the corresponding sample data, the predicted value being obtained by processing the corresponding sample data based on the first preset model; Performing at least one data cleaning operation on the first sample data set based on the training information to obtain a target sample data set; wherein the data cleaning operation includes: Sort the negative sample data and the positive sample data in the first sample data set from small to large based on the degree of similarity indicated by the training information; Starting from the largest ranked negative sample data and the smallest ranked positive sample data, respectively, sequentially delete the target ratio of negative sample data and positive sample data to obtain a second sample data set; Re-training the first preset model based on the second sample data set to obtain a second preset model, and performing model testing on the second preset model based on the test data set to obtain test parameters; In response to the test parameter being equal to a preset indicator, the second sample data set is determined as the target sample data set.

2. The method according to claim 1, wherein Also includes: In response to the test parameter not being equal to the preset indicator, the target ratio is reselected and the next data cleaning operation is triggered.

3. The method according to claim 1 or 2, wherein: Performing model training on a first preset model based on a first sample data set to obtain training information for each sample data in the first sample data set includes: Performing model training on the first preset model based on the first sample data set, obtaining a first probability and a second probability of each sample data in the first sample data set in each training period; wherein the first probability is the probability that the predicted value of any sample data is the labeled value, and the second probability is the maximum probability that the predicted value is not the labeled value; Obtain a difference between the first probability and the second probability, and use an average of the sum of the differences in each training period as training information of the corresponding sample data.

4. The method according to claim 2, wherein Before obtaining the second sample data set, the method further includes: A preset ratio set is obtained, the ratios in the preset ratio set are sorted from small to large, and a first ratio is selected as the target ratio; wherein the first ratio is the middle ratio in the preset ratio set.

5. The method according to claim 4, wherein In response to the test parameter not being equal to the preset indicator, reselecting the target ratio includes: In response to the test parameter not being equal to the preset indicator, grouping the proportions in the preset proportion set to obtain a first sub-set of preset proportions and a second sub-set of preset proportions; wherein the ranking of the proportions in the first sub-set of preset proportions is lower than the ranking of the proportions in the second sub-set of preset proportions; If the test parameter is greater than the preset index, reselecting a second ratio as the target ratio; wherein the second ratio is the middle ratio in the first sub-preset ratio set; If the test parameter is less than the preset indicator, a third ratio is reselected as the target ratio; wherein the third ratio is the middle ratio in the second sub-preset ratio set.

6. A data cleaning device, characterized in that: include: a training module, configured to perform model training on a first preset model based on a first sample data set to obtain training information for each sample data in the first sample data set; wherein the first preset model is an image processing model based on deep learning, and the training information is used to indicate the degree of similarity between a predicted value and a labeled value of the corresponding sample data, the predicted value being obtained by processing the corresponding sample data based on the first preset model; A cleaning module is configured to perform at least one data cleaning operation on the first sample data set based on the training information to obtain a target sample data set; wherein the data cleaning operation includes: Sort the negative sample data and the positive sample data in the first sample data set from small to large based on the degree of similarity indicated by the training information; Starting from the largest ranked negative sample data and the smallest ranked positive sample data, respectively, sequentially delete the target ratio of negative sample data and positive sample data to obtain a second sample data set; Re-training the first preset model based on the second sample data set to obtain a second preset model, and performing model testing on the second preset model based on the test data set to obtain test parameters; In response to the test parameter being equal to a preset indicator, the second sample data set is determined as the target sample data set.

7. A data cleaning system, characterized in that: include: a memory for storing program instructions; A processor is configured to call the program instructions stored in the memory and execute the steps included in the method according to any one of claims 1 to 5 according to the obtained program instructions.

8. A storage medium, characterized in that: The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the steps included in the method according to any one of claims 1 to 5.

Images