Data cleaning and labeling method and device, electronic equipment and storage medium

Abstract

The invention provides a data cleaning and labeling method and device, electronic equipment and a storage medium, and belongs to the field of data processing. The method comprises the following steps: training a basic model by using a first data set to obtain an identification model; based on the recognition model, reasoning results of the first data set and a second data set are obtained, and the second data set comprises unlabeled complex samples; according to a reasoning result, performing label modification on the first data set, and labeling the second data set; and obtaining a new first data set and a new second data set from the first data set and the second data set under the condition that the reasoning result does not meet the end condition. Therefore, labeling of the to-be-labeled complex samples and label correction of the labeled simple samples are automatically realized based on a small number of labeled simple samples in a circulating manner, the labeling efficiency is greatly improved, the labor cost is reduced, and introduction of manual errors is avoided to a certain extent, so that the accuracy of the sample labels is ensured.

Description

Technical Field

[0001] The present application relates to the field of data processing, and more specifically, to a data cleaning and annotation method, device, electronic device and storage medium. Background Art

[0002] Data labeling plays a vital role in the training process of machine learning and deep learning models. It provides the necessary supervision signals for model training, has a profound impact on the performance, generalization ability and optimization direction of the model, and becomes one of the key factors to ensure the success of machine learning models. In the subsequent iterative training process of the model, in order to further improve the generalization performance of the model, the labels of the sample data are continuously modified.

[0003] At present, manual labeling is the most commonly used data labeling method, which can ensure the reliability of labeling quality. However, when faced with large-scale data sets, manual labeling is relatively inefficient and accompanied by high labor costs. In addition, manual labeling may also introduce new errors, thereby affecting the effectiveness of model learning results, resulting in the model's prediction effect not accurately reflecting the actual situation. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a data cleaning and labeling method, apparatus, electronic device and storage medium, which can automatically realize the labeling of complex samples to be labeled and the label correction of simple samples that have been labeled based on a small number of labeled simple samples, thereby greatly improving the labeling efficiency, reducing labor costs, and avoiding the introduction of human errors to a certain extent.

[0005] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, the present application provides a data cleaning and annotation method, the method comprising:

[0007] Using a first data set, the basic model is trained to obtain a recognition model; wherein the first data set includes simple samples that are free of recognition ambiguity and have been labeled;

[0008] Based on the recognition model, obtaining inference results of the first data set and the second data set; wherein the second data set includes unlabeled complex samples;

[0009] According to the inference result, modify the label of the first data set and annotate the second data set;

[0010] When the inference result does not satisfy the termination condition, a new first data set and a second data set are obtained from the first data set and the second data set, and the step of using the first data set to train the basic model to obtain the recognition model is returned to be executed.

[0011] Optionally, the inference result includes the prediction value, prediction accuracy and prediction recall of the recognition model for simple samples, and the prediction value and confidence mean for complex samples;

[0012] The step of modifying the label of the first data set and annotating the second data set according to the inference result includes:

[0013] Dividing the first data set into a first subset and a second subset according to the prediction accuracy and the prediction recall;

[0014] According to the confidence mean, dividing the second data set into a third subset and a fourth subset;

[0015] According to the current prediction value, the label of the third subset is confirmed, and the label of the second subset is modified.

[0016] Optionally, the step of confirming the label of the third subset and modifying the label of the second subset according to the current prediction value includes:

[0017] For the third subset, initiating a label confirmation instruction to indicate that a new label of the sample is selected from the label and the current prediction value;

[0018] For the second subset, a label modification instruction is initiated to instruct to modify the label of the sample according to the current prediction value.

[0019] Optionally, the step of obtaining a new first data set and a new second data set from the first data set and the second data set includes:

[0020] taking the first subset, the second subset and the third subset as a new first data set;

[0021] The fourth subset is used as a new second data set.

[0022] Optionally, the step of obtaining the inference results of the first data set and the second data set based on the recognition model includes:

[0023] Input each simple sample and complex sample into the recognition model respectively to obtain the prediction result output by the recognition model;

[0024] For each simple sample, according to the prediction result and label of the simple sample, obtain the prediction performance value of the simple sample;

[0025] For each complex sample, the confidence mean of the complex sample is obtained according to the prediction result of the complex sample.

[0026] Optionally, the prediction result includes at least one prediction value;

[0027] The step of obtaining the prediction performance value of the simple sample according to the prediction result and label of the simple sample includes:

[0028] According to the predicted value and label of the simple sample, the number of predicted values ​​with accurate prediction is counted to obtain the predicted accurate number of the simple sample;

[0029] Obtaining a prediction accuracy rate according to the predicted accuracy number and the total number of predicted values ​​of simple samples;

[0030] The prediction recall rate is obtained according to the prediction accuracy number and the total number of labels of simple samples.

[0031] Optionally, the inference result includes a confidence mean of a complex sample;

[0032] After the step of modifying the label of the first data set and annotating the second data set according to the inference result, the method further includes:

[0033] According to the confidence mean of each complex sample, obtaining the current confidence mean of the second data set;

[0034] Whether the termination condition is met is determined based on the current confidence mean value and the previous confidence mean value of the second data set.

[0035] In a second aspect, the present application provides a data cleaning and annotation device, including a model training module, a reasoning module, a labeling module and a data adjustment module;

[0036] The model training module is used to train the basic model using the first data set to obtain a recognition model; wherein the first data set includes simple samples that are free of recognition ambiguity and have been annotated;

[0037] The reasoning module is used to obtain reasoning results of the first data set and the second data set based on the recognition model; wherein the second data set includes unlabeled complex samples;

[0038] The labeling module is used to modify the label of the first data set and label the second data set according to the inference result;

[0039] The data adjustment module is used to obtain a new first data set and a new second data set from the first data set and the second data set when the inference result does not meet the termination condition, and return to execute the step of using the first data set to train the basic model to obtain the recognition model.

[0040] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the data cleaning and annotation method as described in the first aspect.

[0041] In a fourth aspect, the present application provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data cleaning and annotation method as described in the first aspect.

[0042] The data cleaning and labeling method, device, electronic device and storage medium provided by the embodiment of the present application include: using the first data set to train the basic model to obtain the recognition model, the first data set includes simple samples without recognition ambiguity and labeled; based on the recognition model, the reasoning results of the first data set and the second data set are obtained, and the second data set includes unlabeled complex samples; according to the reasoning results, the label of the first data set is modified and the second data set is labeled; when the reasoning result does not meet the end condition, a new first data set and a second data set are obtained from the first data set and the second data set, and the step of training the basic model using the first data set and obtaining the recognition model is returned. In this way, through a cyclical manner, based on a small number of labeled simple samples, the labeling of the complex samples to be labeled and the label correction of the labeled simple samples are automatically realized, which greatly improves the labeling efficiency, reduces the labor cost, and avoids the introduction of manual errors to a certain extent. In addition, in the cycle process, the samples with incorrect labeling are continuously corrected to ensure the accuracy of the sample labels.

[0043] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1A schematic diagram of the system architecture of a data cleaning and annotation system provided in an embodiment of the present application is shown.

[0046] Figure 2 A schematic diagram of the module architecture of an electronic device provided in an embodiment of the present application is shown.

[0047] Figure 3 One of the flow charts of the data cleaning and annotation method provided in an embodiment of the present application is shown.

[0048] Figure 4 Shows Figure 3 Schematic diagram of the process flow of some sub-steps of step 13.

[0049] Figure 5 Shows Figure 4 A flowchart of some sub-steps of step 133 in FIG.

[0050] Figure 6 Shows Figure 3 Schematic diagram of the process flow of some sub-steps of step 15.

[0051] Figure 7 Shows Figure 3 Schematic diagram of the process of some sub-steps of step 17.

[0052] Figure 8 The second flowchart of the data cleaning and annotation method provided in the embodiment of the present application is shown.

[0053] Fig. 9 A schematic diagram of the module architecture of the data cleaning and annotation device provided in an embodiment of the present application is shown.

[0054] Icons: 10-data cleaning and annotation system; 110-drone; 120-annotation equipment; 130-auxiliary equipment; 20-electronic equipment; 210-memory; 220-processor; 230-communication module; 30-data cleaning and annotation device; 310-model training module; 320-inference module; 330-annotation module; 340-data adjustment module. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0057] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0058] The data cleaning and annotation method provided in the embodiment of the present application can be applied to Figure 1 In the data cleaning and annotation system 10 shown, the data cleaning and annotation system 10 includes a drone 110, an annotation device 120 and an auxiliary device 130, and the auxiliary device 130 is communicatively connected to the annotation device 120 and the drone 110 through a network.

[0059] The drone 110 is used to shoot multiple scenes and objects to be identified at different heights and different pan / tilt angles to obtain video data, and transmit the video data to the auxiliary device 130 .

[0060] The auxiliary device 130 is used to deframe the video data to obtain an initial sample, and for the technician to divide the initial sample into a first data set and a second data set, and transmit the first data set and the second data set to the labeling device 120. The first data set includes a plurality of simple samples, which are samples without identification ambiguity and have been labeled, and the second data set includes a plurality of complex samples to be labeled.

[0061] The labeling device 120 is used to implement the data cleaning and labeling method provided in an embodiment of the present application, including: using a first data set to train a basic model to obtain a recognition model, the first data set includes simple samples that are free of recognition ambiguity and have been labeled; based on the recognition model, obtaining inference results of the first data set and the second data set, the second data set includes unlabeled complex samples; according to the inference results, modifying the labels of the first data set and labeling the second data set; when the inference results do not meet the end conditions, obtaining new first data sets and second data sets from the first data set and the second data set, and returning to execute the step of using the first data set to train the basic model to obtain the recognition model.

[0062] The auxiliary device 130 may be a personal computer, a laptop, a tablet computer, a mobile phone or any other intelligent terminal capable of input and display. The annotation device 120 may be an independent server, a server cluster, a personal computer, a laptop, a tablet computer or any other electronic device 20 or computer device.

[0063] In one example, the service device and the annotation device 120 may be the same electronic device 20 or computer device.

[0064] Please refer to Figure 2 , is a block diagram of an electronic device 20, which may be Figure 1 The labeling system in the data cleaning labeling system 10 is shown. The electronic device 20 includes a memory 210, a processor 220 and a communication module 230. The memory 210, the processor 220 and the communication module 230 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0065] The memory 210 is used to store programs or data and can be, but not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable read-only memory, an electrically erasable read-only memory, and the like.

[0066] The processor 220 is used to read / write data or programs stored in the memory 210 and execute corresponding functions. For example, Figure 1 In the data cleaning and annotation system 10 shown, the processor 220 of the annotation device 120 executes the computer program stored in the memory 210 to implement the data cleaning and annotation method provided in the embodiment of the present application.

[0067] The communication module 230 is used to establish a communication connection between the electronic device 20 and other communication terminals through the network, and to send and receive data through the network. Figure 1In the data cleaning and annotation system 10 shown, the communication module 230 of the annotation device 120 transmits and receives data with the auxiliary device 130 through the network.

[0068] It should be understood that Figure 2 The structure shown is only a schematic diagram of the structure of the electronic device 20. The electronic device 20 may also include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations are shown. Figure 2 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0069] In order to solve the problem that manual labeling has relatively low efficiency and high labor cost, and manual labeling may also introduce new errors, thereby affecting the effectiveness of model learning results and causing the model's prediction effect to fail to accurately reflect the actual situation, the present application embodiment provides a data cleaning and labeling method, referring to Figure 3 , including steps 11 to 17. And, Figure 1 The labeling device 120 in the data cleaning system shown can Figure 2 The structure shown implements the execution of steps 11 to 17 when the processor 220 reads the computer program stored in the memory 210.

[0070] Step 11: Use the first data set to train the basic model to obtain a recognition model.

[0071] Step 13: Obtain inference results of the first data set and the second data set based on the recognition model.

[0072] Step 15: modify the label of the first data set and annotate the second data set according to the inference result.

[0073] Step 17: if the inference result does not meet the termination condition, obtain a new first data set and a new second data set from the first data set and the second data set, and return to step 11.

[0074] The first data set includes simple samples that are unambiguous and labeled, and the second data set includes complex samples that are not labeled, and the number of simple samples is smaller than the number of complex samples.

[0075] For example, in combination Figure 1In the data cleaning and annotation system 10 shown, the drone 110 shoots multiple scenes and objects to be identified at different heights and different gimbal angles to obtain video data, and transmits the video data to the auxiliary device 130. The auxiliary device 130 is used to obtain initial samples after de-framing and de-duplication of the video data. The technician checks each initial sample and annotates the initial samples without identification ambiguity according to the difficulty of identification (i.e., whether there is identification ambiguity, whether there is ambiguity in the objects to be identified in the initial sample at a glance), and puts them into the first data set d1, and the other samples are used as the second data set d2. The auxiliary device 130 transmits the first data set d1 and the second data set d2 to the annotation device 120.

[0076] The labeling device 120 first uses the first data set to train the basic model to obtain a recognition model. Then, based on the recognition model, the inference results of the first data set and the second data set are obtained. Furthermore, the labeling device 120 modifies the label of the first data set and labels the second data set according to the inference results. Finally, if the inference result does not meet the end condition, a new first data set and a new second data set are obtained from the first data set and the second data set, so as to repeat the above process until the inference result meets the end condition, so as to complete the label correction of the first data set and the labeling of the second data set.

[0077] In steps 11 to 17 of the data cleaning and labeling method of the embodiment of the present application, the labeling of complex samples to be labeled and the label correction of simple samples that have been labeled are automatically realized in a cyclic manner based on a small number of labeled simple samples, which greatly improves the labeling efficiency, reduces labor costs, and avoids the introduction of human errors to a certain extent. Moreover, in the cycle process, samples with incorrect labels are continuously corrected to ensure the accuracy of sample labels. This helps to improve the effectiveness of the model learning results and make the model's prediction effect more accurately reflect the actual situation.

[0078] In step 11, the labeling device 120 first divides the first data set d1 into a training set and a test set by using any ratio division method, wherein the ratio of training set:test set=7:3, or training set:test set=8:2, or any other ratio.

[0079] Taking the 7:3 ratio as an example, a random number between 0 and 1 is randomly generated for each simple sample in the first data set d1. When the random number is less than or equal to 0.7, the simple sample is considered to be a training set, otherwise it is a test set. This division of the data set can make full use of the labeled data and improve the stability of the model.

[0080] Furthermore, the basic model is trained using the divided first data set (ie, the training set and the test set) to obtain a recognition model.

[0081] During the model training process, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning or any other model training method can be used.

[0082] In one example, the model training method used in step 11 can be a boosting training method, which combines multiple learners to complete the model's learning task, so that the model can learn some difficult-to-recognize samples as much as possible, thereby obtaining better performance than a single learner and ensuring the performance of the recognition model.

[0083] After the recognition model is obtained, in step 13, the output result after the recognition model is inferred on each sample in the first data set and the second data set can be directly used as the inference result, or the output result can be preprocessed to obtain the inference result. The above two methods are examples, and their implementation methods are not limited.

[0084] In one example, in order to make the inference results more intuitively reflect the prediction performance of the recognition model for simple samples and complex samples, the results of inference on the samples according to the recognition model are introduced in step 13 to obtain the prediction performance value of the simple sample about the difference between the prediction result and the label, as well as the concept of the confidence mean of the responsible sample. Figure 4 The process of obtaining the inference result in step 13 includes steps 131 to 135.

[0085] Step 131, input each simple sample and complex sample into the recognition model respectively to obtain the prediction result output by the recognition model.

[0086] Step 133: for each simple sample, obtain the prediction performance value of the simple sample according to the prediction result and label of the simple sample.

[0087] Step 135 , for each complex sample, obtain the confidence mean of the complex sample according to the prediction result of the complex sample.

[0088] Whether it is a simple sample or a complex sample, the prediction result of the recognition model includes at least one predicted value and the confidence of each predicted value, and each simple sample includes multiple labels.

[0089] In step 133, the prediction accuracy can be obtained based on multiple prediction values ​​and multiple labels of the simple sample, and the prediction accuracy can be used as the prediction performance value, or the prediction performance value can be obtained according to a preset rule. Both of these methods are examples, and the method of obtaining the prediction performance value is not limited.

[0090] In one example, in order to make the prediction performance value more detailed and comprehensive to reflect the gap between the label of a simple sample and the prediction result, the prediction accuracy and the prediction recall are introduced as the concept of the prediction performance value in step 133. Figure 5 The process of obtaining the prediction performance value of a simple sample includes steps 1331 to 1335.

[0091] Step 1331, based on the predicted values ​​and labels of the simple samples, the number of predicted values ​​that are accurately predicted is counted to obtain the number of accurate predictions of the simple samples.

[0092] Step 1333, obtain the prediction accuracy rate according to the number of accurate predictions and the total number of predicted values ​​of simple samples.

[0093] Step 1335, obtain the prediction recall rate according to the prediction accuracy number and the total number of labels of the simple samples.

[0094] For example, when the samples in the first data set and the second data set are used to train the model's ability to recognize target objects in images, the predicted value may be a predicted bounding box and the label may be a true bounding box. At this time, for each predicted bounding box, if the predicted bounding box is consistent with a true bounding box, it means that the predicted bounding box is accurately predicted. Therefore, the number of predicted bounding boxes that are accurately predicted is counted to obtain the number of accurate predictions.

[0095] When the samples in the first data set and the second data set are used to train the model's ability to classify the target object in the image, the predicted value can be the predicted category and the label can be the true category. At this time, for each predicted category, if the predicted category is consistent with a true category, it means that the predicted category is accurately predicted. Therefore, the number of predicted categories with accurate predictions is counted to obtain the number of accurate predictions.

[0096] Furthermore, the ratio between the number of accurate predictions and the total number of predicted values ​​of simple samples is taken as the prediction accuracy. The ratio between the number of accurate predictions and the total number of labels of simple samples is taken as the prediction recall.

[0097] Through the above method, we can obtain the prediction recall rate that reflects the difference between the prediction accuracy number of simple samples and the total number of labels, as well as the prediction accuracy rate that reflects the difference between the prediction accuracy number of simple samples and the total number of predicted values, so as to more carefully and comprehensively reflect the reasoning and evaluation of the model on simple samples.

[0098] In step 135, the confidence levels of the predicted values ​​of the complex sample are averaged to obtain the confidence mean of the complex sample.

[0099] In step 15, the label of the first data set is modified, and the way of labeling the second data set can be flexibly set. For example, the label of each simple sample in the first data set can be confirmed and modified, and the predicted value is used as the label of each complex sample in the second data set. It can also be that the predicted value is used as the label of the complex sample only when the confidence of the predicted value exceeds a threshold, and the label of the simple sample whose label is inconsistent with the predicted value is modified. The implementation method is limited to steps.

[0100] In one example, in order to reduce the error rate and ensure the accuracy of label modification and annotation, after obtaining the prediction performance value of each simple sample in the first data set and the confidence mean value of each complex sample in the second data set according to the above method, the label of the simple sample whose prediction performance value does not meet the requirement is modified, and the complex sample whose confidence mean value reaches the confidence threshold is annotated. Figure 6 In step 15, the label of the first data set is modified, and the process of labeling the second data set includes steps 151 to 155.

[0101] Step 151 , dividing the first data set into a first subset and a second subset according to the prediction accuracy and the prediction recall.

[0102] Step 153: Divide the second data set into a third subset and a fourth subset according to the confidence mean.

[0103] Step 155: According to the current prediction value, the label of the third subset is confirmed, and the label of the second subset is modified.

[0104] The accuracy threshold, recall threshold and confidence threshold are preset, and the three thresholds can be values ​​set based on experience, values ​​set according to accuracy requirements, or values ​​obtained by fitting the index results of the data with a normal distribution.

[0105] Thus, in the above step 151, for each simple sample in the first data set, if the prediction accuracy of the simple sample is greater than the accuracy threshold, and the prediction recall rate is greater than the recall rate threshold, it means that the simple sample is a sample with better performance (i.e., the current prediction value is relatively consistent with the label), and the simple sample is placed in the first subset d11. On the contrary, if the prediction accuracy of the simple sample is not greater than the accuracy threshold, and / or the prediction recall rate is not greater than the recall rate threshold, it means that the simple sample is a sample with poor performance (i.e., the current prediction value is greatly different from the label), and it is placed in the second subset d12.

[0106] Similarly, in step 153, for each complex sample in the first data set, if the confidence mean of the complex sample is greater than the confidence threshold, it means that the complex sample is a sample with good performance (i.e., the probability that the current predicted value is the true value is high), and the complex sample is placed in the third subset d21. On the contrary, if the confidence mean of the complex sample is not greater than the confidence threshold, it means that the complex sample is a sample with poor performance (i.e., the probability that the current predicted value is the true value is low), and it is placed in the fourth subset d22.

[0107] Furthermore, in step 155, for the second subset whose current prediction value differs greatly from the label, label modification is performed. For the third subset whose current prediction value has a high probability of being the true value, label confirmation is performed.

[0108] For the first subset whose predicted value is relatively consistent with the label, the original label is retained. For the fourth dataset whose predicted value is less likely to be the true value, no labeling is performed.

[0109] For each simple sample in the second subset, the current predicted value can be used as a new label, or it can be modified according to a preset rule. For each complex sample in the third subset, the current predicted value can be used as a label, or the predicted value with a confidence greater than the true threshold can be used as a label. The above methods are all examples, and their implementation methods are not limited.

[0110] In order to ensure the accuracy of the labels modified by the second subset and the accuracy of the labels annotated by the third subset, the concept of manual modification and manual label confirmation is introduced in step 155 .

[0111] By way of example, for the third subset, a label confirmation instruction is initiated to indicate that a new label of the sample is selected from the label and the current prediction value.

[0112] For the second subset, a label modification instruction is initiated to instruct to modify the label of the sample according to the current prediction value.

[0113] Combination Figure 1 In the data cleaning and labeling system 10 shown, the labeling device 120 initiates a label confirmation instruction to the auxiliary device 130 for the third subset, and the auxiliary device 130 displays each complex sample in the third subset, the current predicted value of the complex sample and its confidence on the display screen, so that the technician can weigh the predicted value and select the predicted value as the label of the complex sample.

[0114] Similarly, the labeling device 120 initiates a label modification instruction to the auxiliary device 130 for the second subset. The auxiliary device 130 displays the original label of the simple sample, the current prediction value and its confidence on the display screen for the technician to weigh and select from the label and prediction value to obtain a new label for the simple sample.

[0115] In the above manner, for the second subset whose current predicted value differs greatly from the label, the technician uses the predicted value and label as a reference to modify the label, and for the third subset whose current predicted value has a high probability of being the true value, the technician uses the predicted value as a reference to set the label, while the remaining samples remain unchanged. Thus, to a certain extent, it avoids introducing wrong labels for complex samples, further improves the reliability of simple sample labels, and ensures the accuracy of label modification and annotation.

[0116] On the basis of the above, refer to Figure 7 The process of obtaining the new first data set and the second data set in step 17 includes steps 171 to 173. Step 171 and step 173 can be executed in parallel or successively.

[0117] Step 171: Use the first subset, the second subset, and the third subset as a new first data set.

[0118] Step 173: Use the fourth subset as a new second data set.

[0119] Through the above method, the simple answer samples with the original labels (i.e., the simple samples with a small difference between the predicted value and the label of the recognition model), the simple samples with modified labels, and the complex samples with labeled labels (i.e., the complex samples with a high probability that the predicted value of the recognition model is the true value) are used as the new first data set, and the remaining complex samples (i.e., the complex samples with a low probability that the predicted value of the recognition model is the true value) are used as the new second data set, and the above steps 11 to 17 are repeated to continue to correct the labels of the new first data set, label the new second data set, continuously iteratively modify the labels of the samples, and obtain reliable labels for the unlabeled samples. In this way, the recognition model's learning of complex samples is continuously strengthened, the model's prediction performance is improved, and the model with improved performance is used to continuously correct or label the labels of the samples, so that both the simple samples and the complex samples are finally accurately labeled.

[0120] In the above-mentioned cycle process from step 11 to step 17, the basic model in the current step 11 can be the most initial model, or it can be the recognition model trained after step 11 of the previous cycle, and its implementation method is not limited.

[0121] In order to avoid entering an infinite loop, the data cleaning and labeling method provided in the embodiment of the present application also includes a step of detecting whether the end condition is met, which is performed between step 15 and step 17.

[0122] In one example, a target number of iterations may be preset, and when the total number of loops from step 11 to step 17 reaches the target number of iterations, the end condition is met.

[0123] In another example, in order to ensure the efficiency of label modification and labeling of samples while ensuring the accuracy of labels, a concept of judging whether the end condition is met based on the confidence mean of complex samples is introduced. Figure 8 In the data cleaning and annotation method provided in the embodiment of the present application, the step of detecting whether the end condition is met includes step 16A and step 16B.

[0124] Step 16A, obtaining the current confidence mean of the second data set according to the confidence mean of each complex sample.

[0125] Step 16B: Determine whether the end condition is met based on the current confidence mean value and the previous confidence mean value of the second data set. If not, execute step 17; if yes, end.

[0126] For example, the difference between the current confidence mean and the previous confidence mean of the second data set is calculated. If the difference is less than the preset difference threshold, it means that the model has learned the characteristics of complex samples, the prediction performance has stabilized, and the current prediction value is closest to the true value. At this time, the termination condition is met, otherwise it is not met.

[0127] Through the above method, when the prediction performance of the model is stable, that is, when the current prediction value is closest to the true value, the label correction and annotation of the data is completed to ensure the reliability and accuracy of the final label of each sample.

[0128] Based on the same concept as the above data cleaning and annotation method, refer to Fig. 9 The embodiment of the present application also provides a data cleaning and annotation device 30, including a model training module 310, a reasoning module 320, a labeling module 330 and a data adjustment module 340.

[0129] The model training module 310 is used to train the basic model using the first data set to obtain a recognition model, wherein the first data set includes simple samples that are unambiguous and have been labeled.

[0130] The reasoning module 320 is used to obtain reasoning results of the first data set and the second data set based on the recognition model, wherein the second data set includes unlabeled complex samples.

[0131] The labeling module 330 is used to modify the label of the first data set and label the second data set according to the inference result.

[0132] The data adjustment module 340 is used to obtain new first and second data sets from the first and second data sets when the inference result does not meet the termination condition, and instruct the model training module 310 to execute the step of using the first data set to train the basic model to obtain the recognition model.

[0133] In one example, the data cleaning and labeling device 30 further includes an end determination module.

[0134] The end judgment module is used to obtain the current confidence mean of the second data set according to the confidence mean of each complex sample; and judge whether the end condition is met according to the current confidence mean of the second data set and the previous confidence mean.

[0135] The above-mentioned data cleaning and labeling device 30, under the coordinated action of the model training module 310, the reasoning module 320, the labeling module 330 and the data adjustment module 340, automatically realizes the labeling of complex samples to be labeled and the label correction of simple samples that have been labeled in a cyclic manner based on a small number of labeled simple samples, which greatly improves the labeling efficiency, reduces labor costs, and avoids the introduction of human errors to a certain extent. Moreover, in the cycle process, samples with incorrect labels are continuously corrected to ensure the accuracy of sample labels. This helps to improve the effectiveness of the model learning results and make the prediction effect of the model more accurately reflect the actual situation.

[0136] For the specific implementation and effects of the data cleaning and labeling device 30, please refer to the description of the implementation of the data cleaning and labeling method above, such as, for the specific implementation and effects of the model training module 310, please refer to the description of the relevant content of step 11 above, for the specific implementation and effects of the reasoning module 320, please refer to the description of the relevant content of step 13 above, for the specific implementation and effects of the labeling module 330, please refer to the description of the relevant content of step 15 above, for the specific implementation and effects of the data adjustment module 340, please refer to the description of the relevant content of step 17 above, and no further details will be given here.

[0137] In addition, each module of the above-mentioned data cleaning and annotation device 30 can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor 220 in the electronic device 20 in the form of hardware, or can be stored in the memory 210 of the electronic device 20 in the form of software, so that the processor 220 can call and execute the operations corresponding to the above-mentioned modules to implement the data cleaning and annotation method provided above.

[0138] The embodiment of the present application also provides an electronic device 20, including a processor 220 and a memory 210, wherein the memory 210 stores a computer program that can be executed by the processor 220, and the processor 220 can execute the computer program to implement the data cleaning and annotation method provided above.

[0139] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 220, the data cleaning and annotation method proposed in the embodiment of the present application is implemented.

[0140] In summary, the data cleaning and annotation method, device, electronic device, and storage medium provided in the embodiments of the present application have at least the following beneficial effects:

[0141] (1) Based on the labeling of a small number of simple samples, the original labeling can be cleaned and a large number of samples to be labeled can be pre-labeled, thereby reducing the labeling workload of the labelers, and correcting some incorrectly labeled data, thereby improving the reliability and accuracy of the labeling;

[0142] (2) Introduce corrections by annotators during the annotation process to achieve mutual guidance between the model and humans, use the model to identify fuzzy and complex samples, and then introduce annotators to correct the recognition and prediction results of the model in a timely manner, so as to achieve the unification of the annotation results and further improve the reliability and accuracy of the annotation;

[0143] (3) Make the final evaluation indicators and labels of the samples basically consistent with the manual evaluation indicators and labels, so that the prediction indicators of the model can more realistically reflect the manual judgment;

[0144] (IV) During the training process, the boosting learning method is used to strengthen the learning of difficult samples. The sample prediction results are judged by the accuracy, recall and confidence of the model's own output, which makes full use of the model's own characteristics. The labels are corrected by combining the reference model prediction results with manual methods, so that both the labeler and the model can learn information from each other, making the labeling more accurate and reliable.

[0145] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0147] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0148] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data cleaning and annotation method, characterized in that: The method comprises: Using a first data set, the basic model is trained to obtain a recognition model; wherein the first data set includes simple samples that are free of recognition ambiguity and have been labeled; Based on the recognition model, obtaining inference results of the first data set and the second data set; wherein the second data set includes unlabeled complex samples; According to the inference result, modify the label of the first data set and annotate the second data set; When the inference result does not satisfy the termination condition, a new first data set and a second data set are obtained from the first data set and the second data set, and the step of using the first data set to train the basic model to obtain the recognition model is returned to be executed.

2. The data cleaning and annotation method according to claim 1, characterized in that: The inference results include the prediction value, prediction accuracy and prediction recall of the recognition model for simple samples, and the prediction value and confidence mean for complex samples; The step of modifying the label of the first data set and annotating the second data set according to the inference result includes: Dividing the first data set into a first subset and a second subset according to the prediction accuracy and the prediction recall; According to the confidence mean, dividing the second data set into a third subset and a fourth subset; According to the current prediction value, the label of the third subset is confirmed, and the label of the second subset is modified.

3. The data cleaning and annotation method according to claim 2, characterized in that: The step of confirming the label of the third subset and modifying the label of the second subset according to the current prediction value includes: For the third subset, initiating a label confirmation instruction to indicate that a new label of the sample is selected from the label and the current prediction value; For the second subset, a label modification instruction is initiated to instruct to modify the label of the sample according to the current prediction value.

4. The data cleaning and annotation method according to claim 2, characterized in that: The step of obtaining a new first data set and a new second data set from the first data set and the second data set comprises: taking the first subset, the second subset and the third subset as a new first data set; The fourth subset is used as a new second data set.

5. The data cleaning and annotation method according to any one of claims 1 to 4, characterized in that: The step of obtaining the inference results of the first data set and the second data set based on the recognition model includes: Input each simple sample and complex sample into the recognition model respectively to obtain the prediction result output by the recognition model; For each simple sample, according to the prediction result and label of the simple sample, obtain the prediction performance value of the simple sample; For each complex sample, the confidence mean of the complex sample is obtained according to the prediction result of the complex sample.

6. The data cleaning and annotation method according to claim 5, characterized in that: The prediction result includes at least one prediction value; The step of obtaining the prediction performance value of the simple sample according to the prediction result and label of the simple sample includes: According to the predicted value and label of the simple sample, the number of predicted values ​​with accurate prediction is counted to obtain the predicted accurate number of the simple sample; Obtaining a prediction accuracy rate according to the predicted accuracy number and the total number of predicted values ​​of simple samples; The prediction recall rate is obtained according to the prediction accuracy number and the total number of labels of simple samples.

7. The data cleaning and annotation method according to any one of claims 1 to 4, characterized in that: The inference result includes the confidence mean of the complex sample; After the step of modifying the label of the first data set and annotating the second data set according to the inference result, the method further includes: According to the confidence mean of each complex sample, obtaining the current confidence mean of the second data set; Whether the termination condition is met is determined based on the current confidence mean value and the previous confidence mean value of the second data set.

8. A data cleaning and labeling device, characterized in that: Includes model training module, reasoning module, labeling module and data adjustment module; The model training module is used to train the basic model using the first data set to obtain a recognition model; wherein the first data set includes simple samples that are free of recognition ambiguity and have been annotated; The reasoning module is used to obtain reasoning results of the first data set and the second data set based on the recognition model; wherein the second data set includes unlabeled complex samples; The labeling module is used to modify the label of the first data set and label the second data set according to the inference result; The data adjustment module is used to obtain a new first data set and a new second data set from the first data set and the second data set when the inference result does not meet the termination condition, and return to execute the step of using the first data set to train the basic model to obtain the recognition model.

9. An electronic device, characterized in that: It includes a processor and a memory, the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the data cleaning and annotation method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data cleaning and annotation method according to any one of claims 1 to 7 is implemented.

Images