Label correction method and device, computer equipment and computer readable storage medium

Abstract

The invention relates to a label correction method and device, computer equipment and a storage medium. The method comprises the following steps: performing model training on a basic classification model according to a sample data set of a current round to obtain a target model of which the current round is trained; predicting the prediction probability of each sample in the sample data set of the current round under each category label through the target model trained in the current round; according to the prediction probability of each sample under each category label and the current labeling label of each sample, determining a target sample needing label correction in the current round of sample data set and an estimated real label of each target sample; and correcting the current labeling label of the target sample into a pre-estimated real label to obtain a sample data set of which the label correction of the current round is completed, and entering the next round of iteration as an updated sample data set of the current round until an iteration stopping condition is met to obtain a target sample data set of which the label correction is completed. By adopting the method, the universality and generalization of label correction can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a label correction method, apparatus, computer equipment and computer-readable storage medium. Background Art

[0002] In supervised learning model training, labeled sample data is needed to train the model. This is widely used in classification tasks, such as text sentiment classification, image classification, song style classification, etc. However, in practical applications, sample labels are inevitably mislabeled, that is, sample labels are noisy, which has a great impact on the accuracy of model training. Therefore, it is necessary to correct the sample labels.

[0003] In traditional methods, specific rules are usually formulated for label correction for specific types of sample data. For example, for text datasets, it is necessary to manually construct a word list or matching rules. This method is greatly affected by subjective factors and has poor versatility and generalization. Summary of the invention

[0004] The embodiments of the present application provide a label correction method, apparatus, computer device, computer-readable storage medium, and computer program product, which can improve the versatility and generalization of label correction.

[0005] In a first aspect, the present application provides a label correction method. The method comprises:

[0006] Using the initial sample data set to be corrected as the sample data set of the current round;

[0007] In each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training;

[0008] Predicting the prediction probability of each sample in the sample data set of the current round under each category label by using the target model trained in the current round;

[0009] According to the predicted probabilities of each sample under each category label and the current labeled labels of each sample, determine the target samples that need to be labeled corrected in the sample data set of the current round, and the estimated true labels of each target sample; the current labeled label is the category label actually labeled by the sample at present;

[0010] Correct the current annotated label of the target sample to the estimated true label, obtain the sample data set with the current round of label correction, and enter the next round of iteration as the updated sample data set of the current round until the iteration stop condition is met to obtain the target sample data set with the label correction.

[0011] In a second aspect, the present application also provides a label correction device. The device comprises:

[0012] The model training module is used to use the initial sample data set to be corrected as the sample data set of the current round; in each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training;

[0013] The confidence learning module is used to predict the prediction probability of each sample in the sample data set of the current round under each category label through the target model trained in the current round; determine the target samples in the sample data set of the current round that need to be label corrected, and the estimated true label of each target sample according to the prediction probability of each sample under each category label and the current label label of each sample; the current label label is the category label actually labeled by the sample at present;

[0014] The label correction module is used to correct the current annotated label of the target sample to the estimated true label, obtain the sample data set after the current round of label correction, and use it as the updated sample data set of the current round to enter the next round of iteration until the iteration stop condition is met, and obtain the target sample data set after the label correction.

[0015] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0016] Using the initial sample data set to be corrected as the sample data set of the current round;

[0017] In each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training;

[0018] Predicting the prediction probability of each sample in the sample data set of the current round under each category label by using the target model trained in the current round;

[0019] According to the predicted probabilities of each sample under each category label and the current labeled labels of each sample, determine the target samples that need to be labeled corrected in the sample data set of the current round, and the estimated true labels of each target sample; the current labeled label is the category label actually labeled by the sample at present;

[0020] Correct the current annotated label of the target sample to the estimated true label, obtain the sample data set with the current round of label correction, and enter the next round of iteration as the updated sample data set of the current round until the iteration stop condition is met to obtain the target sample data set with the label correction.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0022] Using the initial sample data set to be corrected as the sample data set of the current round;

[0023] In each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training;

[0024] Predicting the prediction probability of each sample in the sample data set of the current round under each category label by using the target model trained in the current round;

[0025] According to the predicted probabilities of each sample under each category label and the current labeled labels of each sample, determine the target samples that need to be labeled corrected in the sample data set of the current round, and the estimated true labels of each target sample; the current labeled label is the category label actually labeled by the sample at present;

[0026] Correct the current annotated label of the target sample to the estimated true label, obtain the sample data set with the current round of label correction, and enter the next round of iteration as the updated sample data set of the current round until the iteration stop condition is met to obtain the target sample data set with the label correction.

[0027] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0028] Using the initial sample data set to be corrected as the sample data set of the current round;

[0029] In each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training;

[0030] Predicting the prediction probability of each sample in the sample data set of the current round under each category label by using the target model trained in the current round;

[0031] According to the predicted probabilities of each sample under each category label and the current labeled labels of each sample, determine the target samples that need to be labeled corrected in the sample data set of the current round, and the estimated true labels of each target sample; the current labeled label is the category label actually labeled by the sample at present;

[0032] Correct the current annotated label of the target sample to the estimated true label, obtain the sample data set with the current round of label correction, and enter the next round of iteration as the updated sample data set of the current round until the iteration stop condition is met to obtain the target sample data set with the label correction.

[0033] The label correction method, device, computer equipment, computer-readable storage medium and computer program product first use the initial sample data set to be corrected as the sample data set of the current round, and then in each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model trained in the current round, and the prediction probability of each sample in the sample data set of the current round under each category label is predicted through the target model trained in the current round, and the target samples that need to be label corrected in the sample data set of the current round and the estimated true label of each target sample are determined according to the prediction probability of each sample under each category label and the current labeled label of each sample, and the current labeled label of the target sample is corrected to the estimated true label to obtain the sample data set with the label correction completed in the current round, and enter the next round of iteration as the updated sample data set of the current round until the iteration stop condition is met, and the label correction is adaptively performed in a multi-round iterative manner, so that the universality and generalization are better, and no specific rules need to be formulated specifically for specific types of sample data, so that any type of sample data can be obtained through iterative label correction to obtain a more accurate corrected target sample data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A diagram showing an application environment of a label correction method in an embodiment;

[0036] Figure 2 is a flow chart of a label correction method in one embodiment;

[0037] Figure 3Basic classification models and parameter search strategies used for various types of sample data sets in one embodiment;

[0038] Figure 4 A schematic diagram of dividing sample subsets in one embodiment;

[0039] Figure 5 A schematic diagram of the meaning of each main symbol in an embodiment;

[0040] Figure 6 A schematic diagram of the overall process of a label correction method in one embodiment;

[0041] Figure 7 It is a schematic diagram of the overall flow of a specific iterative process of a label correction method in an embodiment;

[0042] Figure 8 A schematic diagram of a specific iterative process of each step in an embodiment;

[0043] Fig. 9 A schematic diagram of an interface of a cloud service based on a tag correction method in one embodiment;

[0044] Fig.10 is a schematic diagram of the format of a sample data set in one embodiment;

[0045] Fig.11 A schematic diagram of label calibration parameters to be specified in one embodiment;

[0046] Fig.12 is a structural block diagram of a label correction device in one embodiment;

[0047] Fig.13 is a structural block diagram of a label correction device in another embodiment;

[0048] Fig.14 is an internal structure diagram of a computer device in one embodiment;

[0049] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] The label correction method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can obtain the input initial sample data set to be corrected and send it to the server 104. The server 104 can execute the label correction method in each embodiment of the present application to obtain the target sample data set after the label correction. The server 104 can output the target sample data set to the terminal 102. Among them, the terminal 102 can be but not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0052] In some embodiments, Figure 2 As shown, a label correction method is provided, which is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0053] Step 202: Use the initial sample data set to be corrected as the sample data set of the current round.

[0054] Among them, the initial sample data set to be corrected is a sample data set with inaccurate labels that need to be corrected. The initial sample data set contains multiple samples and the annotation labels corresponding to each sample. The annotation label is the category label actually annotated for the sample. The category label is used to characterize the category to which the sample belongs. It can be understood that the category represented by the annotation label of the sample in the initial sample data set may not match the category to which the sample actually belongs, that is, the label is inaccurate, and therefore, the annotation label of the sample in the initial sample data set needs to be corrected. Using the initial sample data set to be corrected as the sample data set of the current round means using the initial sample data set to be corrected as the sample data set of the current round in the first round of iterations, and entering step 204.

[0055] In some embodiments, the type of the samples in the initial sample data set may be any one of text data, image data, audio data, video data, etc.

[0056] Step 204: In each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training.

[0057] Among them, the basic classification model is the classification model used to train the target model during the label correction process.

[0058] In some embodiments, the basic classification model used in each round of iteration may be a basic classification model for initializing model parameters. In this embodiment, the basic classification model with initialized model parameters is used to train the target model in each round of iteration, which can avoid the problem that the target model is inaccurate due to the influence of the sample data set with inaccurate labels in the previous round. In other embodiments, the basic classification model with initialized model parameters may be used only in the first round of iteration, and the basic classification model used in subsequent rounds of iteration may be the target model trained in the previous round, which is not limited to initializing the model parameters in each round.

[0059] In some embodiments, the server may perform a hyperparameter search in each round of iteration or only in the first round of iteration to obtain the optimal target hyperparameters, perform model training on the basic classification model based on the target hyperparameters, and obtain the target model completed in the current round of training.

[0060] In some embodiments, the basic classification model can be a lightweight model, which is convenient for parameter search and model training in each iteration of the label correction process, thereby improving the efficiency of label correction. In some embodiments, the server can determine a specific basic classification model based on the type of samples in the initial sample data set. Figure 3 As shown, when the type of samples in the initial sample data set is text data, the basic classification model can be a FastText model; when the type of samples in the initial sample data set is image data, the basic classification model can be a MobileNet model; when the type of samples in the initial sample data set is audio data, the basic classification model can be a VGGish model or a MusicNN model; when the type of samples in the initial sample data set is video data, the basic classification model can be a model obtained by combining lightweight models under multiple modalities.

[0061] In some embodiments, the initial sample data set may include an initial training set and an initial validation set. The server may use the initial training set as the training set of the current round in the first round of iterations, and then in each round of iterations, perform model training on the basic classification model according to the training set of the current round to obtain the target model trained in the current round, and then perform steps 206 to 210 for the training set of the current round and the validation set of the current round, respectively, to obtain the training set and validation set of the current round with completed label correction, and enter the next round of iterations as the updated training set and validation set of the current round, until the iteration stop condition is met, and the target training set and target validation set with completed label correction are obtained.

[0062] Step 206, using the target model trained in the current round, predict the prediction probability of each sample in the sample data set of the current round under each category label.

[0063] In some embodiments, the server may use the target model trained in the current round to predict the prediction probability of each sample in the current round of training set and the current round of validation set under each category label.

[0064] For example: The sample data set for the current round is The target model after the current round of training is Through the target model completed in the current round of training, predict the prediction probability of each sample in the current round of sample data set under each category label Among them, the predicted probability of the kth sample under category label j is The sample label is recorded as Then the corresponding sample subset with label i is The true label of the sample is denoted by y * = j. Due to the different understanding of data by labelers and the different difficulty of data labeling, the true label is often y. * = j samples are incorrectly labeled These mislabeled samples constitute different sample subsets Figure 4 (a) in FIG. 5 shows the division of these sample subsets. Figure 4 Each element X in (b) i，j Represents a sample subset Diagonal element X i，i Indicates that the sample's labeled label is consistent with the true label (i.e., there is no labeling error), and the off-diagonal element X i，j (i≠j) means that the samples that should be labeled with category label j are labeled with category label i (i.e., the incorrectly labeled sample subset). Usually, we only know the labeled labels of the samples but not the true labels of the samples. Therefore, we need to estimate the true labels of the samples, i.e., solve the sample subset Estimator The meanings of the main symbols in each embodiment of this application can be found in Figure 5 .

[0065] Step 208, based on the predicted probability of each sample under each category label and the current labeled label of each sample, determine the target samples that need to be labeled corrected in the current round of sample data set, as well as the estimated true label of each target sample; the current labeled label is the category label that the sample is currently actually labeled with.

[0066] The estimated true label is the true category label of the target sample estimated in the current round. It can be understood that in actual scenarios, the true label of the sample is unknown or difficult to obtain, so it is necessary to estimate it to obtain the estimated true label.

[0067] It can be understood that the category label actually labeled by the sample is the category label actually labeled by the sample after the current labeled label of the target sample is corrected to the estimated true label (ie, the target sample is labeled as the estimated true label) in the previous iteration.

[0068] In some embodiments, the server can determine the target samples that need to be labeled corrected in the current round of sample data sets based on the predicted probabilities of each sample under each category label and the current annotated labels of each sample, and determine the estimated true labels of the target samples based on the predicted probabilities of the target samples under each category label.

[0069] Step 210, correct the current annotated label of the target sample to the estimated true label, obtain the sample data set with the current round of label correction, and enter the next round of iteration as the updated sample data set of the current round until the iteration stop condition is met to obtain the target sample data set with the label correction.

[0070] In some embodiments, the server may set the issue flag corresponding to each determined target sample as a first flag, and set the issue flag of each sample that is not a target sample as a second flag. The server may correct the current label of each sample whose corresponding issue flag is the first flag to an estimated true label. In some embodiments, the first flag may be True (yes), and the second flag may be False (no).

[0071] In some embodiments, the server corrects the current annotated label of the target sample to the estimated true label, obtains a sample data set after the current round of label correction, and uses it as an updated sample data set for the current round. The server returns to execute step 204 and subsequent steps to iteratively perform label correction until the iteration stop condition is met, and uses the sample data set after the current round of label correction obtained in the last round as the target sample data set after label correction.

[0072] In some embodiments, after the server obtains the target sample data set after label correction, the terminal can respond to the download instruction for the target sample data set and download the target sample data set from the server. The user can deploy the downloaded target sample data set to the computer device used, and use the computer device to perform model training using the target sample data set to obtain a target classification model. The target classification model is a model used to perform classification tasks obtained by training the target sample data set after label correction.

[0073] It can be understood that the target model trained in each iteration and the target classification model trained using the target sample data set after label correction can be different models. The target model is a model used in the label correction process to predict the prediction probability of samples under each category label as the basis for label correction in each iteration; and the target classification model is a model trained using the target sample data set after label correction (i.e., a clean sample data set without noise) for performing classification tasks. Therefore, the model structures of the target model and the target classification model may be inconsistent, but they may also be consistent in structure without limitation.

[0074] In some embodiments, the target model can be a lightweight type of model, which facilitates parameter search and model training in each iteration of the label correction process, thereby improving the efficiency of label correction; and the target classification model can be a more complex type of model, which facilitates more accurate performance of classification tasks.

[0075] like Figure 6 As shown, it is a schematic diagram of the overall process of the label correction method in each embodiment of the present application. First, an initial sample data set with noise is input, and then parameter search, model training, confidence learning and label correction are iteratively performed in sequence to perform iterative label correction on the samples in the initial sample data set. After the iteration stops, clean data (i.e., the target sample data set) is output, and the target classification model can be trained based on the clean data.

[0076] like Figure 7 As shown, it is a flowchart of the specific iterative process of the label correction method in each embodiment of the present application. In the first round of iteration, it is executed based on the initial sample data set, and in the subsequent iterations, it is executed based on the sample data set after the previous round of correction. First, parameter search is performed, and then model training is performed. After model training, it is determined whether the iteration stop condition is met to determine whether to stop the iteration. If the iteration is not stopped, confidence learning and label correction are performed to complete a round of iteration and enter the next round of iteration. Until the iteration is stopped, the target sample data set (i.e., clean data) is output, and the target classification model is trained according to the target sample data set.

[0077] The above-mentioned label correction method first uses the initial sample data set to be corrected as the sample data set of the current round, and then in each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model trained in the current round. The target model trained in the current round is used to predict the prediction probability of each sample in the sample data set of the current round under each category label. According to the prediction probability of each sample under each category label and the current labeled label of each sample, the target samples that need to be labeled corrected in the sample data set of the current round and the estimated true label of each target sample are determined, and the current labeled label of the target sample is corrected to the estimated true label. The real label is obtained, and the sample data set after the label correction of the current round is obtained. As the updated sample data set of the current round, it enters the next round of iteration until the iteration stop condition is met. The label correction is adaptively performed in a multi-round iterative manner, so that the universality and generalization are better. There is no need to formulate specific rules for specific types of sample data. It can make it possible to obtain a more accurate target sample data set through iterative label correction for any type of sample data, avoiding the problem that the label of the sample data set after a single label correction is inaccurate, unstable and easy to introduce new noise, so that the target sample data set obtained after the label correction is more accurate. Moreover, the label correction is fully automatic, and there is no need to manually process the data in the intermediate process, which improves the efficiency of label correction. In addition, as an independent framework, this method does not need to be deeply embedded in the classification module of other downstream tasks, does not rely on the data feature form of the sample data set and the network structure of the target classification model, and can be deployed as a cloud service to provide support for label correction of various types of sample data sets for classification, such as text, image, audio and video.

[0078] In some embodiments, in each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training, including: in the first round of iteration, according to the sample data set of the current round and the basic classification model, hyperparameter search is iteratively performed to obtain target hyperparameters; in each round of iteration, based on the target hyperparameters, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training.

[0079] Among them, hyperparameter search is the process of automatically searching for the optimal hyperparameter suitable for the basic classification model. The target hyperparameter is the optimal hyperparameter found by the hyperparameter search.

[0080] It can be understood that the basic classification model used in the hyperparameter search process is consistent with the model structure of the basic classification model used in model training in each round of iteration.

[0081] Different basic classification models may need to search for different hyperparameters. In some embodiments, the hyperparameters may include at least one of the number of training rounds, learning rate, hidden layer dimension, number of hidden layers, or loss function. Figure 8 The specific iterative process of each step of parameter search, model training, confidence learning and label correction is shown in Figure 1. For example: Figure 8 The target hyperparameters searched in the parameter search (i.e., hyperparameter search) include: training_epoch is 25, learning_rate is 0.095, hidden layer dimension hidden_dim is 256, hidden layer number num_layers is 8, and loss function loss_function is ce_loss. Then, the optimal hyperparameters are used to train the basic classification model to obtain the target model.

[0082] In some embodiments, the hyperparameter search may adopt a hyperparameter search framework such as HyperOpt, UltraOpt, or Ray-tune.

[0083] In some embodiments, Figure 3 As shown, when the type of samples in the sample data set is image data, audio data, or video data, etc., the server can perform hyperparameter search only in the first round of iteration to obtain the target hyperparameters, and perform model training based on the target hyperparameters searched in the first round in subsequent iterations.

[0084] In the above embodiment, since the amount of sample data sets in modalities such as image data, audio data, or video data is large, and it takes a long time to use these data for parameter search and model training, hyperparameter search is performed only in the first round of iteration, thereby reducing the time consumption caused by hyperparameter search and improving the efficiency of label correction. In addition, automatic hyperparameter search avoids the problems of low efficiency and complex operation caused by manual adjustment of hyperparameters, improves the efficiency of label correction, and can perform label correction fully automatically without manual intervention.

[0085] In some embodiments, in each round of iteration, model training is performed on the basic classification model according to the sample data set of the current round to obtain the target model trained in the current round, including: in each round of iteration, hyperparameter search is iteratively performed according to the sample data set of the current round and the basic classification model to obtain the target hyperparameters of the current round; based on the target hyperparameters of the current round, model training is performed on the basic classification model according to the sample data set of the current round to obtain the target model trained in the current round.

[0086] In some embodiments, Figure 3As shown, when the type of samples in the sample data set is text data, image data, audio data or video data, the server can perform hyperparameter search in each round of iteration to obtain the target hyperparameters of the current round, and perform model training based on the target hyperparameters of the current round.

[0087] In the above embodiment, a hyperparameter search is performed in each iteration, so that the target model obtained by model training is more accurate, thereby improving the accuracy of label correction. In addition, the automatic hyperparameter search avoids the problems of low efficiency and complex operation caused by manual adjustment of hyperparameters, improves the efficiency of label correction, and can perform label correction fully automatically without manual intervention.

[0088] In some embodiments, before correcting the current labeled labels of the target samples to the estimated true labels to obtain the sample data set for the current round of label correction, the method also includes: determining the label quality score of the current labeled labels of each target sample according to the predicted probability of each target sample under each category label; determining the actual correction number of the current round according to a preset correction ratio and the number of target samples; selecting from the target samples, in order of label quality scores from low to high, the target samples that meet the actual correction number of the current round as samples to be corrected; correcting the current labeled labels of the target samples to the estimated true labels to obtain the sample data set for the current round of label correction, including: correcting the current labeled labels of the samples to be corrected to the estimated true labels to obtain the sample data set for the current round of label correction.

[0089] The preset correction ratio is the ratio of label correction required for target samples in each round of iteration. The label quality score (label_quality) is used to characterize the quality of the current annotation label of the sample.

[0090] In some embodiments, the terminal may obtain the input preset correction ratio and the initial sample data set, and send the preset correction ratio and the initial sample data set to the server.

[0091] In some embodiments, the server may determine the actual number of corrections based on the product of the preset correction ratio and the number of target samples. For example, if the number of target samples is n', and the preset correction ratio is correct_ratio, then the actual number of corrections in the current round n" = n'*correct_ratio. The server may select n" target samples as samples to be corrected in order of label quality scores from low to high.

[0092] In some embodiments, the server may sort the target samples in descending order according to the label quality scores, and select the target samples that meet the actual correction quantity of the current round at the front as the samples to be corrected. In other embodiments, the server may sort the target samples in descending order according to the label quality scores, and select the target samples that meet the actual correction quantity of the current round at the back as the samples to be corrected.

[0093] In some embodiments, the server may correct the current annotated label of the sample to be corrected to the estimated true label, and obtain a sample data set with completed label correction in the current round. Samples other than the sample to be corrected in the target sample are not corrected.

[0094] In the above embodiment, samples to be corrected that meet the preset correction ratio are selected from low to high according to the label quality scores of the target samples for label correction, thereby avoiding the problem of overfitting caused by correcting all target samples and improving the accuracy of label correction.

[0095] In some embodiments, the label quality score of the current labeled label of each sample is determined according to the predicted probability of each sample under each category label, including: for each category label, according to the average of the predicted probabilities of each sample whose current labeled label is the category label, obtaining the probability threshold value corresponding to the category label; for each sample, according to the difference between the predicted probability of the sample under the current labeled label and the probability threshold value corresponding to the current labeled label, determining the label quality score of the current labeled label of the sample.

[0096] In some embodiments, the server may obtain a probability threshold value corresponding to a category label based on the ratio of the sum of predicted probabilities of samples currently labeled as category labels under the category label to the number of samples currently labeled as category labels.

[0097] In some embodiments, the corresponding probability threshold value of each category label can be calculated using the following formula:

[0098]

[0099] Among them, t j Represents the probability threshold corresponding to category label j. Indicates the current annotation label is the sample subset composed of each sample of j, Represents a sample subset The number of samples in . Indicates the current annotation label is a sample of j. Indicates the target model that has completed the current round of training The predicted probability of the predicted sample x under the category label j is equivalent to It can be understood that the meaning of the above formula is that the probability threshold value corresponding to category label j is equal to the sum of the predicted probabilities of the samples with the current label j under category label j divided by the number of samples with the current label j, that is, the probability threshold value corresponding to category label j is equal to the mean of the predicted probabilities of the samples with the current label j under category label j.

[0100] In some embodiments, the server can determine the difference between the predicted probability of the sample under the current annotation label and the probability threshold value corresponding to the current annotation label, and then normalize the difference between the predicted probability of the sample under the current annotation label and the probability threshold value corresponding to the current annotation label to obtain the label quality score of the current annotation label of the sample.

[0101] In some embodiments, the server may add an offset based on the difference between the predicted probability of the sample under the current label and the probability threshold value corresponding to the current label to obtain a target difference, and then normalize the target difference to obtain a label quality score of the current label of the sample. In this embodiment, adding an offset based on the difference can prevent the difference from being negative.

[0102] In some embodiments, the offset may be the maximum value of the probability threshold values ​​corresponding to each category label. That is, the offset may be expressed as α=max(Γ), where α represents the offset. Γ represents the probability threshold value t corresponding to each category label. j Assuming there are m category labels, the vector

[0103] In other embodiments, the offset may be a preset scalar.

[0104] In the above embodiment, for each category label respectively, the probability threshold value corresponding to the category label is obtained according to the mean value of the predicted probability of each sample whose current label is the category label under the category label, and for each sample respectively, according to the difference between the predicted probability of the sample under the current label label and the probability threshold value corresponding to the current label label, the label quality score of the current label label of the sample can be accurately determined, and then the sample to be corrected can be determined more accurately, thereby improving the accuracy of label correction.

[0105] In some embodiments, for each sample respectively, the label quality score of the current labeled label of the sample is determined according to the difference between the predicted probability of the sample under the current labeled label and the probability threshold value corresponding to the current labeled label, including: for each sample respectively, determining the difference between the predicted probability of the sample under each category label and the probability threshold value corresponding to the category label; normalizing the difference between the predicted probability of the sample under the current labeled label and the probability threshold value corresponding to the current labeled label according to the difference corresponding to each category label, to obtain the label quality score of the current labeled label of the sample.

[0106] In some embodiments, the server can determine the sum of the differences corresponding to each category label for each sample respectively, obtain the total difference, and then calculate the ratio between the difference corresponding to the sample under the current labeled label and the total difference to achieve normalization and obtain the label quality score of the current labeled label of the sample.

[0107] In some embodiments, the server can add an offset to the difference values ​​corresponding to each category label for each sample, obtain the target difference values ​​corresponding to each category label, and then normalize the target difference values ​​corresponding to the sample under the current label label according to the target difference values ​​corresponding to each category label, to obtain the label quality score of the current label label of the sample. The target difference values ​​corresponding to sample k under each category label can be calculated using the following formula:

[0108]

[0109] in, Represents a vector consisting of the target difference values ​​corresponding to sample k under each category label. represents the vector composed of the predicted probabilities of sample k under each category label. Γ represents the probability threshold value t corresponding to each category label j A vector composed of . α represents the offset, is a scalar.

[0110] In some embodiments, the server may determine the sum of the target differences corresponding to each category label, obtain the sum of the differences, and then calculate the ratio between the target difference corresponding to the current label and the sum of the differences to achieve normalization and obtain the label quality score of the current label of the sample. The following formula may be used to normalize the target differences corresponding to each category label of the sample:

[0111]

[0112] in, represents a vector consisting of the target differences corresponding to sample k under each category label. It represents the sum of the target differences corresponding to the sample k in each category label, and sum is the summation function. represents a vector consisting of the normalized results of the target differences corresponding to sample k under each category label. The following formula can be used according to Determine the label quality score of the current annotation label of the sample:

[0113]

[0114] in, Represents the current annotation label of sample x The label quality score of . Representation vector The i-th element in .

[0115] In the above embodiment, the difference between the predicted probability of the sample under the current labeled label and the probability threshold value corresponding to the current labeled label is normalized according to the difference values ​​corresponding to each category label, so as to obtain the label quality score of the current labeled label of the sample, thereby enabling the label quality scores of the samples under each category label to be under the same standard, making the label quality scores more reliable and accurate.

[0116] In some embodiments, target samples that need label correction in the current round of sample data set and the estimated true label of each target sample are determined based on the predicted probability of each sample under each category label and the current labeled label of each sample, including: determining the target samples that need label correction in the current round of sample data set based on the predicted probability of each sample under each category label and the current labeled label of each sample; determining the estimated true label of the target sample based on the predicted probability of the target sample under each category label.

[0117] In some embodiments, the maximum predicted probability of each sample under each category label is determined, and the category label corresponding to the maximum predicted probability is determined as the estimated true label of the sample. In other embodiments, other methods can also be used to determine the estimated true label of the target sample, without limitation.

[0118] In the above embodiment, the target samples that need to be label corrected in the current round of sample data set are determined based on the predicted probability of each sample under each category label and the current annotated label of each sample; the estimated true label of the target sample is determined based on the predicted probability of the target sample under each category label.

[0119] In some embodiments, target samples that need label correction in the current round of sample data sets are determined based on the predicted probability of each sample under each category label and the current annotated label of each sample, including: determining the predicted label of each sample based on the predicted probability of each sample under each category label; determining the target samples that need label correction in the current round of sample data sets based on the predicted label of each sample and the current annotated label.

[0120] Among them, the predicted label is a category label estimated based on the predicted probability and used as a reference in the process of determining the target sample. It can be understood that the predicted label of the sample and the estimated true label of the sample may be the same or different, depending on the way the predicted label and the estimated true label are determined. In the case where the predicted label and the estimated true label are both the category label corresponding to the maximum predicted probability among the predicted probabilities of the sample under each category label, the predicted label and the estimated true label are the same.

[0121] In some embodiments, for each sample, the server may determine the maximum predicted probability of the sample under each category label, and determine the category label corresponding to the maximum predicted probability as the predicted label of the sample. In other embodiments, for each sample, the server may first determine a target predicted probability that is greater than or equal to a probability threshold value corresponding to the category label, and determine the category label corresponding to the maximum predicted probability in the target predicted probability as the predicted label of the sample.

[0122] In some embodiments, the server may determine all samples whose predicted labels are inconsistent with the current labeled labels as target samples. In other embodiments, the server may, based on a preset retention amount, not use samples whose predicted labels are consistent with the current labeled labels under each category label, and selected samples from samples where the current labeled labels are inconsistent with the predicted labels as target samples, and use the remaining samples from samples where the current labeled labels are inconsistent with the predicted labels, excluding the selected samples, as target samples. The total number of samples whose predicted labels are consistent with the current labeled labels under each category label, and selected samples from samples where the current labeled labels are inconsistent with the predicted labels, meets the preset retention amount.

[0123] In the above embodiment, the predicted label of each sample is determined according to the predicted probability of each sample under each category label, and then the target sample that needs to be label corrected in the current round of sample data set is determined according to the predicted label of each sample and the current annotation label, so that the target sample that needs to be label corrected in the current round of sample data set can be determined efficiently and accurately.

[0124] In some embodiments, the predicted label of each sample is determined according to the predicted probability of each sample under each category label, including: for each sample, determining a target predicted probability from the predicted probabilities of the sample under each category label; the target predicted probability is a predicted probability greater than or equal to a probability threshold value corresponding to the corresponding category label; and determining the category label corresponding to the maximum predicted probability in the target predicted probability as the predicted label of the sample.

[0125] In some embodiments, when the number of target prediction probabilities is one (i.e., the predicted probability of the sample under only one category label is greater than or equal to the probability threshold value corresponding to the category label), the server can directly determine the category label corresponding to the target prediction probability as the predicted label of the sample.

[0126] In other embodiments, when the number of target prediction probabilities is multiple (i.e., the prediction probability of the sample under multiple category labels is greater than or equal to the probability threshold value corresponding to the category label), the server may determine the category label corresponding to the maximum prediction probability among the multiple target prediction probabilities as the prediction label of the sample.

[0127] In some embodiments, the server may divide the samples with the same current annotation label and the same predicted label into the same sample subset to obtain the sample subset Estimator When the number of target prediction probabilities is one (i.e., the prediction probability of the sample under only one category label is greater than or equal to the probability threshold corresponding to the category label), When the number of target prediction probabilities is multiple (i.e., the prediction probability of the sample under multiple category labels is greater than or equal to the probability threshold corresponding to the category label),

[0128]

[0129]

[0130] In the above embodiment, for each sample, a target prediction probability that is greater than or equal to the probability threshold value corresponding to the corresponding category label is determined from the prediction probability of the sample under each category label, and the category label corresponding to the maximum prediction probability in the target prediction probability is determined as the prediction label of the sample, which can more accurately determine the prediction label of each sample.

[0131] In some embodiments, the method further includes: if the predicted probability of the sample under each category label is less than the probability threshold value corresponding to the category label, no processing is performed.

[0132] In some embodiments, if the predicted probability of a sample under each category label is less than the probability threshold corresponding to the category label (i.e., the predicted probability of the sample under each category label does not contain the target predicted probability), the server does not process it, i.e., the server does not determine the predicted label of the sample. It can be understood that because the sample has no predicted label, the sample is not divided into any sample subset and is not used as a target sample. Under each category label j Then the sample does not belong to any sample subset

[0133] In the above embodiment, if the predicted probability of a sample under each category label is less than the probability threshold value corresponding to the category label, indicating that the true label of the sample is difficult to determine, the sample will not be classified as a target sample, thereby avoiding the problem of inaccurate label correction and improving the accuracy of label correction.

[0134] In some embodiments, based on the predicted label of each sample and the current labeled label, the target samples in the current round of sample data set that need label correction are determined, including: for each sample, if the predicted label of the sample is inconsistent with the current labeled label, the sample is determined to be the target sample in the current round of sample data set that needs label correction.

[0135] In some embodiments, the server may divide all samples with the same current annotation label and the same predicted label into the same sample subset, and use samples in each sample subset with the predicted label inconsistent with the current annotation label as target samples.

[0136] For example, assuming there are m category labels, the server can divide the samples with the same current annotation label and the same predicted label into the same sample subset to obtain the sample subset. Estimator That is, we get m×m=m 2 Sample subset The server can divide the m sample subsets The samples other than m in (i.e., samples whose predicted labels are consistent with the current labeled labels) 2 -m samples (i.e., samples whose predicted labels are inconsistent with the current labeled labels) are used as target samples.

[0137] In some embodiments, the server may store samples in each sample subset whose predicted labels are inconsistent with the current labeled labels (except for m sample subsets) m outside the sample 2 -m samples) is set as the first flag, and the samples in each sample subset that have the same predicted label as the current label (m sample subset) are The problem mark corresponding to the sample in ( ) is set as the second mark.

[0138] In the above embodiment, if the predicted label of a sample is inconsistent with the current annotated label, the sample is determined to be a target sample in the current round of sample data sets that needs label correction, so that the target sample can be determined efficiently.

[0139] In some embodiments, target samples that need label correction in the current round of sample data sets are determined based on the predicted labels and current labeled labels of each sample, including: dividing samples whose current labeled labels and predicted labels are the same category label into target sample subsets corresponding to the category label; if the number of samples in the target sample subset is greater than or equal to a preset retention amount, then samples whose current labeled labels are the category label corresponding to the target sample subset and whose predicted labels are not the category label corresponding to the target sample subset are determined as target samples that need label correction; the preset retention amount is the minimum number of samples that do not require label correction under a preset single category label; if the number of samples in the target sample subset is less than the preset retention amount, then from each sample whose current labeled label is the category label corresponding to the target sample subset, samples that meet the target correction amount are selected in ascending order of the predicted probability of the samples under the category label as target samples that need label correction; the target correction amount is determined based on the difference between the number of samples whose current labeled label is the category label corresponding to the target sample subset and the preset retention amount.

[0140] In some embodiments, the server may divide all samples with the same current label and predicted label into the same sample subset. The sample subset to which the current label and predicted label are the same category label (i.e., the sample with the same current label and predicted label) are divided is the target sample subset. For example: the m sample subsets in the above example This is the target sample subset.

[0141] In some embodiments, if the number of samples in the target sample subset is greater than or equal to a preset retention amount, the samples whose current annotation labels are the category labels corresponding to the target sample subset and whose predicted labels are not the category labels corresponding to the target sample subset are determined as target samples requiring label correction. That is, the number of samples in the target sample subset is greater than or equal to the preset retention amount, indicating that the number of samples in the target sample subset is sufficient to meet the preset retention amount, then the samples whose current annotation labels are the category labels corresponding to the target sample subset and whose predicted labels are not the category labels corresponding to the target sample subset (that is, samples whose predicted labels under the category labels corresponding to the target sample subset are inconsistent with the current annotation labels) are all determined as target samples requiring label correction.

[0142] In some embodiments, if the number of samples in the target sample subset is less than the preset retention amount, the server can determine the corresponding target correction amount based on the difference between the number of samples whose current label is the category label corresponding to the target sample subset and the preset retention amount, and then select samples that meet the target correction amount from each sample whose current label is the category label corresponding to the target sample subset in order of the predicted probability under the category label from small to large as the target samples that need to be label corrected. Maximum predicted probability

[0143] In some embodiments, the server may sort the samples whose current label is the category label corresponding to the target sample subset in descending order according to the predicted probability under the category label, and select the sample that meets the target correction amount at the front as the target sample that needs label correction. In other embodiments, the server may sort the samples whose current label is the category label corresponding to the target sample subset in descending order according to the predicted probability under the category label, and select the sample that meets the target correction amount at the end as the target sample that needs label correction.

[0144] In the above embodiment, according to the preset retention amount, when the number of samples in the target sample subset is sufficient to meet the preset retention amount, all samples whose current labeled labels are the category labels corresponding to the target sample subset and whose predicted labels are not the category labels corresponding to the target sample subset are determined as target samples that need to be label corrected. When the number of samples in the target sample subset is less than the preset retention amount, from each sample whose current labeled label is the category label corresponding to the target sample subset, samples that meet the corresponding target correction amount are selected in order from small to large according to the predicted probability under the category label as the target samples that need to be label corrected, thereby retaining a certain amount of samples under each category label without label correction, avoiding overfitting, and improving the accuracy of label correction.

[0145] In some embodiments, the method also includes: dividing all samples with the same current annotation label and the same predicted label into the same sample subset; the sample subset into which the samples with the current annotation label and the predicted label being the same category label are divided is the target sample subset; generating a confidence joint matrix based on the number of samples in each sample subset; each element in the confidence joint matrix represents the number of samples in each sample subset; and determining the number of samples in each target sample subset based on the confidence joint matrix.

[0146] In some embodiments, the confidence union matrix The elements in can be expressed as:

[0147]

[0148] in, Represents the confidence joint matrix The elements in Represents a sample subset The number of samples in, that is, the current annotation label is And the predicted label is The number of samples. The confidence joint matrix The diagonal elements in Indicates the current annotation label With predicted labels The number of consistent samples, that is, the number of samples in the target sample subset.

[0149] In some embodiments, the server may determine the number of samples in each target sample subset based on the values ​​of the diagonal elements in the confidence joint matrix. For example, the server may determine the vector composed of each diagonal element in the confidence joint matrix This vector is used to represent the number of samples in each target sample subset.

[0150] In some embodiments, for each target sample subset, the server may determine the maximum value between the number of samples in the target sample subset and the preset retention amount, and determine the target correction amount corresponding to the target sample subset according to the difference between the number of samples whose current label is the category label corresponding to the target sample subset and the maximum value. The server may select samples that meet the corresponding target correction amount from each sample whose current label is the category label corresponding to the target sample subset, in order from small to large in the order of the predicted probability of the sample under the category label, as target samples that need to be label corrected. It can be understood that in this embodiment, by determining the maximum value between the number of samples in the target sample subset and the preset retention amount, the processing of the two cases in which the number of samples in the target sample subset in the aforementioned embodiment is greater than or equal to the preset retention amount, and the number of samples in the target sample subset is less than the preset retention amount can be covered.

[0151] In some embodiments, the server may vector The maximum value between each element in and the preset retention amount n constitutes the target vector

[0152] In some embodiments, the server may select the current annotation label from each current sample data set for each category label i as The samples constitute the sample subset Get the number of samples whose current annotation label is the label of this category Then the sample subset The number of samples that need to be retained (i.e., samples that do not need label correction) is nremain =d′[i], then the sample subset The number of target samples that need to be label-corrected in (i.e., the number of target corrections corresponding to the category label) is:

[0153]

[0154] The server can then The predicted probability of the sample in class label i In order from small to large, samples that meet the corresponding target correction amount are selected in sequence (i.e., n canlibrate samples) as the target samples that need label correction in the current round.

[0155] In the above embodiment, by generating a confidence joint matrix, determining the number of samples in each target sample subset, and then determining the target correction amount, label correction can be performed efficiently.

[0156] In some embodiments, the estimated true label of the target sample is determined based on the predicted probability of the target sample under each category label, including: for each sample, determining the maximum predicted probability among the predicted probabilities of the sample under each category label, and determining the category label corresponding to the maximum predicted probability as the estimated true label of the sample.

[0157] In some embodiments, the estimated true label y of the sample can be determined using the following formula: * :

[0158]

[0159] Among them, y * Represents the estimated true label. Indicates the target model that has completed the current round of training The predicted probability of sample x under class label j. It represents the maximum value of the predicted probability of each category label of sample x. Represents a vector consisting of the predicted probabilities of sample k under each category label. Representation vector The maximum value among the elements in .

[0160] In the above embodiment, the category label corresponding to the maximum predicted probability is determined as the estimated true label of the sample, which can efficiently and relatively accurately determine the estimated true label of the sample.

[0161] In some embodiments, satisfying the iteration stop condition includes satisfying any one of a first iteration stop condition and a second iteration stop condition; the first iteration stop condition is that the current number of iterations is greater than or equal to a first preset number of iterations; the second iteration stop condition is that the optimization degree of the performance indicator of the target model in the current round is less than or equal to a preset indicator threshold value; the optimization degree of the performance indicator of the target model in the current round is used to characterize the optimization degree of the performance indicator of the target model completed in the current round of training compared with the target model completed in the previous round of training.

[0162] The first preset number of iterations is the maximum threshold value of the preset number of iterations of label correction. The preset index threshold value is the minimum threshold value of the optimization degree of the preset performance index of the target model. The performance index is used to characterize the performance of the target model.

[0163] In some embodiments, the performance indicator may be any one of precision (P), recall (R), and F1 measure, etc. Among them, the F1 measure is determined according to the precision and recall.

[0164] In some embodiments, the server may determine the performance indicator of the target model in the current round in each iteration, and determine the degree of optimization of the performance indicator of the target model in the current round based on the difference between the performance indicator of the target model in the current round and the performance indicator of the target model in the previous round.

[0165] The iteration stops when any of the following conditions is met:

[0166] (1) step>max_epoch, that is, the current number of iterations step is greater than or equal to the first preset number of iterations max_epoch.

[0167] (2)F1 t -F1 t-1 <delta_score, that is, the optimization degree of the performance index of the target model in the current round F1 t -F1 t-1 Less than or equal to the preset indicator threshold delta_score.

[0168] In the above embodiment, label correction is adaptively performed in a multi-round iterative manner until the current number of iterations is greater than or equal to the first preset number of iterations or the optimization degree of the performance indicator of the target model in the current round is less than or equal to the preset indicator threshold value, thereby stopping the iteration. This avoids the problem that the labels of the sample data set are unstable and easy to introduce new noise after a single label correction, and makes the target sample data set after the label correction more accurate.

[0169] In some embodiments, the sample data set of the current round includes the training set of the current round and the validation set of the current round; the sample data set of the current round is used to perform model training on the basic classification model; and the validation set of the current round is used to determine the performance indicators of the target model completed by the previous round of training.

[0170] In some embodiments, the initial sample data set includes an initial training set and an initial validation set. Accordingly, in each round of iteration, the sample data set of the current round includes a training set of the current round and a validation set of the current round. The server can iteratively perform label correction on the training set of the current round and the validation set of the current round in each round of iteration, respectively, and perform model training on the basic classification model according to the training set of the current round in each round of iteration to obtain the target model completed by the current round of training, and use the validation set of the current round to determine the performance indicators of the target model completed by the previous round of training.

[0171] In the above embodiment, the sample data set of the current round includes the training set of the current round and the verification set of the current round, so that model training can be performed to obtain the target model, and the performance indicators of the target model can be determined, so as to determine whether to stop the iteration based on the performance indicators, thereby providing a good iteration solution.

[0172] In some embodiments, the method also includes: recording the number of times each sample is corrected by the label in each round of iteration; for each sample, when the proportion of the number of corrections of the sample is greater than or equal to a preset proportion threshold, the sample is determined to be an unstable sample; the proportion of corrections is the ratio between the number of times the sample is corrected by the label and the number of iterations; and outputting relevant information of the unstable sample.

[0173] In some embodiments, a dictionary may be set to record the number of times each sample is corrected by the label in each round of iteration and the updated label after each correction. The server may record the number of times each sample is corrected by the label in each round of iteration and the updated label after each correction in the dictionary.

[0174] In some embodiments, the server may determine the proportion of the number of corrections of the sample based on the ratio between the number of corrections and the number of iterations of the sample, and determine the sample as an unstable sample when the proportion of the number of corrections of the sample is greater than or equal to a preset proportion threshold. For example, the preset proportion threshold may be 0.5, and when the ratio between the number of corrections of the sample and the number of iterations is greater than or equal to 0.5, the sample is determined to be an unstable sample, and the set of unstable samples is X′={x|N(x)>0.5*max_epoch}, where N(x) represents the number of corrections of sample x, and max_epoch represents the number of iterations.

[0175] In the above embodiments, due to different designs of the labeling system in different business scenarios, some category labels may be semantically close to each other, resulting in some samples being in the fuzzy area of ​​two or more category labels. When performing label correction, the labels of these samples are easily corrected repeatedly (that is, these samples are unstable). For example, the text "eat vegetables" is sometimes labeled as "healthy" and sometimes labeled as "diet", and the label is repeatedly corrected. Therefore, by recording the number of times each sample is corrected by the label in each round of iteration, when the proportion of the number of corrections of the sample is greater than or equal to the preset proportion threshold, the sample is determined to be an unstable sample, and the relevant information of the unstable sample is output. This is conducive to evaluating whether the design of the labeling system for the classification task in a specific business scenario is reasonable based on the relevant information of the unstable sample, and formulating corresponding optimization measures.

[0176] In some embodiments, Fig. 9 As shown, the label correction method in each embodiment of the present application can be deployed as a cloud service, which can be Fig. 9 The terminal can upload the initial sample data set (training set and verification set) through the interface in the terminal, and the terminal can respond to the trigger operation of "start correction" to instruct the server to start label correction. The server can automatically perform label correction on the initial sample data set. In response to the trigger operation of "download results", the terminal can download the corrected target sample data set from the server. In addition, the log of label correction can also be downloaded through the terminal.

[0177] In some embodiments, for a sample data set of text type, the text content and the corresponding annotation tags can be specified in the uploaded file. For a sample data set of image type, the storage address and the corresponding annotation tags of each image can be specified in the uploaded file. For a sample data set of audio type, the storage address and the corresponding annotation tags of the spectrum data of each audio can be specified in the uploaded file. The address for storing the spectrum data of the image or audio can be a storage address of a file system such as cfs or ceph, and it is necessary to ensure that the cloud service has the permission to read the storage address of the file system.

[0178] The format of the sample dataset of uploaded text type can be found in Fig.10 For (a) in the example, the formats of the uploaded image and audio data sets can be found in Fig.10 (b) in .

[0179] from Fig. 9 It can be seen that the type of the initial sample data set (ie, text, image, voice, etc.) can also be selected, so that the server can determine the corresponding basic classification model according to the type of the initial sample data set.

[0180] In some embodiments, the terminal can be Fig. 9 The interface shown in the figure obtains the input label correction parameters. Fig.11 As shown, the label correction parameters may include the number of iterations max_epoch required for label correction, the number of iterations search_step required for parameter search, a preset indicator threshold delta_score, a preset correction ratio correct_ratio, and a performance indicator model_metric of the target model.

[0181] In the above embodiment, label correction can be performed fully automatically only by initially setting label correction parameters and uploading an initial sample data set, without the need for manual processing of intermediate data, thereby improving label correction efficiency.

[0182] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0183] Based on the same inventive concept, the embodiment of the present application also provides a label correction device for implementing the label correction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the label correction device provided below can refer to the limitations of the label correction method above, and will not be repeated here.

[0184] In one embodiment, Fig.12 As shown, a label correction device 1200 is provided, comprising: a model training module 1202, a confidence learning module 1204 and a label correction module 1206, wherein:

[0185] The model training module 1202 is used to use the initial sample data set to be corrected as the sample data set of the current round; in each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training.

[0186] The confidence learning module 1204 is used to predict the prediction probability of each sample in the current round of sample data set under each category label through the target model completed in the current round of training; according to the prediction probability of each sample under each category label and the current labeled label of each sample, determine the target samples that need to be labeled corrected in the current round of sample data set, as well as the estimated true label of each target sample; the current labeled label is the category label that is actually labeled with the sample.

[0187] The label correction module 1206 is used to correct the current annotated label of the target sample to the estimated true label, obtain the sample data set after the current round of label correction, and use it as the updated sample data set of the current round to enter the next round of iteration until the iteration stop condition is met to obtain the target sample data set after the label correction.

[0188] In some embodiments, Fig.13 As shown, the label correction device 1200 also includes:

[0189] The parameter search module 1208 is used to iteratively perform hyperparameter search in the first round of iterations based on the sample data set and the basic classification model of the current round to obtain the target hyperparameter.

[0190] The model training module 1202 is also used to perform model training on the basic classification model based on the target hyperparameters and the sample data set of the current round in each iteration to obtain the target model completed by the current round of training.

[0191] In some embodiments, the parameter search module 1208 is also used to iteratively perform hyperparameter search in each iteration according to the sample data set and the basic classification model of the current round to obtain the target hyperparameter of the current round. The model training module 1202 is also used to perform model training on the basic classification model according to the sample data set of the current round based on the target hyperparameter of the current round to obtain the target model trained in the current round.

[0192] In some embodiments, the confidence learning module 1204 is further used to determine the label quality score of the current annotated label of each target sample according to the predicted probability of each target sample under each category label. The label correction module 1206 is further used to determine the actual correction quantity of the current round according to the preset correction ratio and the number of target samples; from the target samples, select the target samples that meet the actual correction quantity of the current round in order from low to high in the order of label quality scores as samples to be corrected; correct the current annotated label of the sample to be corrected to the estimated true label, and obtain the sample data set with the label correction completed in the current round.

[0193] In some embodiments, the confidence learning module 1204 is also used to obtain, for each category label respectively, a probability threshold value corresponding to the category label based on the mean of the predicted probabilities of each sample whose current label is the category label under the category label; and for each sample respectively, determine the label quality score of the current label label of the sample based on the difference between the predicted probability of the sample under the current label label and the probability threshold value corresponding to the current label label.

[0194] In some embodiments, the confidence learning module 1204 is also used to determine, for each sample, the difference between the predicted probability of the sample under each category label and the probability threshold value corresponding to the category label; based on the differences corresponding to each category label, the difference between the predicted probability of the sample under the current labeled label and the probability threshold value corresponding to the current labeled label is normalized to obtain a label quality score for the current labeled label of the sample.

[0195] In some embodiments, the confidence learning module 1204 is also used to determine the target samples that need to be label corrected in the current round of sample data sets based on the predicted probabilities of each sample under each category label and the current annotated labels of each sample; and determine the estimated true labels of the target samples based on the predicted probabilities of the target samples under each category label.

[0196] In some embodiments, the confidence learning module 1204 is also used to determine the predicted label of each sample based on the predicted probability of each sample under each category label; based on the predicted label of each sample and the current annotation label, determine the target sample that needs to be labeled corrected in the current round of sample data set.

[0197] In some embodiments, the confidence learning module 1204 is also used to determine, for each sample, a target prediction probability from the prediction probabilities of the sample under each category label; the target prediction probability is a prediction probability that is greater than or equal to a probability threshold value corresponding to the corresponding category label; and the category label corresponding to the maximum prediction probability in the target prediction probability is determined as the prediction label of the sample.

[0198] In some embodiments, the confidence learning module 1204 is further configured to not process a sample if the predicted probability of the sample under each category label is less than the probability threshold value corresponding to the category label.

[0199] In some embodiments, the confidence learning module 1204 is further used to determine, for each sample, if the predicted label of the sample is inconsistent with the current labeled label, that the sample is a target sample in the current round of sample data sets that requires label correction.

[0200] In some embodiments, the confidence learning module 1204 is also used to divide samples whose current annotation labels and predicted labels are both the same category label into target sample subsets corresponding to the category label; if the number of samples in the target sample subset is greater than or equal to a preset retention amount, then the samples whose current annotation labels are the category labels corresponding to the target sample subset and whose predicted labels are not the category labels corresponding to the target sample subset are determined as target samples that require label correction; the preset retention amount is the minimum number of samples that do not require label correction under a preset single category label; if the number of samples in the target sample subset is less than the preset retention amount, then from each sample whose current annotation label is the category label corresponding to the target sample subset, samples that meet the corresponding target correction amount are selected in ascending order of the predicted probability of the samples under the category label as target samples that require label correction; the target correction amount is determined based on the difference between the number of samples whose current annotation label is the category label corresponding to the target sample subset and the preset retention amount.

[0201] In some embodiments, the confidence learning module 1204 is also used to divide all samples with the same current annotation label and the same predicted label into the same sample subset; the sample subset into which the samples with the current annotation label and the predicted label being the same category label are divided is the target sample subset; a confidence joint matrix is ​​generated based on the number of samples in each sample subset; each element in the confidence joint matrix represents the number of samples in each sample subset; and the number of samples in each target sample subset is determined based on the confidence joint matrix.

[0202] In some embodiments, the confidence learning module 1204 is further used to determine, for each sample, the maximum predicted probability of the sample under each category label, and determine the category label corresponding to the maximum predicted probability as the estimated true label of the sample.

[0203] In some embodiments, satisfying the iteration stop condition includes satisfying any one of a first iteration stop condition and a second iteration stop condition; the first iteration stop condition is that the current number of iterations is greater than or equal to a first preset number of iterations; the second iteration stop condition is that the optimization degree of the performance indicator of the target model in the current round is less than or equal to a preset indicator threshold value; the optimization degree of the performance indicator of the target model in the current round is used to characterize the optimization degree of the performance indicator of the target model completed in the current round of training compared with the target model completed in the previous round of training.

[0204] In some embodiments, the sample data set of the current round includes the training set of the current round and the validation set of the current round; the sample data set of the current round is used to perform model training on the basic classification model; and the validation set of the current round is used to determine the performance indicators of the target model completed by the previous round of training.

[0205] In some embodiments, the label correction module 1206 is also used to record the number of times each sample is corrected by the label in each round of iteration; for each sample, when the proportion of the number of corrections of the sample is greater than or equal to a preset proportion threshold, the sample is determined to be an unstable sample; the proportion of corrections is the ratio between the number of corrections of the sample and the number of iterations; and relevant information of the unstable sample is output.

[0206] The above-mentioned label correction device first uses the initial sample data set to be corrected as the sample data set of the current round, and then in each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model trained in the current round, and the prediction probability of each sample in the sample data set of the current round under each category label is predicted through the target model trained in the current round. According to the prediction probability of each sample under each category label and the current labeled label of each sample, the target sample that needs to be label corrected in the sample data set of the current round and the estimated true label of each target sample are determined, and the current labeled label of the target sample is corrected to the estimated The real label is obtained, and the sample data set after the label correction of the current round is obtained. As the updated sample data set of the current round, it enters the next round of iteration until the iteration stop condition is met. The label correction is adaptively performed in a multi-round iterative manner, so that the universality and generalization are better. There is no need to formulate specific rules for specific types of sample data. It can make it possible to obtain a more accurate target sample data set through iterative label correction for any type of sample data, avoiding the problem that the label of the sample data set after a single label correction is inaccurate, unstable and easy to introduce new noise, so that the target sample data set obtained after the label correction is more accurate. Moreover, the label correction is fully automatic, and there is no need to manually process the data in the intermediate process, which improves the efficiency of label correction. In addition, as an independent framework, this method does not need to be deeply embedded in the classification module of other downstream tasks, does not rely on the data feature form of the sample data set and the network structure of the target classification model, and can be deployed as a cloud service to provide support for label correction of various types of sample data sets for classification, such as text, image, audio and video.

[0207] Each module in the above-mentioned label correction device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module above.

[0208] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.14As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample data sets. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a label correction method is implemented.

[0209] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.15 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a label correction method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0210] Those skilled in the art will understand that Fig.14 or Fig.15The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0211] The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the label correction method.

[0212] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the label correction method.

[0213] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0214] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0215] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0216] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A label correction method, characterized in that: include: Using the initial sample data set to be corrected as the sample data set of the current round; In each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training; Predicting the prediction probability of each sample in the sample data set of the current round under each category label by using the target model trained in the current round; Determine, according to the predicted probabilities of each of the samples under each of the category labels and the current annotated labels of each of the samples, the target samples that need to be label corrected in the sample data set of the current round, and the estimated true labels of each of the target samples; The current annotation label is the category label actually annotated by the sample at present; Correct the current annotated label of the target sample to the estimated true label, obtain the sample data set with the current round of label correction, and enter the next round of iteration as the updated sample data set of the current round until the iteration stop condition is met to obtain the target sample data set with the label correction.

2. The method according to claim 1, characterized in that In each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training, including: In the first round of iteration, the hyperparameter search is iteratively performed based on the sample data set and basic classification model of the current round to obtain the target hyperparameters; In each round of iteration, based on the target hyperparameters, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training.

3. The method according to claim 1, characterized in that In each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training, including: In each round of iteration, hyperparameter search is performed iteratively based on the sample data set and basic classification model of the current round to obtain the target hyperparameters of the current round; Based on the target hyperparameters of the current round, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training.

4. The method according to claim 1, characterized in that: Before correcting the current annotated label of the target sample to the estimated true label to obtain a sample data set in which the current round of label correction is completed, the method further includes: Determine the label quality score of the current annotated label of each target sample according to the predicted probability of each target sample under each category label; Determining the actual number of corrections for the current round according to a preset correction ratio and the number of target samples; From the target samples, target samples that meet the actual correction quantity of the current round are selected in order from low to high according to the label quality scores as samples to be corrected; Correcting the current annotated label of the target sample to the estimated true label to obtain a sample data set after the current round of label correction, includes: Correct the current annotated label of the sample to be corrected to the estimated true label, and obtain a sample data set with completed label correction in the current round.

5. The method according to claim 4, characterized in that Determining the label quality score of the current annotated label of each sample according to the predicted probability of each sample under each category label includes: For each category label, according to the average of the predicted probabilities of each sample whose current label is the category label under the category label, the probability threshold value corresponding to the category label is obtained; For each of the samples, a label quality score of the current label of the sample is determined according to a difference between a predicted probability of the sample under the current label and a probability threshold value corresponding to the current label.

6. The method according to claim 5, characterized in that The step of determining, for each of the samples respectively, a label quality score of the current label of the sample according to a difference between a predicted probability of the sample under the current label and a probability threshold value corresponding to the current label, comprises: For each of the samples, determine the difference between the predicted probability of the sample under each of the category labels and the probability threshold value corresponding to the category label; According to the differences corresponding to each of the category labels, the difference between the predicted probability of the sample under the current labeling label and the probability threshold value corresponding to the current labeling label is normalized to obtain a label quality score of the current labeling label of the sample.

7. The method according to claim 1, characterized in that The step of determining the target samples that need to be label corrected in the current round of sample data set and the estimated true labels of the target samples according to the predicted probabilities of the samples under the category labels and the current labeled labels of the samples, comprises: Determine the target samples that need label correction in the current round of sample data sets according to the predicted probabilities of the samples under the category labels and the current labeled labels of the samples; The estimated true label of the target sample is determined according to the predicted probability of the target sample under each of the category labels.

8. The method according to claim 7, characterized in that The step of determining the target samples that need to be label corrected in the current round of sample data sets according to the predicted probabilities of the samples under the respective category labels and the current label labels of the samples comprises: Determining a predicted label for each of the samples according to the predicted probability of each of the samples under each of the category labels; According to the predicted label of each sample and the current annotated label, the target sample that needs label correction in the sample data set of the current round is determined.

9. The method according to claim 8, characterized in that Determining the predicted label of each sample according to the predicted probability of each sample under each category label includes: For each sample, determine a target prediction probability from the prediction probabilities of the sample under each of the category labels; the target prediction probability is a prediction probability that is greater than or equal to a probability threshold value corresponding to the corresponding category label; The category label corresponding to the maximum predicted probability among the target predicted probabilities is determined as the predicted label of the sample.

10. The method according to claim 9, characterized in that The method further comprises: If the predicted probability of the sample under each of the category labels is less than the probability threshold corresponding to the category label, no processing is performed.

11. The method according to claim 8, characterized in that The step of determining target samples in the current round of sample data sets that require label correction according to the predicted labels of the samples and the current labeled labels includes: For each sample, if the predicted label of the sample is inconsistent with the current labeled label, the sample is determined to be a target sample in the current round of sample data set that needs label correction.

12. The method according to claim 8, characterized in that The step of determining target samples in the current round of sample data sets that require label correction according to the predicted labels of the samples and the current labeled labels includes: Divide the samples whose current annotation labels and predicted labels are the same category label into the target sample subset corresponding to the category label; If the number of samples in the target sample subset is greater than or equal to the preset retention amount, the samples whose current annotated labels are the category labels corresponding to the target sample subset and whose predicted labels are not the category labels corresponding to the target sample subset are determined as target samples that need to be label corrected; the preset retention amount is the minimum number of samples that do not need to be label corrected under a preset single category label; If the number of samples in the target sample subset is less than the preset retention amount, then from the samples whose current labeled labels are the category labels corresponding to the target sample subset, samples that meet the corresponding target correction amount are selected in ascending order of the predicted probability of the samples under the category labels as target samples that need to undergo label correction; the target correction amount is determined based on the difference between the number of samples whose current labeled labels are the category labels corresponding to the target sample subset and the preset retention amount.

13. The method according to claim 12, characterized in that The method further comprises: The samples with the same current annotation label and the same predicted label are divided into the same sample subset; the sample subset to which the samples with the current annotation label and the predicted label of the same category label are divided is the target sample subset; Generate a confidence joint matrix according to the number of samples in each of the sample subsets; each element in the confidence joint matrix represents the number of samples in each of the sample subsets; The number of samples in each of the target sample subsets is determined according to the confidence joint matrix.

14. The method according to claim 7, characterized in that Determining the estimated true label of the target sample according to the predicted probability of the target sample under each of the category labels includes: For each sample, the maximum predicted probability of the sample under each of the category labels is determined, and the category label corresponding to the maximum predicted probability is determined as the estimated true label of the sample.

15. The method according to any one of claims 1 to 14, characterized in that The satisfying of the iteration stop condition includes satisfying any one of a first iteration stop condition and a second iteration stop condition; the first iteration stop condition is that the current number of iterations is greater than or equal to a first preset number of iterations; The second iteration stopping condition is that the optimization degree of the performance indicator of the target model in the current round is less than or equal to the preset indicator threshold value; the optimization degree of the performance indicator of the target model in the current round is used to characterize the optimization degree of the performance indicator of the target model completed in the current round of training compared with the target model completed in the previous round of training.

16. The method according to claim 15, characterized in that The sample data set of the current round includes a training set of the current round and a validation set of the current round; the sample data set of the current round is used to perform model training on the basic classification model; the validation set of the current round is used to determine the performance indicators of the target model completed by the previous round of training.

17. The method according to any one of claims 1 to 14, characterized in that The method further comprises: Recording the number of times each sample is corrected by label correction in each round of iteration; For each sample, when the correction times ratio of the sample is greater than or equal to a preset ratio threshold, the sample is determined as an unstable sample; the correction times ratio is the ratio between the correction times of the sample and the iteration times; Output relevant information of the unstable sample.

18. A label correction device, characterized in that: include: A model training module, used to use the initial sample data set to be corrected as the sample data set of the current round; In each round of iteration, the basic classification model is trained according to the sample data set of the current round to obtain the target model completed by the current round of training; The confidence learning module is used to predict the prediction probability of each sample in the sample data set of the current round under each category label through the target model trained in the current round; determine the target samples in the sample data set of the current round that need to be label corrected, and the estimated true label of each target sample, according to the prediction probability of each sample under each category label and the current label label of each sample; The current annotation label is the category label actually annotated by the sample at present; The label correction module is used to correct the current annotated label of the target sample to the estimated true label, obtain the sample data set after the current round of label correction, and use it as the updated sample data set of the current round to enter the next round of iteration until the iteration stop condition is met, and obtain the target sample data set after the label correction.

19. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the label correction method according to any one of claims 1 to 17.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images