A label anti-noise text classification method based on semi-supervised learning

Abstract

The application discloses a label anti-noise text classification method based on semi-supervised learning. The method first predicts the probability of each sample in each category through task adaptive pre-training, and then uses a confidence learning method to evaluate the label quality of each sample, divides all training data into noise samples and clean samples; the noise samples are processed, the pseudo-label of the noise samples is predicted by means of data enhancement, integrated learning and other strategies, the MixMatch method is used to integrate consistency regularization, entropy minimization and traditional regularization method, the mainstream paradigm idea and components of semi-supervised learning are combined, and the noise samples are relabeled by combining the original label and the pseudo-label; finally, the noise samples and the clean samples are combined, the MixUp method is used to reconstruct new training samples and labels in a linear interpolation manner, and the final text classification model is retrained. The method greatly utilizes the noise label data, maximizes the utilization rate of the noise samples, and improves the robustness of the neural network.

Description

TECHNICAL FIELD

[0001] The application belongs to the cross field of natural language processing and noisy label learning, and particularly relates to a label anti-noise text classification method based on semi-supervised learning. BACKGROUND

[0002] In recent years, with the deepening of deep learning research, relevant research finds that the achievements of deep neural networks in many scenarios cannot be achieved without a large amount of real label information, and high-quality data is the basis for training of a classification model. However, in real complex life scenarios, noisy labels are a normal phenomenon, and learning from labels with noise is also a great challenge. There are many reasons for the generation of noisy labels. For example, there are many factors that cause errors in labeling data sets, such as poor data quality, low recognition, different labelers marking labels from different angles according to their subjective ideas, and the like; or errors in data encoding or communication process, resulting in a large number of unavoidable errors in sample data. Generally, data sets with noisy labels are ubiquitous in real life. In the field of text classification in natural language processing, there is also the problem of interference of noisy labels, which seriously affects the generalization performance of deep learning models and leads to inaccurate text classification prediction. For example, in the classification task of telecom customer complaint work orders, after users submit fault, complaint and business handling tasks, employees need to process complaint work orders according to the submitted information and classify the complaint work orders, and the accuracy of classification is very critical to business processing efficiency. In the field of natural language processing, there are relatively few methods for processing noisy labels. At present, most of the methods for processing noisy labels are mainly applied in the field of vision and cannot be directly applied in the field of natural language processing. Therefore, it is of great research and application value to explore an efficient and feasible method for processing noisy labels in text classification.

[0003] Therefore, a better technology should be proposed to deal with the problem of a large number of noisy labels in a data set. SUMMARY

[0004] The application aims to solve the problem that a large number of noisy labels in a data set in the real world lead to poor training effect of a deep learning model, and proposes a label anti-noise text classification method based on semi-supervised learning.

[0005] The application aims to solve the problem that a large number of noisy labels in a data set in the real world lead to poor training effect of a deep learning model, and proposes a label anti-noise text classification method based on semi-supervised learning.

[0006] S1: Pre-training: Use the training data of the text classification task itself as unlabeled data for pre-training and predict the probability of each sample in each category The training sample, i.e. each text sample to be classified in the text classification task;

[0007] S2: Noise data division: Use the probability predicted in step S1 and the original noisy label as the input of confidence learning, divide the noise data U and clean data C through the confidence learning method; the original noisy label is the label artificially marked by the original training sample;

[0008] S3: Data augmentation: Perform various data augmentation processing on the noise data U obtained in step S2 to obtain expanded text;

[0009] S4: Ensemble learning: Expand the text obtained in step S3 and predict the probability of each sample belonging to each category by using multiple models respectively;

[0010] S5: Noise sample relabeling: Calculate the probability mean obtained in step S4 using the hybrid matching method, and guess its pseudo label, i.e. assign a new predicted label to each sample, and perform entropy minimization processing using the sharpening method;

[0011] S6: Re-training and obtaining the final model: Mix the noise data U and clean data C divided in S2 using a hybrid method, construct new training samples and labels, and retrain the model. When the loss function is minimized, the training is completed, and the final text classification model is obtained.

[0012] Further, in step S1, the pre-training uses a task adaptive pre-training method TAPT;

[0013] (1) Use BERT as a pre-training model;

[0014] (2) The task adaptive pre-training method TAPT is to use the training data of the task itself as unlabeled data for pre-training in the second stage, and obtain a task adaptive pre-training language model.

[0015] Further, in step S2, the counting and filtering steps of the confidence learning method are used for division. The confidence learning does not require any hyperparameters, only two inputs, i.e. the predicted probability and the noisy label The following steps are mainly used to achieve:

[0016] (1) Counting: Estimate the joint distribution of noisy labels and true labels; the calculation formula is as follows

[0017]

[0018]

[0019] wherein X represents samples; the true label y * is an estimated value, the probability of the i-th sample under the j-th class is a confidence threshold t j represents the average probability of each artificial labeling class j, the count matrix that is, the number of samples with artificial label k but actual label j, the sample set with artificial label k is screened out, and the number of samples with actual label j in the sample set is calculated, that is, the number of samples satisfying is determined.

[0020] (2) filtering Clean: finding and filtering out error samples, after obtaining the count matrix, the samples entering the non-diagonal unit are selected as samples containing noise labels, that is,

[0021] Further, in the step S3, three data enhancement methods are used to perform data enhancement on the noise samples; the three data enhancement methods are respectively using back-translation method, conditional bidirectional transformer encoder (method and using random inactivation method.

[0022] Further, in the step S4, three pre-training models are integrated for probability prediction; the three pre-training models are respectively: bidirectional transformer encoder pre-training model BERT, large-scale bidirectional transformer encoder pre-training model RoBERT and generalized autoregressive pre-training model for language understanding XLNet.

[0023] Further, in the step S5, the noise samples are re-labeled using a hybrid matching method combined with the ensemble learning method of step S4, mainly including the following steps:

[0024] (1) label guessing: for a single unlabeled sample, the probability mean after data enhancement and ensemble learning model prediction is calculated, and the "pseudo" label is guessed, which is added to the unsupervised loss term , and the mathematical expression is:

[0025]

[0026] wherein K represents the number of data enhancement methods; M represents the number of prediction models; m represents; p model represents the probability of model prediction y; y represents the label result of model prediction; represents the k-th data enhancement method b-th unlabeled sample;

[0027] (2) sharpening Sharpen: according to the label confidence wi The noise label with the label predicted by the model is linearly combined to obtain a new noise label is the average prediction of the M models, and the sharpening method is used to perform entropy minimization processing:

[0028]

[0029]

[0030] wherein is the average prediction of the M models, T is a temperature parameter that can adjust the classification entropy, and adjusting T tends to 0, then tends to a one-hot distribution, that is, the output probability of a certain class is 1, and the output probability of all other classes is 0, at this time the classification entropy is the lowest, and L is the class of classification.

[0031] Specifically, in step 6, a mixed method is used to construct new training samples and labels in a linear interpolation manner. For samples (x1, y1) and samples (x2, y2), the mixed samples (x', y') are obtained by the following way:

[0032] B(alpha, alpha) = ∫0 1 x α-1 (1-x) α-1 dx

[0033] lambda ~ Beta(alpha, alpha)

[0034] lambda' = max(lambda, 1-lambda)

[0035] x' = lambda' * x1 + (1-lambda') * x2

[0036] y' = lambda' * y1 + (1-lambda') * y2

[0037] Wherein B(alpha, alpha) represents the Beta distribution; lambda' represents a hyperparameter; x' represents the sample after linear interpolation processing; alpha is a hyperparameter, and lambda is a random variable and is subject to Beta distribution;

[0038] For the data C = ((c i , p i ); i ∈ (1,..., N)) containing clean labels and the noise data Concatenate and shuffle to form the data source W required by MixUp:

[0039]

[0040] For the i-th sample, calculate MixUp(C i ,W i ) is added to the set C', and is added to the set U', so that C is changed to C' and U is changed to U'.

[0041] Specifically, L2 loss is used on U', and cross-entropy loss is used on C', and the mathematical expressions of the two are as follows:

[0042]

[0043]

[0044] Where |C'| is equal to the batch size batchsize, |U'| is equal to k times the batch size batchsize, p model (y|c;θ) represents; p model (y|u;θ) represents; L is the number of classification categories;

[0045] Then the final loss function is obtained as:

[0046] L=L c +λ U L U

[0047] Where λ U is the unsupervised loss weight.

[0048] The beneficial effects of the present application are: based on the semi-supervised learning idea, the present application comprehensively uses a series of methods such as data enhancement, ensemble learning, BERT pre-training and confidence learning, so that it has good performance in different data sets, and can effectively reduce the interference of noisy label data. Compared with other noise label processing methods, the noise data is discarded or a relatively small weight is given to the noise data, which leads to sample waste and reduces the robustness of the neural network. The method proposed in the present application greatly utilizes the noise data, improves the tolerance to noise labels, can perform fault-tolerant learning, and to some extent solves the problem of the influence of the existence of noise labels on the accuracy of the final text classification result. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is the flowchart designed by the present application;

[0050] Figure 2 is the flowchart of the confidence learning method used by the present application;

[0051] Figure 3 is the experimental result graph of the present application. DETAILED DESCRIPTION

[0052] The method proposed by the present application will be further described in detail below in combination with the drawings and specific examples.

[0053] The present application proposes a label noise-resistant text classification method based on semi-supervised learning. Based on the BERT pre-training language model, task-adaptive pre-training and confidence learning methods are used to divide noise data and clean data. Then a series of processing is performed on the noise data to enhance the diversity of the text and improve the robustness of the neural network. Three data enhancement methods are mainly used, and three methods are integrated for class probability prediction. Finally, based on the semi-supervised idea, the noise label is re-labeled using the MixMatch method, and the training sample is re-constructed and re-trained using the MixUp method. There are four modules in total, and each module is organically combined. The specific flow chart is shown in Figure 1 , which includes the following steps:

[0054] S1: Pre-training: The training data of the text classification task itself is pre-trained as unlabeled data and the probability of each sample in each class is predicted The training sample, i.e. each text sample that needs to be classified in the text classification task;

[0055] S2: Noise data division: the probability and the original noisy label are used as the input of confidence learning, and noise data U and clean data C are divided by confidence learning method; the original noisy label is the label artificially marked by the original training sample;

[0056] S3: Data augmentation: the noise data U obtained in step S2 is subjected to various data augmentation processing to obtain expanded text;

[0057] S4: Ensemble learning: the expanded text obtained in step S3 is subjected to various model prediction respectively to predict the probability of each sample belonging to each class;

[0058] S5: Noise sample re-labeling: using the mixed matching method, the probability mean obtained in step S4 is calculated, and a new predicted label is guessed for each sample, i.e. a new predicted label is assigned to each sample, and the sharpening method is used for entropy minimization processing.

[0059] S6: Re-training and obtaining the final model: using the mixed method, the noise data U and clean data C divided in S2 are mixed to construct new training samples and labels and perform model re-training. When the loss function is minimized, the training is ended, and the final text classification model is obtained.

[0060] 1. Noise data division

[0061] For raw datasets with noisy labels in the real world, these are treated as unlabeled corpora and used as training data for the pre-trained model. For each text, BERT encoding is first used to convert each character into a vector representation (word embedding). Then, sentence embedding and positional embedding are used to map these three embeddings to a high-dimensional space of the same dimension. Finally, they are summed and used as input for BERT pre-training. Two unsupervised tasks are used for pre-training: Masked Language Modeling (Masked LM) and Next Sentence Prediction. After pre-training, the pre-trained model is obtained. Each sample data is used as input to predict its probability in each category, which serves as one of the inputs to the confidence learning method.

[0062] After the pre-trained model makes predictions, a confidence learning method is needed to separate noisy and clean data. Confidence learning is based on the assumption of a noisy classification process, assuming that noisy labels are class-conditional, depending only on the potential correct class and not on the data itself. It classifies and identifies incorrect labels by estimating the conditional probabilities between a given noisy label and a potential correct label. For confidence learning, its learning framework requires only two inputs: one is the probability predicted by the pre-trained model. A 3D matrix, where m represents the number of labels and n represents the number of training samples. (This represents the predicted probability that the i-th training sample belongs to the k-th label); the second is noisy labels. (Perform one-hot encoding on m original tags, so that...) (It is an m-dimensional vector). For example... Figure 2 As shown, a total of two steps are required.

[0063] (1) Count: Estimates the joint distribution of noise labels and true labels. The calculation formula is as follows:

[0064]

[0065]

[0066] Where X represents the sample; y represents the true label. * (But in reality, the true value is unknown and needs to be estimated), the probability of the i-th sample in the j-th category is: Confidence threshold t j The counting matrix represents the average probability of each manually labeled category j. That is, the number of samples manually labeled k but actually labeled j; filtering out the sample set manually labeled k; and calculating the number of samples in the sample set whose actual label is j, i.e., determining... The number of samples.

[0067] (2) Find and filter out error samples. After obtaining the count matrix, select the samples entering the non-diagonal unit as the samples containing noise labels, that is

[0068] Through the above two steps, all training data X can be divided into data C containing noise labels and clean labels U, for each sample x i The label confidence refers to the probability of belonging to its original label

[0069] 2、Data augmentation module

[0070] In deep learning, it is generally required that the number of samples is sufficient and the quality is high, and the trained model will have better effect and stronger generalization ability. However, in fact, such a data set is very few, so it is necessary to do data augmentation on the samples to improve the quality of the samples. According to the consistency regularization of semi-supervised learning, the label of the data augmented sample is the same as the original label, so the existing separated noise samples can be data augmented.

[0071] In order to improve the robustness of the final training model, the present application does not use only one data augmentation method, but selects relatively representative text data methods from different angles, a total of three data augmentation methods, which are back translation method, conditional bidirectional transformer encoder CBert method and use random inactivation (Dropout) method. Among them, the back translation method is a good enhancement method in the aspect of text data enhancement and the operation is simple; the CBert method is a context enhancement method based on language model and the method is obviously better than the existing text data enhancement method, which is one of the representatives of context data enhancement. Since the present application is based on the language model BERT, that is, the bidirectional transformer encoder pre-training model BERT, using this method can not only obtain good effect but also reduce code amount and framework; the main purpose of using the random inactivation (Dropout) method is to regularize the neural network, which is a data enhancement technique realized by modifying the neural network itself, and is one of the representatives of the noise method and the operation method is simple. The three methods respectively perform data augmentation from the aspects of semantics, context and neural network itself, and have a certain representativeness. After the three data augmentation methods, the original sample can be expanded into three different corpus samples, which is beneficial to improve the robustness of the training corpus.

[0072] 3、Data re-labeling

[0073] After obtaining the noise-containing labeled data U and its augmented data After that, the noise data needs to be relabeled so that it can be divided into relatively correct labels. The present application uses a similar idea to the MixMatch method to relabel the noise data and makes improvements. Using the MixMatch method mainly has two steps, which are label guessing and sharpening.

[0074] Before using Labeling Guessing, in order to reduce the prediction bias of a single model, the present application uses ensemble learning to average the prediction results of the model, and a total of three models are used, which are bidirectional transformer encoder pre-training model BERT, large-scale bidirectional transformer encoder pre-training model RoBERT, and generalized autoregressive pre-training model for language understanding XLNet. Bidirectional transformer encoder pre-training model BERT is a large-scale language model; RoBERTa is an improved version of BERT, which uses more data, longer training sequences, and other methods; XLNet is a general autoregressive pre-training method that solves the problem that BERT cannot be used for generation and the inconsistency between training data and test data. By using three models to average the prediction results, the bias can be further reduced, and then the MixMatch method is operated.

[0075] (1) Label Guessing: For a single unlabeled example, calculate the probability mean after data augmentation and ensemble learning model prediction, guess its "pseudo" label, and this obtained label will be added to the unsupervised loss term.

[0076]

[0077] (2) Sharpening: According to the label confidence w i , the noise label is linearly combined with the label predicted by the model to obtain a new noise label is the average prediction of M models, and the sharpening method is used for entropy minimization processing:

[0078]

[0079]

[0080] wherein, is the average prediction of M models, T is the temperature parameter, which can adjust the classification entropy, and when T tends to 0, tends to one-hot distribution, that is, the output probability of a certain class is 1, and the output probability of all other classes is 0, at this time the classification entropy is the lowest, and L is the classification category.

[0081] 4. retrain

[0082] After re-labeling the noisy samples, a new training sample and label are constructed in a linear interpolation manner using the MixUp method. For a sample (x1, y1) and a sample (x2, y2), a mixed sample (x', y') is obtained by:

[0083] B(a, a) = ∫0 1 x α-1 (1-x) α-1 dx

[0084] λ ~ Beta(a, a)

[0085] λ' = max(λ, 1-λ)

[0086] x' = λ'x1 + (1-λ')x2

[0087] y' = λ'y1 + (1-λ')y2

[0088] where a is a hyperparameter.

[0089] For data C = ((c i , p i ); i ∈ (1, …, N)) containing clean labels and noisy data U = ((u i , p i ); i ∈ (1, …, N)) with re-predicted labels, concatenate them and shuffle to form the data source W required by MixUp:

[0090]

[0091] For the i-th example, calculate MixUp(C i , W i ) and add it to the set C', calculate and add it to the set U', so that C is transformed into C' and U is transformed into U'.

[0092] During model training, L2 loss is used on U' and cross-entropy loss is used on C'.

[0093]

[0094]

[0095] where |C'| is equal to batchsize, |U'| is equal to k times batchsize, and L is the number of classification categories.

[0096] The final loss function is obtained as:

[0097] L = L c + λU L U

[0098] wherein, λ U is an unsupervised loss weight.

[0099] After the final training, a final trained model can be obtained. At this time, for the input classification sample, a prediction label can be obtained by predicting through the trained model.

[0100] In order to verify the effectiveness of the application, the method is applied to a real data set, taking the telecom complaint work order data as an example, 100,000 data are selected as original data, the baseline is selected as the traditional BERT model, the confidence learning and the divide mixed (DivideMix) method, the accuracy and the F1 value are used as evaluation indexes, and the final experimental results are as shown in Figure 3 It can be seen that the application has the best effect.

Claims

1. A label anti-noise text classification method based on semi-supervised learning, characterized in that, The method comprises the following steps: S1: pre-training: pre-training the text classification task itself with the training data as unlabeled data and predicting the probability of each sample in each category The training sample, that is, each text sample to be classified in the text classification task; S2: noise data division: divide the probability predicted by step S1 into noise data U and clean data C by a confidence learning method and the original noisy label as the input of confidence learning, divide the noise data U and the clean data C by the confidence learning method; the original noisy label is the label artificially marked by the original training sample; S3: data enhancement: performing multiple data enhancement processing on the noise data U obtained in step S2 to obtain expanded text; S4: ensemble learning: performing multiple model prediction on the expanded text obtained in step S3 to predict the probability of each sample belonging to each category; S5: noise sample relabeling: using a hybrid matching method, calculating the mean of the probabilities obtained in step S4, and guessing the pseudo-label, that is, assigning a new predicted label to each sample, and using a sharpening method to perform entropy minimization processing; S6: retraining and obtaining a final model: using a hybrid method to mix the noise data U and clean data C relabeled in S5 to construct new training samples and labels and retrain the model, and the training ends when the loss function is minimized, obtaining a final text classification model; the hybrid method is to construct new training samples and labels in a linear interpolation manner, and for sample (x1, y1) and sample (x2, y2), the mixed sample (x', y') is obtained by the following method: B(a,a) = ∫0 1 x α-1 (1-x) α-1 dx λ ~ Beta (α, α) λ' = max (λ, 1-λ) x' = λ'x1 + (1-λ')x2 y' = λ'y1 + (1-λ')y2 Where B (α, α) represents the Beta distribution; λ' represents the hyperparameter; x' represents the sample after linear interpolation processing; α is a hyperparameter, and λ is a random variable and follows the Beta distribution; For clean-labeled data C = ((c i ,p i ) ; i ∈ (1,..., N)) and noisy data with re-predicted labels Concatenate and shuffle them to form the data source W needed for MixUp: For the i-th example, compute MixUp(C i ,W i ) is added to the set C′, compute is added to the set U′, so that C is transformed into C′, is transformed into U′. 2.The label anti-noise text classification method based on semi-supervised learning according to claim 1, wherein, In step S1, the pre-training uses a task adaptive pre-training method TAPT; (1) using BERT as a pre-training model; (2) the task adaptive pre-training method TAPT uses the task itself training data as unlabeled data for the second stage of pre-training to obtain a task adaptive pre-training language model. 3.The label anti-noise text classification method based on semi-supervised learning according to claim 1, wherein, In the step S2, the two steps of counting and filtering using a confident learning method are used, the confident learning does not need any hyperparameters, only two inputs, i.e. the prediction probability and noisy labels This is mainly achieved using the following steps: (1) Count: estimate the joint distribution of noise labels and true labels; the calculation formula is as follows where X represents the sample; y * is the estimated value of the true label; the confidence threshold t j represents the average probability of each artificial labeling category j, the count matrix that is, the number of samples with artificial label i but actual label j, the sample set with artificial label i is screened out, and the number of samples with actual label j in the sample set is calculated, that is, it is determined that the number of samples is less than the confidence threshold t (2) Filter Clean: find and filter out the error samples. After obtaining the count matrix, select the samples entering the off-diagonal cells as samples containing noise labels, that is 4.The label anti-noise text classification method based on semi-supervised learning according to claim 1, wherein, In step S3, three data enhancement methods are used to perform data enhancement on the noise samples; the three data enhancement methods are using back-translation, conditional bidirectional transformer encoder (method, and using random inactivation. 5.The label-robust text classification method based on semi-supervised learning according to claim 1, wherein, In step S4, three pre-training models are integrated to predict the probability; the three pre-training models are: bidirectional transformer encoder pre-training model BERT, large-scale bidirectional transformer encoder pre-training model RoBERT, and language understanding generalized autoregressive pre-training model XLNet. 6.The label-robust text classification method based on semi-supervised learning according to claim 1, wherein, In step S5, the noise samples are relabeled using a hybrid matching method combined with the ensemble learning method of step S4, and the following steps mainly exist: (1) Label Guessing: For single unlabeled instance, compute the mean of the probabilities predicted by data augmentation and ensemble learning model, guess its "pseudo" label, this obtained label will be added to the unsupervised loss term The mathematical expression thereof is: wherein K represents the number of data augmentation methods; M represents the number of prediction models; p model represents the probability of the model predicting y; y represents the label result predicted by the model; represents the bth unlabeled sample of the kth data augmentation method; (2) Sharpening: according to label confidence w i The noise label is linearly combined with the label predicted by the model to obtain a new noise label is the average prediction of M models, and the entropy minimization process is performed using the sharpening method: wherein is the average prediction over M models, T is a temperature parameter that can adjust the classification entropy, adjusting T towards 0, then tends to a one-hot distribution, i.e. outputting probability 1 for one class and 0 for all other classes, in which case the classification entropy is minimal, L is the class of the classification.

7. The label anti-noise text classification method based on semi-supervised learning according to claim 1, characterized in that, L2 loss is used on U', and cross-entropy loss is used on C', and the mathematical expressions of the two are as follows: Where |C'| is equal to the batch size batchsize, |U'| is equal to k times the batch size batchsize, and L is the number of classification categories; Then the final loss function is obtained as: L = L c + λ U L U where λ U is an unsupervised loss weight.

Images