A noisy semi-supervised text classification method based on BERT-base network

Abstract

The present invention discloses a noisy semi-supervised text classification method based on a BERT-base network. The present invention comprises the following steps: S1, using a model to predict new samples generated after back-translation of unlabeled samples and constructing predicted labels; S2, after inputting labeled and unlabeled samples into the model again, performing perturbation and interpolation processing on a specific Transformer hidden layer in BERT, and finally obtaining an interpolation model output; S3, constructing a loss function loss=l s +l sce , the first item is the cross entropy for labeled samples, and the second item is the symmetric cross entropy between the interpolated label #imgabs0# and the interpolated model output #imgabs1#; S4, training and updating model parameters; S5, using the trained model to predict and classify new samples. The present invention combines the symmetric cross entropy and interpolation methods in the field of noisy learning, so that labeled and unlabeled samples can be trained collaboratively, while reducing the impact of model misjudgment in the label prediction process and reducing the overfitting phenomenon of labeled data in the training process.

Description

Technical Field

[0001] The present invention is applied to the field of natural language processing (NLP), and specifically relates to a noisy semi-supervised text classification method based on a BERT-base network. Background Art

[0002] With the development of communication and Internet technologies, data has become an industrial raw material. How to mine and utilize the information hidden in the data has formed multiple industries and created huge economic benefits. Text data is generated by direct human expression, so it has the lowest cost, the largest amount, and contains the richest information.

[0003] In many applications of text data, classification is a basic task. Traditional text classification is generally based on supervised learning with labeled data, but in actual specific industrial scenarios, most of the data is unlabeled raw data. Data labeling may require a lot of human resources, and the quality of manually labeled labels is also questionable. In actual situations, it may even happen that "the more training data, the worse the training effect." Therefore, semi-supervised learning methods that only use a small amount of labeled data and a large amount of unlabeled data have become a more practical and low-cost option.

[0004] Semi-supervised learning is currently mostly applied to image classification. In recent years, mainstream methods have used the ideas of consistency regularization and minimization of entropy. On the one hand, if noise is added to the data input, the output of a robust model should be similar; on the other hand, the entropy of the model on unlabeled data should be minimized as much as possible. The unlabeled data is input into the model after one or several enhancements, and the lower entropy output of the model is used as its predicted label. At the same time, the unlabeled data is perturbed in another way and then input into the model. At this time, the output of the model should be similar to the artificial label just obtained. The method of measuring "lower entropy" is generally to see whether the maximum probability in the probability distribution of the model output exceeds a certain threshold. How to measure "similarity" depends on the choice of loss function. The general choices are KL divergence, cross entropy, L2 regularization, etc. However, there are two problems in text semi-supervised classification. On the one hand, how to apply the ideas of consistency regularization and minimization of entropy in image semi-supervised classification to text processing. Simple operations such as rotating, mirroring, and converting grayscale of images can achieve data enhancement without changing the sample labels. However, in NLP, text data has certain discrete characteristics, and it is difficult to generate a large number of semantically unchanged extended samples through simple transformation. Especially in short texts, changes to some words are likely to directly change the overall semantics, so adding disturbances directly to word vectors is not the best choice. On the other hand, judging the confidence of the model's output by setting a probability threshold is relatively simple and direct, but the problem in actual operation is that due to the existence of the threshold, unlabeled data may not be added until the middle or even late stage of model training, and at this time, the small amount of labeled data that has already participated in the training may have been overfitted. Using an overfitted model to predict the label of the data is bound to contain misjudgments, and this error is not uniformly dispersed random noise, but systematic noise produced by the model after "reading too hard". Therefore, how to deal with the collaborative training of labeled and unlabeled data and how to deal with the noise in model prediction is also a problem. Summary of the invention

[0005] The present invention provides a noisy semi-supervised text classification method based on BERT-base (BERT is a bidirectional encoder based on Transformer architecture), which generates new samples for unlabeled samples through a back-translation method, and then uses the prediction result with higher confidence as the predicted label of the new sample through model prediction; during the training process, an interpolation method is used to train labeled samples and unlabeled samples collaboratively, and a gradient ascent method is used to add disturbances to the output vector of the third hidden layer of the unlabeled sample in the model, and finally an improved loss function is used to improve the robustness of the model to the noise in its predicted label.

[0006] To achieve the above purpose, the technical solution steps adopted by the present invention are as follows:

[0007] S1, data initialization stage. Back-translate the unlabeled samples to generate new samples, then input the new samples into the model, and finally convert the high-confidence output vectors into corresponding one-hot labels. The specific steps are as follows:

[0008] S11, unlabeled samples After being translated into other languages ​​through machine translation tools and then translated back to the original language, the sample Get the sample where j∈{1,2,…,N u},N u is the number of unlabeled samples in a batch of samples.

[0009] S12, the sample Input classification model to get output vector like That is, the classification model classifies the sample with a confidence level higher than the threshold τ, and assigns A one-hot prediction label Label At the same time, the corresponding samples and samples θ represents the model parameters.

[0010] S13, if the sample is assigned a one-hot label, then the unlabeled samples Input model to get output vector The labeled samples Input model to get output vector

[0011] S2, data enhancement stage. The FGM (Fast Gradient Method) method is used to add disturbance and perform interpolation. The specific steps are as follows:

[0012] S21, BERT-base network is mainly composed of 1 word vector encoding layer and 12 hidden layers of Transformer architecture. Here for unlabeled samples Add perturbation r to the output of the third hidden layer of BERT-base j =ε·g j / ||g j ||, where That is, vector The L2 regularization of the hidden layer output h is used to find the gradient, ε is a positive hyperparameter, and ||·|| means finding the vector norm. Assume that the unlabeled sample The output of the third hidden layer is The result after disturbance is

[0013] S22. Randomly select one of the 7th, 9th, and 12th hidden layers in the BERT-base network to perform interpolation operations on the hidden layer outputs of unlabeled samples and labeled samples. and They are samples with labels. and unlabeled samples The output of the kth layer in the hidden layer of the classification model (k∈{7,9,12}), the corresponding true label and predicted label are and The steps of the interpolation operation are as follows:

[0014] S221. Generate a random number λ between 0 and 1 using Beta distribution.

[0015] S222. Take λ1=max{λ,1-λ}, λ2=min{λ,1-λ}, that is, λ1>λ2 and λ1+λ2=1.

[0016] S223, Calculation

[0017] S224, will Input the k+1th hidden layer, and obtain the interpolation model output through calculations of the following hidden layers. The entire process of step S2 is recorded as f'(·), and the interpolation model output is recorded as The two-hot interpolation label is recorded as

[0018] S3, the construction phase of the loss function. Loss function loss = l s +l sce , the first item is the model output of the labeled sample With its real label Find the cross entropy, the second term is the interpolation label With interpolation model output The symmetric cross entropy is as follows:

[0019] S31, l sce =μ1l ce +μ2l rce The symmetric cross entropy consists of two parts: ce is the cross entropy, l rce Anti-cross entropy. μ1 and μ2 are the hyperparameter weights of the two. The specific formula is:

[0020] Cross Entropy:

[0021] Inverse Cross Entropy:

[0022] where i∈{1,2,…,Nl}, j∈{1,2,…,N u},because is a two-hot label vector. There must be 0 elements in the label vector, so log0 exists in the calculation of anti-cross entropy. The hyperparameter A=log0 is defined, A<0, and I{·} indicates that if the condition in the curly brackets is met, 1 is returned, otherwise 0 is returned.

[0023] S32, l s is the cross entropy for labeled samples.

[0024] S33, calculate the final loss function loss: loss = l s +l sce = l s +μ1l ce +μ2l rce .

[0025] S4: Model parameter updating stage.

[0026] S41 updates the model parameter θ, i.e. θ←θ-η▽ θ loss, η is the learning rate.

[0027] S42, repeating steps S1 to S4 in the next batch of samples, verifying the prediction accuracy of the model after each batch of samples is trained, and saving the model parameters if the current highest accuracy is updated. If the prediction accuracy of the model does not improve after 10 batches of training, the training process is terminated.

[0028] S5. Use the trained model for text classification applications.

[0029] Compared with the existing methods, the present invention has the following advantages:

[0030] The present invention implements a semi-supervised noisy learning method for text classification based on the popular BERT-base model. (1) Compared with the traditional semi-supervised learning method, the present invention takes into account the noise problem in the model prediction label more carefully, and introduces the symmetric cross entropy in the field of image noisy learning, which reduces the impact of model misjudgment. (2) It is then integrated with the interpolation method to form a new symmetric cross entropy under the two-hot mode, which improves the generalization performance of the model while training labeled and unlabeled samples together, minimizing the overfitting phenomenon of some data during the training process. (3) In the interpolation operation, compared with the rough method of randomly shuffling all samples and then interpolating them in the traditional approach, the present invention more carefully maintains the crossover between unlabeled samples and labeled samples, and tries to keep the bias towards the true label when the loss function gradient descends in the selection of weights, which further reduces the impact of model misjudgment. (4) Compared with the traditional approach of directly adding perturbations to word vectors, the present invention selects the BERT-base hidden layer output vector as the perturbation object, thereby minimizing the excessive modification of the sample semantics by the perturbation operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is the main structure diagram of the model;

[0032] Figure 2 This is a single-round training flow chart of the BERT-base-based semi-supervised noisy learning method in the present invention;

[0033] Figure 3 Schematic diagram of the perturbation and interpolation operations in the present invention; DETAILED DESCRIPTION

[0034] The specific implementation details of the present invention will be further described below in conjunction with the accompanying drawings.

[0035] A semi-supervised noisy text classification method based on BERT-base, the model structure is as follows Figure 1 As shown in the figure, it consists of a word vector encoder and a 12-layer Transformer architecture hidden layer. After multiple layers of operations, BERT-base takes the final hidden layer output vector of the sentence start mark (CLS) as the semantic expression vector of the whole sentence, and then uses a two-layer linear classifier to perform the final classification on the semantic vector.

[0036] This method is mainly divided into four stages: S1 data initialization stage, S2 data enhancement stage, S3 loss function construction stage, S4 model parameter update stage, S5 model application stage. Figure 2 , the steps of the present invention are as follows:

[0037] S1, data initialization stage. Back-translate the unlabeled samples and generate new samples. Input the new samples into the model and convert the high-confidence output vectors into corresponding one-hot labels. Specifically include:

[0038] S11, unlabeled samples Where N u is the number of unlabeled samples in a batch of samples) is translated into other languages ​​through machine translation tools and then translated back to the original language. Get the sample The two are expressed differently, but have the same semantics.

[0039] S12, the sample Input classification model to get output vector like That is, the classification model classifies the sample with a confidence level higher than the threshold τ, and assigns A one-hot prediction label Label At the same time, the corresponding samples and samples θ represents the model parameters.

[0040] S13, if the sample is assigned a one-hot label, then the unlabeled samples Input model to get output vector The labeled samples Input model to get output vector

[0041] S2, data enhancement stage. Combined Figure 3 As shown in the figure, the FGM (Fast Gradient Method) method is used to add disturbance and interpolation. Since the semantic expression in the text is not continuous, the change of some keywords may lead to excessive contrast of the overall semantics. Studies have shown that some specific hidden layer outputs of BERT are more effective in extracting syntactic semantics and other information. Therefore, in BERT, the specific Transformer hidden layer is perturbed and interpolated instead of the word vector encoding layer. Step S2 is specifically as follows:

[0042] S21, BERT-base network is mainly composed of 1 word vector encoding layer and 12 hidden layers of Transformer architecture. Here for unlabeled samples Add perturbation r to the output of the third hidden layer of BERT-base j =ε·g j / ||g j ||, where That is, vector The L2 regularization of the hidden layer output h is used to find the gradient, ε is a positive hyperparameter, and ||·|| means finding the vector norm. Assume that the unlabeled sample The output of the third hidden layer is The result after disturbance is

[0043] S22. Randomly select one of the 7th, 9th, and 12th hidden layers in the BERT-base network to perform interpolation operations on the hidden layer outputs of unlabeled samples and labeled samples. and They are samples with labels. and unlabeled samples The output of the kth layer in the hidden layer of the classification model (k∈{7,9,12}), the corresponding true label and predicted label are and The steps of the interpolation operation are as follows:

[0044] S221. Generate a random number λ between 0 and 1 using Beta distribution.

[0045] S222. Take λ1=max{λ,1-λ}, λ2=min{λ,1-λ}, that is, λ1>λ2 and λ1+λ2=1

[0046] S223, Calculation

[0047] S224, will Input the k+1th hidden layer, and obtain the interpolation model output through calculations of the following layers. The entire process of S2 is recorded as f'(·), then the interpolation model output is The two-hot interpolation label is recorded as

[0048] S3, the construction phase of the loss function. Loss function loss = l s +l sce , the first item is the model output for labeled samples and its real label Find the cross entropy, the second term is the interpolation label With interpolation model output During the training process, the symmetric cross entropy will make the probability of the true label position increase faster than the predicted label position, while the probability distribution of other wrong positions in the approximate two-hot label will decrease faster as the probability of the two correct positions increases, which is also in line with the original idea of ​​symmetric cross entropy. The specific process is as follows:

[0049] S31, l sce =μ1l ce +μ2lrce The symmetric cross entropy consists of two parts: ce is the traditional cross entropy, l rce Anti-cross entropy. μ1 and μ2 are the hyperparameter weights of the two. The specific formula is:

[0050] Cross Entropy:

[0051] Inverse Cross Entropy:

[0052] i∈{1,2,…,N l}, j∈{1,2,…,N u},because is a two-hot label vector. There must be 0 elements in the label vector, so log0 exists in the calculation of anti-cross entropy. The hyperparameter A=log0 is defined, A<0, and I{·} indicates that 1 is returned if the condition in the curly brackets is met, otherwise 0 is returned.

[0053] The following is an explanation of the effectiveness of the two-hot mode anti-cross entropy constructed in the present invention, taking a single labeled sample and a single unlabeled sample as examples:

[0054] l rce =-p i logλ1-p j logλ2-(1-p i -p j )A

[0055]

[0056]

[0057]

[0058] Among them, z i and z j Represents the output of the model before softmax at the category to which the correct label and the predicted label belong, respectively, c Represents the output of the model before softmax in the wrong category. p i and p j Represents the final output of the model at the category to which the correct label and the predicted label belong, respectively, c Represents the final output of the model at the error class.

[0059] Because 0<λ1,λ2<1,A<0, in two-hot mode, p i and p j Basically in the range of 0 to 0.5, so:

[0060]

[0061] That is l rce The decrease of will make the output z of the correct category i and z j Rising, output z at the wrong category c Decreases, which is in line with the basic purpose of the loss function, l rce It can accelerate the gradient descent of loss.

[0062] and The size of depends mainly on the first-order term, namely and p i and p j , and because 0<λ2<λ1<1, -(A-logλ1)>-(A-logλ2)>0, p i p j >0, Therefore, in most cases It holds true, that is, the anti-cross entropy is generally more biased towards the true label.

[0063] Studies have shown that the occasional misjudgment causes the model's confidence in predicting the wrong label to be low, so if p j The corresponding predicted label is a misjudgment, then the smaller first-order term can be It plays a limiting role. This is also in line with the original idea of ​​symmetric cross entropy.

[0064] S32, l s is the cross entropy for labeled samples.

[0065] S33. Calculate the final loss function: loss = l s +l sce = l s +μ1l ce +μ2l rce .

[0066] S4: Model parameter updating stage.

[0067] S41, update the model parameter θ, that is, θ←θ-η▽ θ loss, η is the learning rate.

[0068] S42, repeating steps S1 to S4 in the next batch of samples, verifying the prediction accuracy of the model after each batch of samples are trained, and saving the model parameters if the current highest accuracy is updated. If the prediction accuracy of the model does not improve after 10 batches of data training, the training process is terminated.

[0069] S5, model application stage. Use the model trained in step S42 to input the test text into the model. The category corresponding to the maximum probability in the model output vector is the category to which the prediction belongs.

[0070] In summary, the present invention introduces the idea of ​​semi-supervised image classification into the field of natural language processing, and proposes a semi-supervised noisy text classification method based on the BERT-base model.

[0071] By performing perturbations and interpolation processing in the BERT-base hidden layer in a more detailed manner, the present invention reduces the situation where excessive semantic changes are caused by directly changing the word vectors, and enables the model to collaboratively train labeled and unlabeled samples while improving robustness and generalization performance, thereby minimizing overfitting of some data during the training process.

[0072] Considering the noise processing problem in the predicted labels, the present invention introduces the symmetric cross entropy in the field of image noisy learning and combines it with the interpolation process to form the symmetric cross entropy in the two-hot mode. The more detailed weight selection during the interpolation process makes the training process always slightly biased towards the true label, which further reduces the impact of the noise in the predicted labels on the training process and improves the robustness of the model training process.

Claims

1. A noisy semi-supervised text classification method based on BERT-base network, characterized by The steps include: S1, data initialization stage: back-translate the unlabeled samples and generate new samples, then input the new samples into the model, and finally convert the high-confidence output vectors into corresponding one-hot labels; S2, data enhancement stage: The FGM method is used to add perturbations and perform interpolation processing. In BERT, perturbations and interpolation processing are performed on specific Transformer hidden layers rather than word vector encoding layers; S3, loss function construction stage; loss function loss = l s +l sce , the first item is the model output for labeled samples and its real label Find the cross entropy, the second term is the interpolation label With interpolation model output The symmetric cross entropy of ; S4, model parameter update stage: S5, model application stage: Use the trained model to input the test text into the model. The category corresponding to the maximum probability in the model output vector is the category to which the prediction belongs; Step S2 is specifically implemented as follows: S21, BERT-base network is mainly composed of 1 word vector encoding layer and 12 hidden layers of Transformer architecture. Here for unlabeled samples Add perturbation r to the output of the third hidden layer of BERT-base j =ε·g j / ||g j ||, where That is, vector The L2 regularization of the hidden layer output h is used to find the gradient, ε is a positive hyperparameter, and ||·|| means finding the vector norm; suppose that the unlabeled sample The output of the third hidden layer is The result after disturbance is S22. Randomly select one of the 7th, 9th, and 12th hidden layers in the BERT-base network to interpolate the hidden layer outputs of unlabeled samples and labeled samples; let and They are samples with labels. and unlabeled samples The output of the kth layer in the hidden layer of the classification model, k∈{7,9,12}; the corresponding true label and predicted label are and 2. A noisy semi-supervised text classification method based on a BERT-base network according to claim 1, characterized in that Step S1 is specifically implemented as follows: S11, unlabeled samples After being translated into other languages ​​through machine translation tools and then translated back to the original language, the sample Get the sample where j∈{1,2,…,N u },N u is the number of unlabeled samples in a batch of samples; S12, the sample Input classification model to get output vector like That is, the classification model classifies the sample with a confidence level higher than the threshold τ, and assigns A one-hot prediction label Label At the same time, the corresponding samples and samples θ represents the model parameters; S13, if the sample is assigned a one-hot label, then the unlabeled samples Input model to get output vector The labeled samples Input model to get output vector 3. A noisy semi-supervised text classification method based on a BERT-base network according to claim 2, characterized in that The interpolation operation steps are as follows: S221, using Beta distribution to generate a random number λ between 0 and 1; S222. Take λ1=max{λ,1-λ}, λ2=min{λ,1-λ}, that is, λ1>λ2 and λ1+λ2=1; S223, Calculation S224, will Input the k+1th hidden layer, and obtain the interpolation model output through calculations of the following hidden layers; the entire process of step S2 is recorded as f'(·), then the interpolation model output is recorded as The two-hot interpolation label is recorded as 4. A noisy semi-supervised text classification method based on a BERT-base network according to claim 3, characterized in that Step S3 is specifically implemented as follows: S31, l sce =μ1l ce +μ2l rce ; The symmetric cross entropy consists of two parts, l ce is the cross entropy, l rce Anti-cross entropy; μ1 and μ2 are the hyperparameter weights of the two; the specific formula is: Cross Entropy: Inverse Cross Entropy: where i∈{1,2,…,N l }, j∈{1,2,…,N u },because is a two-hot label vector. There must be 0 elements in the label vector, so log0 exists in the calculation of anti-cross entropy. The hyperparameter A=log0 is defined, A<0, and Ι{·} means that if the condition in the curly brackets is met, 1 is returned, otherwise 0 is returned; S32, l s is the cross entropy for labeled samples; S33, calculate the final loss function loss: loss = l s +l sce = l s +μ1l ce +μ2l rce .

5. A noisy semi-supervised text classification method based on a BERT-base network according to claim 4, characterized in that Step S4 is specifically implemented as follows: S41 updates the model parameters θ, that is, η is the learning rate; S42. Repeat steps S1 to S4 in the next batch of samples, verify the prediction accuracy of the model after each batch of samples is trained, and save the model parameters if the current highest accuracy is updated; if the prediction accuracy of the model has not improved after 10 batches of training, end the training process.

6. A noisy semi-supervised text classification method based on a BERT-base network according to claim 4, characterized in that The effectiveness of the constructed two-hot mode anti-cross entropy is achieved as follows: l rce =-p i logλ1-p j logλ2-(1-p i -p j )TO Among them, z i and z j Represents the output of the model before softmax at the category to which the correct label and the predicted label belong, respectively, c Represents the output of the model before softmax in the wrong category; p i and p j Represents the final output of the model at the category to which the correct label and the predicted label belong, p c Represents the final output of the model at the error category; Because 0<λ1,λ2<1,A<0, in two-hot mode, p i and p j Basically in the range of 0 to 0.5, so: That is l rce The decrease in the output z of the correct category will make i and z j Rising, output z at the wrong category c Decreases, which is in line with the basic purpose of the loss function, l rce Can accelerate the gradient descent of loss; and The size of depends mainly on the first-order term, namely and p i and p j , and because 0<λ2<λ1<1, -(A-logλ1)>-(A-logλ2)>0, p i p j >0, Therefore, in most cases It holds true, that is, the anti-cross entropy is generally more biased towards the true label; The occasional misjudgment causes the model's prediction confidence for the wrong label to be low, so if p j The corresponding predicted label is a misjudgment, then the smaller first-order term can be Play a restrictive role.

Images