A semi-supervised text classification method based on consistency regularization

Abstract

This paper introduces a semi-supervised text classification method based on consistency regularization, which includes the following steps: obtaining source text with strong augmentation; obtaining source text with weak augmentation; using a pre-trained language model to encode strong-weak augmented text pairs and measure the classification uncertainty of the model for text; screening unlabeled data based on uncertainty, combining the consistency alignment self-training objective of strong-weak augmented text pairs with the supervised training objective of labeled data, and thus obtaining a target text classifier. This method makes full use of the smoothness assumption and low-density separation assumption of the sample feature space, and for the first time applies text style transfer to consistency regularization. Under the premise of reliability constraints, it fully amplifies diversity, realizes semi-supervised text classification, and significantly improves model performance, providing an innovative idea for solving semi-supervised learning problems.

Description

Technical Field

[0001] The present invention relates to the technical field of text classification in a semi-supervised scenario, and specifically to a semi-supervised text classification method based on consistency regularization. Background Art

[0002] Semi-supervised learning is a form of machine learning task. Unlike traditional supervised learning, semi-supervised learning uses unlabeled data samples in addition to labeled data samples during training. In practical applications, collecting and labeling a large number of data samples requires a lot of time, manpower and resources. This form of task aims to overcome the shortage of labeled data by utilizing unlabeled data, providing an effective solution to the problems of data scarcity and high labeling costs.

[0003] Semi-supervised learning was first widely studied and applied in the field of images. In the field of images, semi-supervised learning methods assist the classification task of labeled images by utilizing the similarity between unlabeled images. Existing methods include generation-based EM algorithms, divergence-based collaborative training methods, and graph-based methods. These methods are usually based on the visual features of images, such as color, texture, and shape, to learn more information from unlabeled data to improve classification performance. In the field of natural language processing, semi-supervised learning is widely used in tasks such as text classification, sentiment analysis, and named entity recognition. For example, the unsupervised data augmentation (UDA) method uses the strategy of training signal annealing to enhance the data of unlabeled samples, generates samples that are different from the original samples but semantically similar, and then uses these enhanced samples for training; MixText enhances the diversity and robustness of data by using MixUp and other technologies in the feature space during the training process; in addition, the VAT (Virtual Adversarial Training) method introduces adversarial perturbations to make the model more robust to small changes in input data.

[0004] Due to the discreteness and complexity of text, some data augmentation methods used in existing semi-supervised text classification methods have some shortcomings. For example, traditional data augmentation methods such as synonym replacement and random insertion may introduce text differences, thereby affecting the performance of the model. Although the back-translation method used by UDA does not change the semantics, it has little effect on the semantics of the translated text and is easily affected by the translation quality. In addition, data augmentation from the perspective of manifold learning, such as adding adversarial perturbations or Mixup, although it shows good results in the image field, is not as intuitive and understandable in text as images. Summary of the invention

[0005] The present invention discloses a semi-supervised text classification method based on consistency regularization, and proposes a consistency regularization model training method. In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A semi-supervised text classification method based on consistency regularization includes the following steps:

[0007] S1. Use the text strong enhancement method to establish a text library with unlabeled strong enhancement;

[0008] S2. Based on the text weak enhancement method, a text library with unlabeled weak enhancement form is established, and the text library with unlabeled strong enhancement form is combined to obtain a text library with unlabeled strong-weak enhancement form;

[0009] S3. Encode text pairs in the unlabeled strong-weak augmented text library through a pre-trained language model and measure the classification uncertainty of the text;

[0010] S4. Combine the consistency alignment self-training objective of text pairs in the text library in the form of unlabeled strong-weak enhancement with the supervised training objective of labeled data to jointly train the text classification encoder to obtain a trained text classification encoder;

[0011] S5. Input the text into the trained text classification encoder to obtain the text classification prediction result.

[0012] Preferably, the text strong enhancement method in step S1 is specifically as follows: select the style to be used, randomly sample 1000 unlabeled texts, generate texts of different styles through the STRAP model, pre-classify the texts of different styles through the unsupervised sentence vector model SimCSE, and use the top three styles with the highest classification accuracy as candidates, and then select the style with the lowest similarity among the three candidate styles through the text similarity model P-SP to establish a text library in the form of unlabeled strong enhancement.

[0013] Preferably, the text weak enhancement method in step S2 is specifically as follows:

[0014] S21. Perform zero-shot translation based on the Seq2Seq model library fairseq based on the WMT19 model;

[0015] S22. Perform word segmentation using the Moses word segmentation algorithm based on maximum forward matching, encode using the FastBPE tool, and set the WMT19 model temperature to 0.9;

[0016] S23. Based on the English to German en-de and German to English de-en versions of the WMT19 model, generate texts through back translation, establish an unlabeled weakly enhanced text library, and combine the established unlabeled strongly enhanced text library to obtain an unlabeled strong-weakly enhanced text library.

[0017] Preferably, step S3 specifically includes the following steps:

[0018] S31. Define an instruction template for the text classification task dataset. When using the sentence vector model SimCSE for prediction, use the instruction template to guide the sentence vector model SimCSE to generate text. In the defined instruction template, use the placeholder "placeholder" to indicate the part to be filled, and use "mask" to fill in the position of the label information.

[0019] S32. For each text classification task data set, establish a respective vocabulary mapper, associate the original label of the text classification task data set with multiple description words in the vocabulary through the vocabulary mapper, and convert the Softmax probability of the classification task into the result word prediction probability of the text generation task in the form of a masked language model;

[0020] S33. Using the sentence embedding model SimCSE, select several samples with the highest probability value in each category from the unlabeled text by beam search to construct the support set S = ∪ i∈{1,...,c} top k p(y i |x), where: c represents the number of categories, i represents the i-th category sample, k represents the number of selections in each category, p(y i |x) is the sentence vector model SimCSE based on the input text x and the output result is y i The probability of , top means the highest, ∪ means the union, and then the prior of the mapped vocabulary is estimated by the mode of the masked language model MaskedLM:

[0021]

[0022]

[0023] Where: P(y) represents the category probability, v represents the mapping vocabulary obtained by the vocabulary mapper from the original label, V y represents the mapping vocabulary belonging to the original label y, P M The probability of predicting the position where the label information is filled in the "mask" after the Masked Language Model Masked LM receives the input text x is the word v;

[0024] S34. Estimate the output category probability for all unlabeled texts

[0025]

[0026] Where: y i ,y j Indicates that the text is predicted as the i, jth category, C represents the number of categories, P is the category probability predicted by the SimCSE model after receiving the input text x, and then the sample uncertainty u(x) is measured by entropy:

[0027]

[0028] in: represents the estimated probability of being predicted as the i-th category, and C represents the number of categories.

[0029] Preferably, step S4 comprises the following steps:

[0030] S41. Use the pre-trained Transformer-based RoBERTa-base model as the training base model, and use the supervised and unsupervised joint training mode;

[0031] For labeled text, calculate the cross entropy loss function between the RoBERTa-base model output and the true label:

[0032]

[0033] in: represents the supervision loss, B represents the batch size, represents the cross entropy, p b represents the probability of the true label of the one-hot encoding, p m represents the model output probability, y represents the predicted category, α() represents the weak enhancement method, x b Indicates labeled samples;

[0034] S42. For unlabeled text, the parameters of the RoBERTa-base model are optimized by the consistency loss of the output probability of the RoBERTa-base model by the strong-weak enhanced text from the same unlabeled text. Combined with the uncertainty described in S3, a threshold trigger function is set for the uncertainty and the model output category probability to screen the training samples. The threshold trigger function T(x u ) is defined as:

[0035]

[0036] in: represents the i-th unlabeled text, represents the switch control function that is 1 if the condition is met, p mrepresents the output probability of the RoBERTa-base model, y represents the predicted category, τ is the set confidence threshold, u() represents the sample uncertainty, and η is the set uncertainty threshold;

[0037] S43. Based on the threshold trigger function T(x u ), calculate the consistency loss function for unlabeled strong-weak text pairs

[0038]

[0039] in: Table 2 shows the text enhancement method, B represents the batch size, μ represents the relative ratio of the batch size of unlabeled and labeled text, b represents the parameter, represents cross entropy, y represents the predicted category, x u represents an unsupervised sample;

[0040] S44. Calculate the total training loss Where λ is a linear coefficient that jointly optimizes the model network parameters during training iterations.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention introduces the concept of text style transfer and applies it to consistency regularization, thereby significantly increasing the diversity of text while keeping the text semantics unchanged. By performing text style transfer on unlabeled text data, a source text in a strongly augmented form is obtained, which provides richer data for training models. A machine translation model is used to perform text back-translation, and a source text in a weakly augmented form is obtained. While introducing additional changes, the quality and diversity of the data are improved. During the model training process, a pre-trained language model is used to encode strongly augmented and weakly augmented text pairs, and the classification uncertainty of the model for the text is measured. Through uncertainty-based screening, unlabeled data is preferred to make it more conducive to the training and generalization capabilities of the model. The consistency alignment of strongly augmented and weakly augmented text pairs is used as the goal of self-training, and combined with the supervised training goal of labeled data, a target text classifier with high performance is trained. The present invention makes full use of the smoothness hypothesis and low-density separation hypothesis of the sample feature space, which brings a significant improvement effect to semi-supervised text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of the present invention.

[0044] Figure 2 This is the overall framework diagram of the model training of the present invention.

[0045] Figure 3This is the structural diagram of the text feature extractor model RoBERTa. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the embodiments.

[0047] like Figure 1 As shown, a semi-supervised text classification method based on consistency regularization includes the following steps:

[0048] S1. Use the text style transfer model to rewrite the style of unlabeled text;

[0049] S11. Prepare the STRAP model (Style Transfer via Paraphrasing) for text style transfer, using GPT-2_large as a decoder-only Seq2Seq model, where the maximum length of the context password is 1024, the dimension of the word embedding is 1280, a vocabulary of size 50257, and a 36-layer decoder stack. Both the input and output sequences are fed into a decoder network separated by special passwords and using different segment embeddings;

[0050] S12. Set the STRAP model to a two-stage mode: in the first stage, the target text is converted into a general text without style information through the base version of the STRAP model, and in the second stage, the general text is converted into the target style version through the target style version of the STRAP model. Assume that S input =(t1,...,t n ) indicates the input sequence password, and S output =(w bos ,w1,…,w m ,w eos ) indicates the output password, where w bos and w eos Corresponding to the special passwords for the start and end, in the generation phase, (t1,…,t n ,w bos ) Input the STRAP model, generate the command word by autoregression until the special command word w indicating the end is output eos , enter the password sequence S input And output password sequence S output The password in is passed through the shared input embedding layer to obtain a vector representation, encode the position and segment information, and add the learnable position and segment embedding to the input embedding consistent with the GPT-2 architecture. The segment embedding is used to indicate that the password belongs to the sequence S input Or S output ;

[0051] S13. Select the style to be used. Randomly sample 1,000 samples from the target dataset. Generate texts in styles such as "Shakespeare", "James Joyce", "Tweets", "Poetry", "Switchboard", "Lyrics", and "Bible" through the above-mentioned style transfer model STRAP. Pre-classify texts of different styles through the unsupervised sentence embedding model SimCSE, and use the top three styles with the highest classification accuracy as candidates. Use the text similarity model P-SP (PARAGRAM-SP) to select the candidate styles with the lowest similarity and establish a text library in a strongly enhanced form.

[0052] S2. Back-translate and rewrite the text based on the WMT model;

[0053] S21. Combined with relevant research, the WMT (Workshop on Statistical Machine Translation) 2019 model was selected as the translation model used, and zero-shot translation was performed based on the WMT19 model and the Seq2Seq model library fairseq;

[0054] S22. Words are segmented using the Moses segmentation algorithm based on maximum positive matching and encoded using the FastBPE tool. The model temperature is set to 0.9 to provide a certain degree of differentiation while ensuring output stability.

[0055] S23. Based on the English to German en-de and German to English de-en versions of WMT19, generate texts through back translation and establish a text library in weakly enhanced form.

[0056] S3. Encode the strong-weak augmented text pairs through the pre-trained language model and measure the classification uncertainty of the model for the text;

[0057] S31. Define the corresponding instruction template for the text classification task dataset. When performing SimCSE model prediction, use the instruction template to guide the model to generate specific text. Use the placeholder placeholder in the defined instruction template to represent the part to be filled. For example, define a template as follows: {"placeholder":"text_a"} In summary, it was {"mask"}. In this template, "placeholder":"text_a" means that there is a placeholder named "text_a", which will be replaced with the actual input text content in specific applications, and "mask" is used to fill in the position of the label information. The rest of the instruction template is a fixed text part;

[0058] S32. Establish a vocabulary mapper for the dataset label to convert the results generated by the model into a more readable text form, associate the original label with multiple descriptive words in the vocabulary, and convert the classification task into a result word prediction task in the form of a masked language model. For example, for the comment sentiment classification dataset, the model is expected to predict the result to be positive or negative. Define a vocabulary mapper containing words with positive and negative sentiment polarities as follows: "negative": ["bad", "terrible"], "positive": ["good", "wonderful", "great"], when the model generates a result representing a negative emotion, the vocabulary mapper maps it to "bad" or "terrible"; otherwise, it is mapped to "good", "wonderful" or "great". For each text classification dataset, establish its own vocabulary mapper.

[0059] S33. Using the same unsupervised SimCSE model as step S13, select several samples with the highest probability value in each class from the unlabeled samples by beam search to construct a support set S = ∪ i∈{1,...,c} top k p(y i |x), where: S represents the support set, c represents the number of categories, k represents the number of choices in each category, p(y i |x) is the SimCSE model based on the input text x and the output result is y i The probability of , and then the prior of the mapping vocabulary is estimated by the mode of the Masked Language Model Masked LM:

[0060]

[0061] in: P(y) represents the category probability, S is the support set, v represents the mapping vocabulary obtained by the label, V y represents the mapping word belonging to label y, P M The probability that the mask position "mask" filled with label information after the mask language model receives the input text x is predicted to be word v;

[0062] S34. Estimate the output category for all unlabeled samples:

[0063]

[0064] in: represents the estimated probability of being predicted as the i-th category, y i ,y j Indicates that the sample is predicted to be the i, jth category, C represents the number of categories, P is the category probability predicted by the model after receiving the input text x, and then the sample uncertainty is measured by entropy:

[0065]

[0066] Where u(x) represents the uncertainty of sample x, represents the estimated probability of being predicted as the i-th category, and C represents the number of categories.

[0067] S4. Combine the consistency alignment self-training objective of unlabeled strong-weak augmented text pairs with the supervised training objective of labeled data to jointly train the text classification encoder. The overall process is as follows Figure 2 As shown;

[0068] S41. Use the pre-trained Transformer-based RoBERTa-base model as the training base model, learn the text representation of the target dataset through supervised and unsupervised joint training. For labeled text, calculate the cross entropy loss function between the RoBERTa-base model output and the true label:

[0069]

[0070] in: represents the supervision loss, B represents the batch size, represents the cross entropy, p b represents the probability of the true label of the one-hot encoding, p m represents the model output probability, y represents the predicted category, α() represents the text weak enhancement method described in S2, and x b Indicates labeled samples;

[0071] Specifically: RoBERTa-base model adopts a modular structure, such as Figure 3 As shown in the figure, it consists of an Embedding layer, an attention-based Transformer encoding module, and a classification head with a fully connected layer + dropout operation + GELU activation function. The 768-dimensional Embedding layer is used to convert the input text sequence into a word embedding vector. It contains a 12-layer Transformer encoding layer. The encoding layer has an attention module with 12 self-attention heads to capture the contextual dependencies in the input sequence. The size of each layer of the feedforward neural network hidden layer is 3072, which is used to perform nonlinear transformation and feature extraction on the output of the attention mechanism. The classification head maps the output of the last hidden layer to the space of the predicted label, combines the dropout operation to reduce overfitting, and introduces nonlinear change properties through the GELU activation function.

[0072] S42. For unlabeled samples, the model parameters are optimized by the consistency loss of the model output probability of the strong-weak enhanced text from the same sample. Combined with the uncertainty described in S3, a threshold is set for the uncertainty and the model output category probability to screen the training samples. The threshold control is as follows:

[0073]

[0074] Where: T() represents the threshold trigger function, represents the i-th unlabeled sample, represents the switch control function that is 1 if the condition is met, p m represents the pseudo label output by the model, y represents the predicted category, α() represents the text weak enhancement method described in S2, τ is the set confidence threshold, u() represents the sample uncertainty described in S34, and η is the set uncertainty threshold;

[0075] S43. Based on the threshold T(x u ), the consistency loss function for unlabeled strong-weak text pairs is calculated as follows:

[0076]

[0077] in: represents the unlabeled consistency loss, represents the strong text enhancement method, α() represents the weak text enhancement method, T() represents the threshold trigger function, B represents the batch size, μ represents the relative ratio of the batch size of unlabeled and labeled text, represents the cross entropy, p m represents the model output pseudo label, y represents the predicted category, and x u represents an unsupervised sample;

[0078] S44. Calculate the total training loss by a linear combination of the supervision loss and the unlabeled consistency loss in: represents the total loss, and represents the above-mentioned supervision loss and unlabeled consistency loss, and λ is a linear coefficient, which jointly optimizes the model network parameters in training iterations.

[0079] Specifically: for parameter settings, the number of training iterations is 51200, the number of iterations in the learning rate warm-up phase is 2560, the evaluation iteration interval is 2048, the number of labeled samples is set to [20, 40, 100, 500, 1000], the number of unlabeled samples is [20, 40, 100, 200, 500, 1000, 10000, 50000], adjust the training combination with different numbers of labeled and unlabeled samples, the number of samples in each batch during training is 16, the temperature parameter is 0.5, which is used for pseudo-label sharpening, the uncertainty threshold is 0.7, the confidence threshold is 0.9, which is used to determine which pseudo-labels are accepted, the loss weight ratio of unlabeled data is 1.0, AdamW is selected as the optimizer, the momentum parameter is 0.9, the weight decay parameter is 0.0005, the multi-layer learning rate decay parameter is 0.65, and the maximum length of the input sequence is limited to 512.

[0080] S5. Input text into the encoder model to obtain text classification prediction results.

[0081] The semi-supervised text classification method based on consistency regularization emphasizes text diversity while maintaining the consistency of text semantics, introduces text style transfer and self-training strategies, and achieves remarkable model performance improvement.

[0082] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A semi-supervised text classification method based on consistency regularization, characterized in that: The following steps are involved: S1. Use the text strong enhancement method to build a text library with unlabeled strong enhancement; S2. Based on the text weak enhancement method, an unlabeled weakly enhanced text library is established, and an unlabeled strong-weakly enhanced text library is obtained by combining the unlabeled strong-enhanced text library with the unlabeled strong-enhanced text library; S3. Encode text pairs in the unlabeled strong-weak augmented text library through a pre-trained language model and measure the classification uncertainty of the text; S4. Combine the consistency alignment self-training objective of text pairs in the text library in the form of unlabeled strong-weak enhancement with the supervised training objective of labeled data to jointly train the text classification encoder to obtain a trained text classification encoder; S5. Input the text into the trained text classification encoder to obtain the text classification prediction result; Step S3 specifically includes the following steps: S31. Define an instruction template for the text classification task dataset. When using the sentence vector model SimCSE for prediction, use the instruction template to guide the sentence vector model SimCSE to generate text. In the defined instruction template, use the placeholder "placeholder" to indicate the part to be filled, and use "mask" to fill in the position of the label information. S32. For each text classification task dataset, establish a respective vocabulary mapper, associate the original label of the text classification task dataset with multiple description words in the vocabulary through the vocabulary mapper, and convert the Softmax probability of the classification task into the result word prediction probability of the text generation task in the form of a masked language model; S33. Use the sentence embedding model SimCSE to select several samples with the highest probability value in each category from the unlabeled text by beam search to construct a support set ,in: c represents the number of categories, i Indicates i Class sample, k represents the number of choices in each category, For the sentence embedding model SimCSE based on the input text x The output is The probability of top Denotes the highest, U denotes the union, and then the prior of the mapped vocabulary is estimated through the mode of the masked language model Masked LM: ， ， in: represents the class probability, represents the mapped vocabulary obtained by the vocabulary mapper for the original label, Indicates the original tag y The mapping vocabulary of The Masked LM receives the input text x The position of the "mask" filled with label information is predicted as a word probability; S34. Estimate the output category probability for all unlabeled texts : ， in: , Indicates that the text prediction is i , j kind, C represents the number of categories, P To accept input text x The class probabilities predicted by the post-SimCSE model, followed by entropy to measure sample uncertainty : ， in: Indicates the prediction i The estimated probability of the class, C Indicates the number of categories; Step S4 specifically includes the following steps: S41. Use the pre-trained Transformer-based RoBERTa-base model as the training base model, and use the supervised and unsupervised joint training mode; For labeled text, calculate the cross entropy loss function between the RoBERTa-base model output and the true label: ， in: represents the supervision loss, B represents the batch size, represents the cross entropy, represents the true label probability of one-hot encoding, represents the model output probability, y represents the predicted category, represents the weak enhancement method, Indicates labeled samples; S42. For unlabeled text, the parameters of the RoBERTa-base model are optimized by the consistency loss of the output probability of the RoBERTa-base model input by the strong-weak enhanced text from the same unlabeled text. Combined with the uncertainty described in S3, a threshold trigger function is set for the uncertainty and the model output category probability to screen the training samples. The threshold trigger function Defined as: ， in: Indicates i Unlabeled text, represents a switch control function that is 1 if the condition is met, Represents the output probability of the RoBERTa-base model, y represents the predicted category, is the confidence threshold set, represents the sample uncertainty, is the uncertainty threshold set; S43. Threshold-based triggering function , calculate the consistency loss function for unlabeled strong-weak text pairs : ， in: Table text strong enhancement method, B represents the batch size, is the relative ratio of the batch size of unlabeled and labeled text, b Indicates the parameters, represents the cross entropy, y represents the predicted category, represents an unsupervised sample; S44. Calculate the total training loss , where Linear coefficients, jointly optimize the model network parameters during training iterations.

2. A semi-supervised text classification method based on consistency regularization as described in claim 1, characterized in that: The specific method of text strong enhancement in step S1 is as follows: select the style to be used, randomly sample 1,000 unlabeled texts, generate texts of different styles through the STRAP model, pre-classify texts of different styles through the unsupervised sentence vector model SimCSE, and use the top three styles with the highest classification accuracy as candidates. Then, use the text similarity model P-SP to select the style with the lowest similarity among the three candidate styles, and establish a text library in the form of unlabeled strong enhancement.

3. A semi-supervised text classification method based on consistency regularization as described in claim 2, characterized in that: The text weak enhancement method in step S2 is specifically as follows: S21. Zero-shot translation based on the Seq2Seq model library fairseq based on the WMT19 model; S22. Segment the words using the Moses segmentation algorithm based on maximum forward matching, encode them using the FastBPE tool, and set the WMT19 model temperature to 0.9; S23. Based on the WMT19 model, the English to German en-de and German to English de-en versions are translated back to generate texts, and an unlabeled weakly enhanced text library is established. The unlabeled strong-weakly enhanced text library is combined with the established unlabeled strong-enhanced text library to obtain an unlabeled strong-weakly enhanced text library.

Images