Zero-sample short text classification method based on balance calibration prompt and self-training

Abstract

The invention discloses a zero-sample short text classification method based on balance calibration prompt and self-training. The zero-sample short text classification method comprises the following steps: 1, packaging a short text data set by using a prompt template; 2, constructing a prompt classification model based on a pre-training model; 3, label prediction deviation of the prompt classification model is calibrated, and a data set is preliminarily predicted; 4, dividing the predicted confidence degree into high-confidence-degree data and low-confidence-degree data according to the model; 5, training a plurality of teacher models by using the high-self-confidence-degree data to guide the low-self-confidence-degree data to train student models; and 6, completing a final short text classification task by using the student model. According to the method, the short text classification performance can be improved under the scene that the short text data labels are completely missing.

Description

Technical Field

[0001] The present invention belongs to the field of zero-shot short text classification in practical applications, and specifically is a zero-shot short text classification method based on balanced calibration prompts and self-training. Background Art

[0002] In an era of rapid expansion of the Internet and social media, the generation of short text data is growing exponentially. This phenomenon highlights the increasing urgency of effective short text classification techniques, including applications such as sentiment analysis and intent classification. However, the inherent brevity of short texts (usually containing only a few dozen words) leads to a significant lack of semantic information, namely the data sparsity problem. At the same time, short text labels are difficult to obtain in practical applications, namely the label missing problem.

[0003] Considering the data sparsity problem, the short text classification method enriches the short text by introducing supplementary external knowledge. Since the pre-trained language model can incorporate a large amount of external knowledge into the feature representation of the short text during the pre-training process, the feature space of the short text can be further expanded. The pre-trained language model performs well in short text classification.

[0004] Considering the label missing problem, existing supervised methods are limited by the need for a large amount of labeled text. Then there is semi-supervised short text classification, which has significantly reduced the requirement for labeled short text and has attracted increasing attention in recent years. The success of semi-supervised methods stems from the utilization of a large amount of unlabeled short text, which enhances the generalization ability of the model. Although the annotation burden is reduced, semi-supervised methods still require manual work from domain experts. Therefore, it is a major challenge to achieve short text classification in a completely unlabeled zero-shot scenario.

[0005] Fortunately, since pre-trained language models are pre-trained on large-scale unlabeled data, they no longer require a large number of manually labeled documents for text classification tasks. Therefore, more and more research is turning to applying pre-trained language models to zero-shot scenarios. But when they are applied to downstream tasks, the downstream tasks of pre-trained language models usually do not match the upstream tasks. Although prompt learning can alleviate the gap between upstream and downstream tasks of pre-trained language models to a certain extent, existing prompt learning methods face significant prediction bias problems. Summary of the invention

[0006] In order to avoid the above-mentioned deficiencies in the prior art, the present invention provides a zero-sample short text classification method based on balanced calibration prompts and self-training, so as to perform preliminary classification of completely unlabeled data and efficiently utilize these unlabeled data so that the classification model can better adapt to downstream tasks, thereby improving the accuracy of short text classification.

[0007] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0008] The zero-sample short text classification method based on balanced calibration prompts and self-training of the present invention is characterized in that it is performed according to the following steps:

[0009] Step 1: Encapsulate the short text dataset using the prompt template:

[0010] Step 1.1: Given an unlabeled short text dataset and label sets ,in, and Respectively The Short text and The tags, and Respectively represent the unlabeled short text dataset The number of short texts and label sets in The number of labels in ;

[0011] Step 1.2: Construct a network consisting of several words and a token A piece of natural text that together serves as a custom template , using template For the unlabeled short text dataset After each short text in is encapsulated separately, the encapsulated short text dataset is obtained. ,in, express The Encapsulate short text;

[0012] Step 1.3: After random sampling, we get the encapsulated support short text dataset ,in, Indicates The package supports short text, Indicates the number of short texts supported by the package;

[0013] Step 2: Build a prompt classification model based on the pre-trained model , including: pre-trained masked language model, label mapper;

[0014] Step 2.1: The pre-trained mask language model Process it and get Word Prediction Embedding ;

[0015] Step 2.2: The label mapper Process and obtain The label is The predicted probability , so that the predicted probability of each label under Q encapsulated support short texts constitutes a dimension The probability matrix ;

[0016] Step 3: Utilize the prompt classification model right After making a preliminary prediction, use the probability matrix Calibrate the results after the initial prediction to obtain the calibrated prediction probability , so that the label corresponding to the maximum predicted probability after calibration is taken as Pseudo-labels ;

[0017] Step 4: Build a high confidence encapsulated short text dataset and low confidence encapsulated short text dataset ;

[0018] Step 5: Build m teacher models And use high confidence to encapsulate short text datasets Perform training to obtain m teacher models after training ,in, Indicates A teacher model, After training, A teacher model;

[0019] Step 6: Encapsulate short text dataset based on low confidence ,use Mentoring Student Model Train to obtain the trained student model ;

[0020] Step 7: Encapsulated short text Input the trained student model is processed and finally obtained The predicted label .

[0021] The zero-shot short text classification method based on balanced calibration prompts and self-training described in the present invention is also characterized in that the pre-trained mask language model in step 2.1 includes: a size of The word list, word embedding encoder, multiple Trasnformer encoders, and a model head composed of multiple linear layers, where the output dimension of the last linear layer of the model head is the size of the word list ;

[0022] Step 2.1.1: After being processed by the word embedding encoder, we get The dimensions are Word embedding encoding ,in, Indicates the maximum length of a short text. Represents the output dimension of the word embedding encoder;

[0023] Step 2.1.2: After being processed by multiple Trasnformer encoders in sequence, the first The dimensions are Transformer embedding encoding ;

[0024] Step 2.1.3: Input model header for processing and output The dimensions are Word embedding encoding .

[0025] Further, the step 2.2 includes:

[0026] Step 2.2.1: From Remove the mark One dimension at the location is Linear embedding of ;

[0027] Step 2.2.2: Remove the label set The serial number corresponding to each tag in the vocabulary ,in, express The serial number in the vocabulary;

[0028] Step 2.2.3: According to , from the linear embedding Extract the logical value of each label , and after After operation, we get Package supports short text The predicted probability of each label , thus obtaining the predicted probability of each label under Q encapsulated support short texts, and forming a dimension of The probability matrix ;in, Indicates Tags in The logical value in express The label is The predicted probability of .

[0029] Furthermore, the step 3 comprises:

[0030] Step 3.1: Take out The jth column vector in , and In After the probability values ​​are sorted in reverse order, the jth column vector is obtained ;in, Representation Matrix The probability stored in the sth row and jth column of ;

[0031] Step 3.2: Calculate the prompt classification model using formula (1) right The predicted bias value of :

[0032] (1)

[0033] In formula (1), express The probability stored in the sth row of represents the number of samples of a single category, and ;

[0034] Step 3.3: Input Prompt Classification Model And follow the process of step 2.1-step 2.2 to get the predicted probability of each label ;in, express The label is The predicted probability of

[0035] Step 3.4: Use formula (2) to get the calibrated predicted probability :

[0036] (2)

[0037] In formula (2), express The label is The calibrated predicted probability of

[0038] Step 3.5: The label corresponding to the maximum predicted probability after calibration is taken as Pseudo-labels .

[0039] Further, the step 4 comprises:

[0040] Step 4.1: Compute the prompt classification model right Confidence ,in, for The maximum classification probability, for The second largest classification probability;

[0041] Step 4.2: If Greater than threshold , then Add to the high confidence encapsulated short text dataset Otherwise, add it to the low confidence encapsulated short text dataset .

[0042] Further, the step 5 comprises:

[0043] Step 5.1: Classify the prompt into the model As the Teacher Model ;

[0044] Step 5.2: From Random sampling High-confidence encapsulated short texts and form a high-confidence encapsulated short text sampling set ;

[0045] Step 5.3: The High confidence encapsulated short text enter And process it according to step 2-step 3 to get The label is The calibrated predicted probability and its pseudo labels ;

[0046] Step 5.4: Calculate using formula (3) The cross entropy loss :

[0047] (3)

[0048] In formula (3), express Output Is the tag Pseudo-labels If so, then let =1, otherwise, let =0;

[0049] Step 5.5: Use gradient descent to Train and calculate To update the model parameters until Until convergence, we can get the first Teacher Model , and then get the m teacher models after training .

[0050] Further, the step 6 comprises:

[0051] Step 6.1: The Low confidence encapsulated short text enter is processed and obtained The optimal pseudo label for , thereby obtaining the optimal pseudo-label set ;

[0052] Step 6.2: The pseudo-label with the largest number of occurrences is regarded as The final pseudo label of ;

[0053] Step 6.3: Classify the prompt model As a student model ;

[0054] Step 6.4: enter and process according to step 2 - step 3 to obtain The label is The calibrated predicted probability ;

[0055] Step 6.5: Calculate the student model using equation (4) The cross entropy loss :

[0056] (4)

[0057] In formula (4), express Output Is the tag The final pseudo label of , if so, then =1, otherwise, =0;

[0058] Step 6.6: Use gradient descent to Train and calculate To update the model parameters until Until convergence, the trained student model is obtained .

[0059] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the zero-sample short text classification method, and the processor is configured to execute the program stored in the memory.

[0060] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the zero-sample short text classification method when the computer program is executed by a processor.

[0061] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0062] 1. The present invention comprehensively considers the difference in probability distribution of matching short text label pairs and unmatched short text label pairs, and designs a balanced calibration strategy to calibrate the prediction bias of prompt learning. This strategy can significantly improve the classification accuracy of prompt learning in zero-sample scenarios.

[0063] 2. This paper proposes a new self-training strategy to effectively integrate the information of high-confidence short texts into low-confidence short texts. This method not only reduces the impact of low-confidence short text noise, but also enriches its information. Importantly, this method not only improves accuracy but also simplifies computation time, making it a dual-benefit solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A schematic diagram of the framework of the present invention for calibrating the prediction bias of the prompt classification model. DETAILED DESCRIPTION

[0065] In this embodiment, a zero-sample short text classification method based on balanced calibration prompts and self-training is provided. The specific balanced calibration process is as follows: Figure 1 As shown, the following steps are included:

[0066] Step 1: Encapsulate the short text dataset using the prompt template:

[0067] Step 1.1: Given an unlabeled short text dataset and label sets ,in, and Respectively The Short text and The tags, and Represents the unlabeled short text dataset The number of short texts and label sets in In this example, we use the news dataset AGnews, which contains news titles and brief descriptions, divided into four categories: world, sports, business, and technology. , .

[0068] Step 1.2: Construct a network consisting of several words and a token A piece of natural text that together serves as a custom template , using template Unlabeled short text dataset After each short text in is encapsulated separately, the encapsulated short text dataset is obtained. ,in, express The Encapsulate short text; in this example, the template used ,in, The short text data to be filled, for example: ,but:

[0069] .

[0070] Step 1.3: After random sampling, we get the encapsulated support short text dataset ,in, Indicates The package supports short text, Indicates the number of short texts supported by the package; in this example, .

[0071] Step 2: Build a prompt classification model based on the pre-trained model , including: pre-trained masked language model, label mapper;

[0072] Step 2.1: Pre-train the masked language model Process it and get Word Prediction Embedding , where the pre-trained mask language model includes: The vocabulary, word embedding encoder, multiple Trasnformer encoders, and a model head composed of multiple linear layers. The output dimension of the last linear layer of the model head is the size of the vocabulary. ; In this example, the pre-trained masked language model used is RoBerta, .

[0073] Step 2.1.1: After being processed by the word embedding encoder, we get The dimensions are Word embedding encoding ,in, Indicates the maximum length of a short text. represents the output dimension of the word embedding encoder; in this example, , ;

[0074] Step 2.1.2: After being processed by multiple Trasnformer encoders in sequence, the first The dimensions are Transformer embedding encoding ;

[0075] Step 2.1.3: Input model header for processing and output The dimensions are Word embedding encoding .

[0076] Step 2.2: Label mapper pair Process and obtain The label is The predicted probability , so that the predicted probability of each label under Q encapsulated support short texts constitutes a dimension The probability matrix ;

[0077] Step 2.2.1: From Remove the mark One dimension at the location is Linear embedding of ;

[0078] Step 2.2.2: Remove the label set The serial number corresponding to each tag in the vocabulary ,in, express The ordinal number in the vocabulary.

[0079] Step 2.2.3: According to , from the linear embedding Extract the logical value of each label , and after After operation, we get Package supports short text The predicted probability of each label , thus obtaining the predicted probability of each label under Q encapsulated support short texts, and forming a dimension of The probability matrix ;in, Indicates Tags in The logical value in express The label is The predicted probability of .

[0080] Step 3: Utilize the prompt classification model right After making a preliminary prediction, use the probability matrix Calibrate the results after the initial prediction to obtain the calibrated prediction probability , so that the label corresponding to the maximum predicted probability after calibration is taken as Pseudo-labels ;

[0081] Step 3.1: Take out The jth column vector in , and In After the probability values ​​are sorted in reverse order, the jth column vector is obtained ;in, Representation Matrix The probability stored in the s-th row and j-th column of .

[0082] Step 3.2: Calculate the prompt classification model using formula (1) right The predicted bias value of :

[0083] (1)

[0084] In formula (1), express The probability stored in the sth row of represents the number of samples of a single category, and ; Here is an estimate. Since it is unclear how much data a certain category has been sampled, for a balanced data set, The number of samples of a single category is calculated in this way. In this example, , ,but .

[0085] Step 3.3: Input Prompt Classification Model And follow the process of step 2.1-step 2.2 to get the predicted probability of each label ;in, express The label is The predicted probability of .

[0086] Step 3.4: Use formula (2) to get the calibrated predicted probability :

[0087] (2)

[0088] In formula (2), express The label is The calibrated predicted probability.

[0089] Step 3.5: The label corresponding to the maximum predicted probability after calibration is taken as Pseudo-labels .

[0090] Step 4: Build a high confidence encapsulated short text dataset and low confidence encapsulated short text dataset ; High confidence data has the characteristics of less noise and less information, and low confidence data has the characteristics of more noise and more information. In order to take advantage of the advantages of the two types of data and avoid the disadvantages of the two types of data, the encapsulated short text data is divided into and .

[0091] Step 4.1: Compute the prompt classification model right Confidence ,in, for The maximum classification probability, for The second largest classification probability;

[0092] Step 4.2: If Greater than threshold , then Add to the high confidence encapsulated short text dataset Otherwise, add it to the low confidence encapsulated short text dataset In this example, .

[0093] Step 5: Build m teacher models And use high confidence to encapsulate short text datasets Perform training to obtain m teacher models after training ,in, Indicates A teacher model, After training, teacher model; in order to avoid step 6.2 There are two cases where the number of labels is equal, and m must be set to an odd number. In this example, .

[0094] Step 5.1: Classify the prompt into the model As the Teacher Model ;

[0095] Step 5.2: From Random sampling High-confidence encapsulated short texts and form a high-confidence encapsulated short text sampling set In this example, is much smaller than On the one hand, it is because too large a number of samples will cause multiple high-confidence encapsulated short text sampling sets to lose diversity, and on the other hand, it is to reduce the training time of a single teacher model.

[0096] Step 5.3: The High confidence encapsulated short text enter And process it according to step 2-step 3 to get The label is The calibrated predicted probability and its pseudo labels ;

[0097] Step 5.4: Calculate using formula (3) The cross entropy loss :

[0098] (3)

[0099] In formula (3), express Output Is the tag Pseudo-labels If so, then let =1, otherwise, let =0.

[0100] Step 5.5: Use gradient descent to Train and calculate To update the model parameters until Until convergence, we can get the first Teacher Model , and then get the m teacher models after training .

[0101] Step 6: Encapsulate short text dataset based on low confidence ,use Mentoring Student Model Train to obtain the trained student model ;

[0102] Step 6.1: The Low confidence encapsulated short text enter is processed and obtained The optimal pseudo label for , thereby obtaining the optimal pseudo-label set ;

[0103] Step 6.2: The pseudo-label with the largest number of occurrences is regarded as The final pseudo label of .

[0104] Step 6.3: Classify the prompt model As a student model ;

[0105] Step 6.4: enter and process it according to step 2 - step 3 to obtain The label is The calibrated predicted probability .

[0106] Step 6.5: Calculate the student model using equation (4) The cross entropy loss :

[0107] (4)

[0108] In formula (4), express Output Is the tag The final pseudo label of , if so, then =1, otherwise, =0;

[0109] Step 6.6: Use gradient descent to Train and calculate To update the model parameters until Until convergence, the trained student model is obtained .

[0110] Step 7: Encapsulated short text Input the trained student model is processed and finally obtained The predicted label .

[0111] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0112] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

[0113] The test results of the present invention are further described in conjunction with the following charts:

[0114] In order to verify the effectiveness of the balance calibration of the present invention, a comparative experiment was conducted, which included two aspects: (1) learning using only prompts; (2) learning using prompts with calibration; (3) learning using prompts with balance calibration;

[0115] Table 1 Accuracy of different prompt learning strategies on multiple short text datasets (unit %)

[0116]

[0117] Agnews in Table 1: This is a news dataset. It contains a title and a short description. It is divided into 4 categories: world, sports, business, and technology.

[0118] Dbpedia: A dataset constructed by selecting text from 14 categories in Wikipedia.

[0119] Snippets: This dataset consists of 12,340 short text messages in 8 categories published by the Google search engine.

[0120] Tagmynews: This dataset contains 32,600 English news stories divided into 7 categories: sports, business, United States, entertainment, world, health and science.

[0121] Tweets: is a series of tweets published on TREC Twitter. The dataset is divided into 5 classes.

[0122] The experimental results in Table 1 show that the prompt learning method using balanced calibration outperforms other methods on all datasets. Specifically, it achieves a 6% improvement on the dbpedia dataset over the prompt learning using calibration, and a 2% and 1% improvement on snippets and tagmynews, respectively.

Claims

1. A zero-shot short text classification method based on balanced calibration prompts and self-training, characterized in that: The steps are as follows: Step 1: Encapsulate the short text dataset using the prompt template: Step 1.1: Given an unlabeled short text dataset and label sets ,in, and Respectively The Short text and The tags, and Respectively represent the unlabeled short text dataset The number of short texts and label sets in The number of labels in ; Step 1.2: Construct a set of words and a token A piece of natural text that together serves as a custom template , using template For the unlabeled short text dataset After each short text in is encapsulated separately, the encapsulated short text dataset is obtained. ,in, express The Encapsulate short text; Step 1.3: After random sampling, we get the encapsulated support short text dataset ,in, Indicates The package supports short text, Indicates the number of short texts supported by the package; Step 2: Build a prompt classification model based on the pre-trained model , including: pre-trained masked language model, label mapper; Step 2.1: The pre-trained mask language model Process it and get Word Prediction Embedding ; Step 2.2: The label mapper Process and obtain The label is The predicted probability , so that the predicted probability of each label under Q encapsulated support short texts constitutes a dimension The probability matrix ; Step 3: Utilize the prompt classification model right After making a preliminary prediction, use the probability matrix Calibrate the results after the initial prediction to obtain the calibrated prediction probability , so that the label corresponding to the maximum predicted probability after calibration is taken as Pseudo labels ; Step 4: Build a high confidence encapsulated short text dataset and low confidence encapsulated short text dataset ; Step 5: Build m teacher models And use high confidence to encapsulate short text datasets Perform training to obtain m teacher models after training ,in, Indicates A teacher model, After training, A teacher model; Step 6: Encapsulate short text dataset based on low confidence ,use Mentoring Student Model Train to obtain the trained student model ; Step 7: Encapsulated short text Input the trained student model is processed and finally obtained The predicted label .

2. A zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 1, characterized in that: The pre-trained mask language model in step 2.1 includes: The word list, word embedding encoder, multiple Trasnformer encoders, and a model head composed of multiple linear layers, where the output dimension of the last linear layer of the model head is the size of the word list ; Step 2.1.1: After being processed by the word embedding encoder, we get The dimensions are Word embedding encoding ,in, Indicates the maximum length of a short text. Represents the output dimension of the word embedding encoder; Step 2.1.2: After being processed by multiple Trasnformer encoders in sequence, the first The dimensions are Transformer embedding encoding ; Step 2.1.3: Input model header for processing and output The dimensions are Word embedding encoding .

3. A zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 2, characterized in that: The step 2.2 comprises: Step 2.2.1: From Remove the mark One dimension at the location is Linear embedding of ; Step 2.2.2: Remove the label set The serial number corresponding to each tag in the vocabulary ,in, express The serial number in the vocabulary; Step 2.2.3: According to , from the linear embedding Extract the logical value of each label , and after After operation, we get Package supports short text The predicted probability of each label , thus obtaining the predicted probability of each label under Q encapsulated support short texts, and forming a dimension of The probability matrix ;in, Indicates Tags in The logical value in express The label is The predicted probability of .

4. A zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 3, characterized in that: The step 3 comprises: Step 3.1: Take out The jth column vector in , and In After the probability values ​​are sorted in reverse order, the jth column vector is obtained ;in, Representation Matrix The probability stored in the sth row and jth column of ; Step 3.2: Calculate the prompt classification model using formula (1) right The predicted bias value of : (1) In formula (1), express The probability stored in the sth row of represents the number of samples of a single category, and ; Step 3.3: Input Prompt Classification Model And follow the process of step 2.1-step 2.2 to get the predicted probability of each label ;in, express The label is The predicted probability of Step 3.4: Use formula (2) to get the calibrated prediction probability : (2) In formula (2), express The label is The calibrated predicted probability of Step 3.5: The label corresponding to the maximum predicted probability after calibration is taken as Pseudo labels .

5. A zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 4, characterized in that: The step 4 comprises: Step 4.1: Compute the prompt classification model right Confidence ,in, for The maximum classification probability, for The second largest classification probability; Step 4.2: If Greater than threshold , then Add to the high confidence encapsulated short text dataset Otherwise, add it to the low confidence encapsulated short text dataset .

6. A zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 5, characterized in that: The step 5 comprises: Step 5.1: Classify the prompt into the model As the Teacher Model ; Step 5.2: From Random sampling High-confidence encapsulated short texts and form a high-confidence encapsulated short text sampling set ; Step 5.3: The High confidence encapsulated short text enter And process it according to step 2-step 3 to get The label is The calibrated predicted probability and its pseudo labels ; Step 5.4: Calculate using formula (3) The cross entropy loss : (3) In formula (3), express Output Is the tag Pseudo labels If so, then let =1, otherwise, let =0; Step 5.5: Use gradient descent to Train and calculate To update the model parameters until Until convergence, we can get the first Teacher Model , and then get the m teacher models after training .

7. A zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 6, characterized in that: The step 6 comprises: Step 6.1: The Low confidence encapsulated short text enter is processed and obtained The optimal pseudo label for , thereby obtaining the optimal pseudo-label set ; Step 6.2: The pseudo-label with the largest number of occurrences is regarded as The final pseudo label of ; Step 6.3: Classify the prompt model As a student model ; Step 6.4: enter and process it according to step 2 - step 3 to obtain The label is The calibrated predicted probability ; Step 6.5: Calculate the student model using equation (4) The cross entropy loss : (4) In formula (4), express Output Is the tag The final pseudo label of , if so, then =1, otherwise, =0; Step 6.6: Use gradient descent to Train and calculate To update the model parameters until Until convergence, the trained student model is obtained .

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the zero-sample short text classification method according to any one of claims 1 to 7, and the processor is configured to execute the program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the zero-sample short text classification method according to any one of claims 1 to 7 are executed.

Images