Zero-shot short text classification method based on balanced calibration hints and self-training

Abstract

This invention discloses a zero-shot short text classification method based on balanced calibration prompts and self-training. The method comprises the following steps: 1. Encapsulating a short text dataset using a prompt template; 2. Building a prompt classification model based on a pre-trained model; 3. Calibrating the label prediction bias of the prompt classification model and performing preliminary predictions on the dataset; 4. Classifying high-confidence data into low-confidence data based on the model's confidence in the prediction; 5. Using the high-confidence data to train multiple teacher models to guide the training of student models using the low-confidence data; and 6. Using the student models to complete the final short text classification task. This method can improve short text classification performance in scenarios where short text data labels are completely missing.

Description

Technical Field

[0001] The present invention belongs to the field of zero-shot short text classification in practical applications, and specifically provides a zero-shot short text classification method based on balanced calibration prompts and self-training. Background Art

[0002] In the era of rapid expansion of the internet and social media, the generation of short text data is growing exponentially. This phenomenon highlights the increasing urgency of effective short text classification techniques, including applications such as sentiment analysis and intent classification. However, the inherent brevity of short texts (typically containing only a few dozen words) leads to a significant lack of semantic information, known as data sparsity. Furthermore, short text labels are difficult to obtain in practical applications, known as the missing label problem.

[0003] To address data sparsity, short text classification methods enrich short texts by introducing supplementary external knowledge. Pretrained language models can incorporate a large amount of external knowledge into the feature representation of short texts during the pretraining process, further expanding the feature space of short texts. Pretrained language models have shown excellent performance in short text classification.

[0004] Existing supervised methods are limited by the requirement for large amounts of labeled text due to the lack of labels. Semi-supervised short text classification, however, significantly reduces the need for labeled short text and has attracted increasing attention in recent years. The success of semi-supervised methods stems from leveraging large amounts of unlabeled short text, enhancing the model's generalization capabilities. While reducing the annotation burden, semi-supervised methods still require manual effort from domain experts. Therefore, achieving short text classification in completely unlabeled, zero-shot scenarios is a significant challenge.

[0005] Fortunately, because pre-trained language models are pre-trained on large amounts of unlabeled data, they no longer require large numbers of manually labeled documents for text classification tasks. Consequently, a growing body of research is turning to applying pre-trained language models to zero-shot scenarios. However, when applied to downstream tasks, the downstream tasks of pre-trained language models often do not match the upstream tasks. While hint learning can alleviate this gap between upstream and downstream tasks of pre-trained language models to some extent, existing hint learning methods suffer from significant prediction bias. Summary of the Invention

[0006] In order to avoid the shortcomings of the above-mentioned existing technologies, the present invention provides a zero-shot short text classification method based on balanced calibration prompts and self-training, so as to perform preliminary classification of completely unlabeled data and efficiently utilize these unlabeled data, so that the classification model can better adapt to downstream tasks, thereby improving the accuracy of short text classification.

[0007] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0008] The zero-shot short text classification method based on balanced calibration prompts and self-training of the present invention is characterized in that it is performed according to the following steps:

[0009] Step 1: Encapsulate the short text dataset using the prompt template:

[0010] Step 1.1: Given an unlabeled short text dataset and label sets ,in, and Respectively The Short text and The tags, and Respectively represent the unlabeled short text dataset The number of short texts and label sets in The number of labels in ;

[0011] Step 1.2: Construct a list of words and a token A piece of natural text that together serves as a custom template , using templates For the unlabeled short text dataset After each short text in is encapsulated separately, the encapsulated short text dataset is obtained. ,in, express The Encapsulate short text;

[0012] Step 1.3: After random sampling, we get the encapsulated support short text dataset ,in, Indicates the This package supports short text, Indicates the number of short texts supported by the package;

[0013] Step 2: Build a prompt classification model based on the pre-trained model , including: pre-trained masked language model, label mapper;

[0014] Step 2.1: The pre-trained mask language model Process and obtain Word Prediction Embedding ;

[0015] Step 2.2: The label mapper Process and obtain The label is The predicted probability , thus the predicted probability of each label under Q encapsulated support short texts constitutes a dimension of The probability matrix ;

[0016] Step 3: Utilize the prompt classification model right After making a preliminary prediction, use the probability matrix Calibrate the results after the initial prediction to obtain the calibrated prediction probability , so that the label corresponding to the maximum predicted probability after calibration is taken as Pseudo labels ;

[0017] Step 4: Build a high-confidence encapsulated short text dataset and low confidence encapsulated short text dataset ;

[0018] Step 5: Build m teacher models And use high confidence to encapsulate short text dataset Perform training to obtain m teacher models after training ,in, Indicates the A teacher model, After training, a teacher model;

[0019] Step 6: Encapsulate short text dataset based on low confidence ,use Mentoring Student Model Train to obtain the trained student model ;

[0020] Step 7: Encapsulated short text Input the trained student model is processed and finally obtained The predicted label .

[0021] The zero-shot short text classification method based on balanced calibration prompts and self-training described in the present invention is also characterized in that the pre-trained mask language model in step 2.1 includes: a size of The vocabulary, word embedding encoder, multiple Trasnformer encoders, and a model head composed of multiple linear layers, where the output dimension of the last linear layer of the model head is the size of the vocabulary ;

[0022] Step 2.1.1: After being processed by the word embedding encoder, we get The dimensions are Word embedding encoding ,in, Indicates the maximum length of a short text. represents the output dimension of the word embedding encoder;

[0023] Step 2.1.2: After being processed by multiple Trasnformer encoders in sequence, the first The dimensions are Transformer embedding code ;

[0024] Step 2.1.3: Input model header for processing, output The dimensions are Word embedding encoding .

[0025] Furthermore, the step 2.2 includes:

[0026] Step 2.2.1: From Remove the mark One dimension at the location is Linear embedding of ;

[0027] Step 2.2.2: Get the label set The corresponding serial number of each tag in the vocabulary ,in, express The serial number in the vocabulary;

[0028] Step 2.2.3: According to , from linear embedding Extract the logical value of each label , and after After operation, we get Package supports short text The predicted probability of each label , thus obtaining the predicted probability of each label under Q encapsulated support short texts, and forming a dimension of The probability matrix ;in, Indicates the Tags in The logical value in express The label is The predicted probability of .

[0029] Furthermore, the step 3 includes:

[0030] Step 3.1: Remove The jth column vector in , and in After the probability values ​​are sorted in reverse order, the sorted j-th column vector is obtained ;in, Representation matrix The probability stored in the sth row and jth column of ;

[0031] Step 3.2: Calculate the prompt classification model using formula (1) right The predicted bias value of :

[0032] (1)

[0033] In formula (1), express The probability stored in the sth row in , represents the number of samples of a single category, and ;

[0034] Step 3.3: Input prompt classification model And follow the process of steps 2.1-2.2 to get the predicted probability of each label ;in, express The label is The predicted probability of

[0035] Step 3.4: Use formula (2) to get the calibrated predicted probability :

[0036] (2)

[0037] In formula (2), express The label is The calibrated predicted probability of

[0038] Step 3.5: The label corresponding to the maximum predicted probability after calibration is used as Pseudo labels .

[0039] Furthermore, the step 4 includes:

[0040] Step 4.1: Compute the prompt classification model right Confidence ,in, for The maximum classification probability, for The second largest classification probability;

[0041] Step 4.2: If Greater than threshold , then Add to the high confidence encapsulated short text dataset Otherwise, add it to the low confidence encapsulated short text dataset .

[0042] Furthermore, the step 5 includes:

[0043] Step 5.1: Classify the prompt into the model As the first Teacher Model ;

[0044] Step 5.2: From Random sampling High confidence encapsulated short texts and form a high confidence encapsulated short text sampling set ;

[0045] Step 5.3: The High confidence encapsulated short text enter And process it according to step 2-step 3 to get The label is The calibrated predicted probability and its pseudo labels ;

[0046] Step 5.4: Calculate using formula (3) Cross entropy loss :

[0047] (3)

[0048] In formula (3), express Output Is the tag Pseudo labels If so, then let =1, otherwise, let =0;

[0049] Step 5.5: Use gradient descent to Perform training and calculate To update the model parameters until Until convergence, the first Teacher Model , and then get the trained m teacher models .

[0050] Furthermore, the step 6 includes:

[0051] Step 6.1: The Low confidence encapsulated short text enter is processed and obtained The optimal pseudo label , thereby obtaining the optimal pseudo-label set ;

[0052] Step 6.2: The pseudo label with the largest number of occurrences is considered The final pseudo label ;

[0053] Step 6.3: Classify the prompt into the model As a student model ;

[0054] Step 6.4: enter and process it according to step 2 - step 3 to obtain The label is The calibrated predicted probability ;

[0055] Step 6.5: Calculate the student model using formula (4) Cross entropy loss :

[0056] (4)

[0057] In formula (4), express Output Is the tag The final pseudo label , if so, then =1, otherwise, =0;

[0058] Step 6.6: Use gradient descent to Train and calculate To update the model parameters until Until convergence, the trained student model is obtained .

[0059] The electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the zero-sample short text classification method, and the processor is configured to execute the program stored in the memory.

[0060] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the zero-sample short text classification method when the computer program is executed by a processor.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. This paper comprehensively considers the differences in the probability distributions of matching short text label pairs and mismatching short text label pairs, and designs a balanced calibration strategy to calibrate the prediction bias of prompt learning. This strategy can significantly improve the classification accuracy of prompt learning in zero-sample scenarios.

[0063] 2. This paper proposes a novel self-training strategy that effectively integrates information from high-confidence short texts into low-confidence short texts. This approach not only reduces the impact of noise in low-confidence short texts but also enriches their information. Importantly, this method not only improves accuracy but also simplifies computation time, making it a dual-benefit solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of the framework of the present invention for calibrating the prediction bias of the prompt classification model. DETAILED DESCRIPTION

[0065] In this embodiment, a zero-sample short text classification method based on balanced calibration prompts and self-training is provided. The specific balanced calibration process is as follows: Figure 1 As shown, the following steps are included:

[0066] Step 1: Encapsulate the short text dataset using the prompt template:

[0067] Step 1.1: Given an unlabeled short text dataset and label sets ,in, and Respectively The Short text and The tags, and Represents the unlabeled short text dataset The number of short texts and label sets in In this example, we use the news dataset AGnews, which contains news headlines and brief descriptions, divided into four categories: world, sports, business, and technology. 、 .

[0068] Step 1.2: Construct a list of words and a token A piece of natural text that together serves as a custom template , using templates Unlabeled short text dataset After each short text in is encapsulated separately, the encapsulated short text dataset is obtained. ,in, express The Encapsulates short text; in this example, the template used ,in, The short text data to be filled, for example: ,but:

[0069] .

[0070] Step 1.3: After random sampling, we get the encapsulated support short text dataset ,in, Indicates the This package supports short text, Indicates the number of short texts supported by the package; in this example, .

[0071] Step 2: Build a prompt classification model based on the pre-trained model , including: pre-trained masked language model, label mapper;

[0072] Step 2.1: Pre-train the masked language model Process and obtain Word Prediction Embedding , where the pre-trained mask language model includes: size The vocabulary, word embedding encoder, multiple Trasnformer encoders, and a model head composed of multiple linear layers. The output dimension of the last linear layer of the model head is the size of the vocabulary. ; In this example, the pre-trained masked language model used is RoBerta, .

[0073] Step 2.1.1: After being processed by the word embedding encoder, we get The dimensions are Word embedding encoding ,in, Indicates the maximum length of a short text. represents the output dimension of the word embedding encoder; in this example, , ;

[0074] Step 2.1.2: After being processed by multiple Trasnformer encoders in sequence, the first The dimensions are Transformer embedding code ;

[0075] Step 2.1.3: Input model header for processing, output The dimensions are Word embedding encoding .

[0076] Step 2.2: Label Mapper Process and obtain The label is The predicted probability , thus the predicted probability of each label under Q encapsulated support short texts constitutes a dimension of The probability matrix ;

[0077] Step 2.2.1: From Remove the mark One dimension at the location is Linear embedding of ;

[0078] Step 2.2.2: Get the label set The corresponding serial number of each tag in the vocabulary ,in, express The sequence number in the vocabulary.

[0079] Step 2.2.3: According to , from linear embedding Extract the logical value of each label , and after After operation, we get Package supports short text The predicted probability of each label , thus obtaining the predicted probability of each label under Q encapsulated support short texts, and forming a dimension of The probability matrix ;in, Indicates the Tags in The logical value in express The label is The predicted probability of .

[0080] Step 3: Utilize the prompt classification model right After making a preliminary prediction, use the probability matrix Calibrate the results after the initial prediction to obtain the calibrated prediction probability , so that the label corresponding to the maximum predicted probability after calibration is taken as Pseudo labels ;

[0081] Step 3.1: Remove The jth column vector in , and in After the probability values ​​are sorted in reverse order, the sorted j-th column vector is obtained ;in, Representation matrix The probability stored in the sth row and jth column in .

[0082] Step 3.2: Calculate the prompt classification model using formula (1) right The predicted bias value of :

[0083] (1)

[0084] In formula (1), express The probability stored in the sth row in represents the number of samples of a single category, and ; Here It is an estimated value. Since it is not clear how much data is sampled for a certain category, the balanced data set can be used The number of samples of a single category is calculated in this way. In this example, , ,but .

[0085] Step 3.3: Input prompt classification model And follow the process of steps 2.1-2.2 to get the predicted probability of each label ;in, express The label is The predicted probability of .

[0086] Step 3.4: Use formula (2) to get the calibrated predicted probability :

[0087] (2)

[0088] In formula (2), express The label is The calibrated predicted probability.

[0089] Step 3.5: The label corresponding to the maximum predicted probability after calibration is used as Pseudo labels .

[0090] Step 4: Build a high-confidence encapsulated short text dataset and low confidence encapsulated short text dataset ; High confidence data has the characteristics of less noise and less information, while low confidence data has the characteristics of more noise and more information. In order to take advantage of the advantages of the two types of data and avoid their disadvantages, the encapsulated short text data is divided into and .

[0091] Step 4.1: Compute the prompt classification model right Confidence ,in, for The maximum classification probability, for The second largest classification probability;

[0092] Step 4.2: If Greater than threshold , then Add to the high confidence encapsulated short text dataset Otherwise, add it to the low confidence encapsulated short text dataset In this example, .

[0093] Step 5: Build m teacher models And use high confidence to encapsulate short text dataset Perform training to obtain m teacher models after training ,in, Indicates the A teacher model, After training, teacher model; in order to avoid step 6.2 There are two cases where the number of labels is equal, so m must be set to an odd number. In this example, .

[0094] Step 5.1: Classify the prompt into the model As the first Teacher Model ;

[0095] Step 5.2: From Random sampling High confidence encapsulated short texts and form a high confidence encapsulated short text sampling set In this example, is much smaller than On the one hand, this is because a large number of samples will cause multiple high-confidence encapsulated short text sampling sets to lose diversity, and on the other hand, it is to reduce the training time of a single teacher model.

[0096] Step 5.3: The High confidence encapsulated short text enter And process it according to step 2-step 3 to get The label is The calibrated predicted probability and its pseudo labels ;

[0097] Step 5.4: Calculate using formula (3) Cross entropy loss :

[0098] (3)

[0099] In formula (3), express Output Is the tag Pseudo labels If so, then let =1, otherwise, let =0.

[0100] Step 5.5: Use gradient descent to Perform training and calculate To update the model parameters until Until convergence, the first Teacher Model , and then get the trained m teacher models .

[0101] Step 6: Encapsulate short text dataset based on low confidence ,use Mentoring Student Model Train to obtain the trained student model ;

[0102] Step 6.1: The Low confidence encapsulated short text enter is processed and obtained The optimal pseudo label , thereby obtaining the optimal pseudo-label set ;

[0103] Step 6.2: The pseudo label with the largest number of occurrences is considered The final pseudo label .

[0104] Step 6.3: Classify the prompt into the model As a student model ;

[0105] Step 6.4: enter and process it according to step 2 - step 3 to obtain The label is The calibrated predicted probability .

[0106] Step 6.5: Calculate the student model using formula (4) Cross entropy loss :

[0107] (4)

[0108] In formula (4), express Output Is the tag The final pseudo label , if so, then =1, otherwise, =0;

[0109] Step 6.6: Use gradient descent to Train and calculate To update the model parameters until Until convergence, the trained student model is obtained .

[0110] Step 7: Encapsulated short text Input the trained student model is processed and finally obtained The predicted label .

[0111] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0112] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0113] The test results of the present invention are further described in conjunction with the following charts:

[0114] In order to verify the effectiveness of the balance calibration of the present invention, a comparative experiment was conducted, which included two aspects: (1) learning with prompts only; (2) learning with prompts of calibration; and (3) learning with prompts of balance calibration.

[0115] Table 1 Accuracy of different prompt learning strategies on multiple short text datasets (unit %)

[0116]

[0117] Agnews in Table 1: This is a news dataset. It contains titles and short descriptions. It is divided into 4 categories: world, sports, business, and technology.

[0118] Dbpedia: A dataset constructed by selecting text from 14 categories of Wikipedia.

[0119] Snippets: This dataset consists of 12,340 short text messages in 8 categories published by the Google search engine.

[0120] Tagmynews: This dataset contains 32,600 English news stories categorized into seven categories: sports, business, United States, entertainment, world, health, and science.

[0121] Tweets: is a series of tweets published on TREC Twitter. The dataset is divided into 5 classes.

[0122] The experimental results in Table 1 show that the balanced calibrated hint learning method outperforms other methods on all datasets. Specifically, it achieves a 6% improvement on the dbpedia dataset, a 2% improvement on snippets, and a 1% improvement on tagmynews.

Claims

1. A zero-shot short text classification method based on balanced calibration prompts and self-training, characterized in that The steps are as follows: Step 1: Encapsulate the short text dataset using the prompt template: Step 1.1: Given an unlabeled short text dataset and label sets ,in, and Respectively The Short text and The tags, and Respectively represent the unlabeled short text dataset The number of short texts and label sets in The number of labels in ; Step 1.2: Construct a list of words and a token A piece of natural text that together serves as a custom template , using templates For the unlabeled short text dataset After each short text in is encapsulated separately, the encapsulated short text dataset is obtained. ,in, express The Encapsulate short text; Step 1.3: After random sampling, we get the encapsulated support short text dataset ,in, Indicates the This package supports short text, Indicates the number of short texts supported by the package; Step 2: Build a prompt classification model based on the pre-trained model , including: pre-trained masked language model, label mapper; Step 2.1: The pre-trained mask language model Process and obtain Word Prediction Embedding ; Step 2.2: The label mapper Process and obtain The label is The predicted probability , thus the predicted probability of each label under Q encapsulated support short texts constitutes a dimension of The probability matrix ; Step 3: Utilize the prompt classification model right After making a preliminary prediction, use the probability matrix Calibrate the results after the initial prediction to obtain the calibrated prediction probability , so that the label corresponding to the maximum predicted probability after calibration is taken as Pseudo labels ; Step 4: Build a high-confidence encapsulated short text dataset and low confidence encapsulated short text dataset ; Step 5: Build m teacher models And use high confidence to encapsulate short text dataset Perform training to obtain m teacher models after training ,in, Indicates the A teacher model, After training, a teacher model; Step 6: Encapsulate short text dataset based on low confidence ,use Mentoring Student Model Train to obtain the trained student model ; Step 7: Encapsulated short text Input the trained student model is processed and finally obtained The predicted label .

2. A zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 1, characterized in that: The pre-trained mask language model in step 2.1 includes: The vocabulary, word embedding encoder, multiple Trasnformer encoders, and a model head composed of multiple linear layers, where the output dimension of the last linear layer of the model head is the size of the vocabulary ; Step 2.1.1: After being processed by the word embedding encoder, we get The dimensions are Word embedding encoding ,in, Indicates the maximum length of a short text. represents the output dimension of the word embedding encoder; Step 2.1.2: After being processed by multiple Trasnformer encoders in sequence, the first The dimensions are Transformer embedding code ; Step 2.1.3: Input model header for processing, output The dimensions are Word embedding encoding .

3. The zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 2, characterized in that: The step 2.2 includes: Step 2.2.1: From Remove the mark One dimension at the location is Linear embedding of ; Step 2.2.2: Get the label set The corresponding serial number of each tag in the vocabulary ,in, express The serial number in the vocabulary; Step 2.2.3: According to , from linear embedding Extract the logical value of each label , and after After operation, we get Package supports short text The predicted probability of each label , thus obtaining the predicted probability of each label under Q encapsulated support short texts, and forming a dimension of The probability matrix ;in, Indicates the Tags in The logical value in express The label is The predicted probability of .

4. The zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 3, characterized in that: The step 3 comprises: Step 3.1: Remove The jth column vector in , and in After the probability values ​​are sorted in reverse order, the sorted j-th column vector is obtained ;in, Representation matrix The probability stored in the s-th row and j-th column of ; Step 3.2: Calculate the prompt classification model using formula (1) right The predicted bias value of : (1) In formula (1), express The probability stored in the sth row in , represents the number of samples of a single category, and ; Step 3.3: Input prompt classification model And follow the process of steps 2.1-2.2 to get the predicted probability of each label ;in, express The label is The predicted probability of Step 3.4: Use formula (2) to get the calibrated predicted probability : (2) In formula (2), express The label is The calibrated predicted probability of Step 3.5: The label corresponding to the maximum predicted probability after calibration is used as Pseudo labels .

5. The zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 4, characterized in that: The step 4 comprises: Step 4.1: Compute the prompt classification model right Confidence ,in, for The maximum classification probability, for The second largest classification probability; Step 4.2: If Greater than threshold , then Add to the high confidence encapsulated short text dataset Otherwise, add it to the low confidence encapsulated short text dataset .

6. The zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 5, characterized in that: The step 5 comprises: Step 5.1: Classify the prompt into the model As the first Teacher Model ; Step 5.2: From Random sampling High confidence encapsulated short texts and form a high confidence encapsulated short text sampling set ; Step 5.3: The High confidence encapsulated short text enter And process it according to step 2-step 3 to get The label is The calibrated predicted probability and its pseudo labels ; Step 5.4: Calculate using formula (3) Cross entropy loss : (3) In formula (3), express Output Is the tag Pseudo labels If so, then let =1, otherwise, let =0; Step 5.5: Use gradient descent to Perform training and calculate To update the model parameters until Until convergence, the first Teacher Model , and then get the m teacher models after training .

7. The zero-shot short text classification method based on balanced calibration prompts and self-training according to claim 6, characterized in that: The step 6 comprises: Step 6.1: The Low confidence encapsulated short text enter is processed and obtained The optimal pseudo label , thereby obtaining the optimal pseudo-label set ; Step 6.2: The pseudo label with the largest number of occurrences is considered The final pseudo label ; Step 6.3: Classify the prompt into the model As a student model ; Step 6.4: enter and process it according to step 2 - step 3 to obtain The label is The calibrated predicted probability ; Step 6.5: Calculate the student model using formula (4) Cross entropy loss : (4) In formula (4), express Output Is the tag The final pseudo label , if so, then =1, otherwise, =0; Step 6.6: Use gradient descent to Perform training and calculate To update the model parameters until Until convergence, the trained student model is obtained .

8. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the zero-sample short text classification method according to any one of claims 1 to 7, and the processor is configured to execute the program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the zero-sample short text classification method according to any one of claims 1 to 7 are executed.

Images