A small sample text classification method based on knowledge comparison and enhanced prompts

Abstract

The application discloses a kind of based on knowledge contrast enhancement prompt small sample text classification method, belong to natural language processing technical field, including the following steps: S1: initialization continuous prompt;S2: knowledge template generation;S3: joint training optimization;S4: mask prediction.The application first utilizes BiLSTM to initialize a continuous prompt vector that can be learned and has relevance, then pre-training language model is used as knowledge base, automatically generates a set of positive and negative prompt templates, on this basis, combined with contrast learning and mask language model are jointly trained, finally obtain effective continuous prompt embedding for accurate mask prediction, and provide an effective method that can automatically construct continuous prompt template, and extensive experiments are carried out on 14 data sets, the results show that the accuracy of the method is improved by more than 3.5% than the optimal contrast model, solve the two major problems that continuous prompt is sensitive to initial parameters and easy to overfit in small sample environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a small sample text classification method based on knowledge comparison enhanced prompts. BACKGROUND

[0002] With the emergence of pre-trained language models (PLMs), such as Bert, RoBerta and GPT, fine-tuning based on pre-trained language models has been the standard solution paradigm for most natural language processing tasks for a long time. However, fine-tuning limits the downstream task to have sufficient labeled data to train the classifier added on top of the PLMs, so it is difficult to achieve satisfactory results in low-resource scenarios. Recently, the emerging GPT-3 has shown amazing few-shot learning ability and has inspired a new paradigm of natural language processing called "prompt learning", which converts the input into a fill-in-the-blank form by introducing a prompt template, thereby narrowing the gap between the downstream task and the pre-training, so as to more fully tap the internal knowledge of the pre-trained language model in the small sample scenario.

[0003] Prompt learning uses prompts to guide the model to generate the expected result, so how to design a suitable prompt template is crucial. Early work mainly focuses on constructing discrete prompt words as templates through manual setting or automatic search, which is time-consuming and laborious, and requires a large amount of validation data for evaluation. Since neural networks are essentially continuous, discrete prompts may not be the best; recently, some research has proposed a trainable continuous prompt vector as a template, avoiding the complex manual template construction process. However, due to the limited number of samples in small sample tasks, the continuous prompt vector is sensitive to the initial parameters and prone to overfitting, and the above problems need to be solved, and therefore a small sample text classification method based on knowledge comparison enhanced prompts is proposed. SUMMARY

[0004] The technical problem to be solved by the present application is how to solve the problem that the continuous prompt vector is sensitive to the initial parameters and prone to overfitting due to the limited number of samples in small sample tasks, and a small sample text classification method based on knowledge comparison enhanced prompts is provided, which integrates external knowledge into the continuous prompt embedding through contrastive learning, explores continuous prompts with knowledge and rich semantics, and improves the accuracy of small sample natural language processing tasks.

[0005] The present application solves the above technical problems by the following technical solutions, and the present application comprises the following steps:

[0006] S1: initialize continuous prompts

[0007] using a bidirectional long short-term memory network to initialize the learnable and contextually relevant continuous prompt embeddings;

[0008] S2: knowledge template generation

[0009] Using a pre-trained language model as a knowledge base, a set of opposite labels is set to automatically generate positive and negative prompt templates required for contrastive learning;

[0010] S3: joint training optimization

[0011] Using the generated positive and negative prompt templates, the continuous prompt embeddings are jointly trained using contrastive learning and masked language models to obtain trained continuous prompts;

[0012] S4: mask prediction

[0013] Using the continuous prompt embeddings trained in step S3, the input sentence is masked to predict the label word probability of the filled mask token position, and the text classification task is completed.

[0014] Further, in the step S1, the specific process is as follows:

[0015] S11: Given a set of inputs (x, y), the text classification task is converted into a fill-in-the-blank task by designing a template function T(·) and a label mapping function M(·), wherein T(·) is used to convert the sentence x into a model input T(x) with a mask token, and M(·) is used to map the label to the corresponding label word M(y) in the model vocabulary;

[0016] S12: Use the masked language model to obtain the probability p of the label word at the mask token position MLM , the calculation formula is as follows:

[0017] p(y|x)=p MLM ([MASK]=M(y)|T(x))

[0018] Where p MLM represents the output probability of the masked language model, and p(y|x) represents the probability of outputting the label y given x.

[0019] S13: Use a bidirectional long short-term memory network with multiple perceptrons as a prompt encoder to initialize the virtual token, the formula is as follows:

[0020]

[0021] Where, and represent the output of the forward LSTM network and the backward LSTM network for the i-th virtual token t i , respectively, hi represents the prompt embedding obtained after the prompt encoder for each virtual token;

[0022] S14: The model input T(x) after the initialization of the continuous prompt embedding is represented as:

[0023] T(x) = {e(x), h1, …, h i , e([MASK]), h i+1 , …, h m+1}

[0024] where e(x) and e([MASK]) are the original embedding vectors of the sentence x and [MASK] in the MLM.

[0025] Further, in the step S11, the prompt template generated by the template function is a discrete template word or a continuous virtual token.

[0026] Further, in the step S2, the specific processing process is as follows:

[0027] S21: Pre-training a language model on a large corpus to obtain a language model as a knowledge base;

[0028] S22: Using the language model as a prompt template generator, adding placeholders in the input, and setting opposite label words to generate a set of positive and negative prompt templates.

[0029] Further, in the step S22, the specific process is as follows: the input (x, y) is converted by the conversion function G(·) as follows:

[0030] <x>→<x><X>M(y)<Y>

[0031]

[0032] where, represents the opposite label of the original label y, the first conversion formula is used to generate a positive prompt template, and the second conversion formula is used to generate a negative prompt template.

[0033] Further, in the step S3, the specific processing process is as follows:

[0034] S31: Based on the generated positive and negative prompt templates and the initialized continuous prompt, the input (T(x), T + (x), T - (x)) of the mask language model is constructed;

[0035] S32: The hidden state of the mask language model at the mask token position for each input is taken as their prompt embedding vector (v, v+ ,v - ), the continuous prompt is optimized through contrast training, the model is trained through a contrast loss function, and the continuous prompt embedding v i is close to the positive prompt embedding and far from the negative prompt embedding

[0036] S33: Combine the mask language model loss The continuous prompt is optimized through joint training, and the trained continuous prompt is obtained.

[0037] Further, in the step S32, the calculation of the contrast loss function in the contrast training optimization is as follows:

[0038]

[0039]

[0040] Wherein, N is the training batch size, τ represents the temperature coefficient used to control the discrimination degree of the model to the positive and negative samples, sim(v i ,v j ) is the cosine similarity, v i , respectively represent the prompt embedding vectors of the i th th sentence.

[0041] Further, in the step S33, the joint training optimization formula is as follows:

[0042]

[0043] Wherein, argmin(·) is the minimum value index function, is the continuous prompt embedding obtained through joint training.

[0044] Compared with the prior art, the small sample text classification method based on knowledge contrast enhanced prompt has the following advantages: firstly, a continuous prompt vector with learnability and correlation is initialized by using BiLSTM, then a pre-trained language model is used as a knowledge base to automatically generate a set of positive and negative prompt templates, and on this basis, contrast learning and mask language model are combined for joint training, finally, effective continuous prompt embedding is obtained for accurate mask prediction, and an effective method for automatically constructing continuous prompt templates is provided, and extensive experiments are carried out on 14 data sets, and the results show that the accuracy of the method is improved by more than 3.5% compared with the optimal contrast model, solving the two problems of sensitivity of continuous prompt to initial parameters and easy overfitting in small sample environment. BRIEF DESCRIPTION OF DRAWINGS

[0045] ​Figure 1 is a flowchart of a small sample text classification based on knowledge comparison and enhanced prompts in embodiment one of the present application;

[0046] Figure 2 (a) is a flowchart of knowledge prompt generation in embodiment one of the present application;

[0047] Figure 2 (b) is a framework diagram of a knowledge comparison and enhanced prompt model in embodiment one of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application will be described in detail below, and the embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation manners and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0049] Embodiment one

[0050] As shown in Figure 1 , the present embodiment provides a technical solution: a small sample text classification method based on knowledge comparison and enhanced prompts, which includes the following main steps:

[0051] (1) Initialize continuous prompts

[0052] A bidirectional long short-term memory neural network (BiLSTM) is used to initialize continuous prompt embeddings that are learnable and have relevance;

[0053] (2) Knowledge template generation

[0054] A pre-trained language model is used as a knowledge base, and a set of opposite labels is set to automatically generate positive and negative prompt templates required for contrastive learning;

[0055] (3) Joint training and optimization

[0056] The generated positive and negative prompt templates are used to combine contrastive learning and masked language models (MLM) to jointly train the continuous prompt embeddings, resulting in knowledge-enhanced and semantically rich continuous prompt embeddings.

[0057] The specific processing procedures of each step are as follows:

[0058] (1) Initialize continuous prompts

[0059] Given a set of input examples (x, y), the prompt learning converts the classification task into a fill-in-the-blank task by designing a template function T(·) and a label mapping function M(·), where T(·) is used to convert the sentence x into a model (masked language model) input T(x) with a "[MASK]" mark, and M(·) is used to map the label y into the corresponding label word M(y) in the model vocabulary. For example, for a movie review "A pleasure to watch.", the prompt template can be set as follows: "A pleasure to watch.[t1]…[t i ][MASK][t i+1 ]…[t m+1 ]", where m is a hyperparameter representing the number of template words. After such conversion, the probability p MLM of the label word at the "[MASK]" position can be directly obtained by the masked language model (MLM), and the calculation formula is as follows:

[0060] p(y|x)=p MLM ([MASK]=M(y)|T(x)) (1)

[0061] where p MLM represents the output probability of the masked language model, and p(y|x) represents the probability of the output label y given x.

[0062] Since the continuous prompt composed of randomly initialized virtual markers has instability, the present application uses a bidirectional long short-term memory network with a multilayer perceptron (MLP) as a prompt encoder to initialize the virtual markers, so as to eliminate the discreteness between the virtual markers and increase their relevance, and the formula is as follows:

[0063]

[0064] where and represent the output of the forward LSTM network and the backward LSTM network for the i-th virtual marker t i , respectively, and h i represents the prompt embedding obtained after each virtual marker passes through the prompt encoder, therefore, the model input T(x) after the continuous prompt embedding initialization can be represented as:

[0065] T(x)={e(x),h1,…,h i ,e([MASK]),h i+1 ,…,h m+1} (3)

[0066] where e(x) and e([MASK]) are the original embedding vectors of the sentence x and [MASK] in the MLM.

[0067] (2) Knowledge template generation

[0068] Pre-training on a large corpus of text results in a language model that is equivalent to a knowledge base, such as T5, which can serve as a kind of text-to-text converter. For example, given an input "Thank you <x>me to your party <y>"week.", T5 will be on the placeholder <x>and <y>The completion is performed to generate a complete sentence "Thank you for inviting me to your party last week." The goal of the knowledge template generation is to automatically generate a set of positive and negative prompt templates that can maximize the probability of positive and negative labels by using a pre-trained language model. Therefore, the application uses T5 as a prompt template generator, which only needs to add placeholders in the input and set opposite label words to generate a set of positive and negative prompt templates. The specific steps are as follows:

[0069] First, the input example (x, y) is converted by the conversion function G(·) as follows:

[0070] <x>→<x> <X> M(y) <Y> (4)

[0071]

[0072] Wherein, represents the opposite label of the original label y, formula (4) is used to generate a positive prompt template, and formula (5) is used to generate a negative prompt template.

[0073] As Figure 2 (a) shows, for the sentiment binary classification task, "terrible" and "great" are defined as a set of opposite label words, and by maximizing the probability of each set of template words (t1,…,t |T| ) generated by the model on the training set D train , the prompt template suitable for all training samples is obtained, and the calculation is as follows:

[0074]

[0075] Wherein, p T5 represents the output probability of the T5 model, P T represents the probability of generating the template T, and |T| represents the length of the template. A large number of candidate templates T * can be generated by controlling P T . When , a negative prompt template is generated. In order to filter out effective prompt templates with opposite semantics, the generated templates are further reordered using the validation set D dev to obtain prompt templates that can maximize the probability of labels, and the calculation formula is as follows:

[0076]

[0077] Wherein, argmax(·) is the maximum value index function, and when y i is the original label, the optimal positive prompt template T + is obtained. When When the optimal negative prompt template T is obtained. - .

[0078] (3) Joint training optimization

[0079] like Figure 2 As shown in (b), based on the generated optimal positive and negative cue templates and the initialized continuous cue, the input (T(x), T) of the MLM can be constructed. + (x),T - (x)), and use the hidden state of the MLM at the "[MASK]" position for each input as their cue embedding vector (v,v) + ,v - This is used to perform comparative training and optimization on continuous prompts, comparing the loss function. The calculation is as follows:

[0080]

[0081] Where N is the training batch size, τ represents the temperature coefficient used to control the model's (knowledge contrast enhancement prompt model) discrimination between positive and negative samples, and sim(v i ,v j ) represents the cosine similarity, v i , Representing the i-th th The sentence's cue embedding vector. The model is trained using a contrastive loss function to embed consecutive cues into v. i Approaching its positive hint embedding And stay away from negative prompt embedding

[0082] Furthermore, combining MLM loss Joint training optimization for continuous prompts:

[0083]

[0084] Where argmax(·) is the minimum value index function. The continuous cue embedding is obtained through joint training.

[0085] Finally, the trained continuous cue embedding is used to perform MLM prediction on the input sentence to obtain the probability of the label word filling the "[mask]" position.

[0086] Example 2

[0087] The present embodiment respectively carries out extensive experimental evaluation on 7 single sentence classification task data sets (SST-2, SST-5, MR, CR, MPQA, Subj, TREC) and 7 sentence pair classification task data sets (SNLI, MNLI, MNLI-mm, QNLI, RTE, MRPC, QQP), wherein the two data sets MRPC and QQP are uneven in class distribution, and F1-score is used as the evaluation index, and the evaluation indexes of the other data sets are all accuracy Acc.

[0088] In order to evaluate the effectiveness of the method of the present application, for different types of data sets, the present application respectively uses accuracy Acc and F1-score (F1) as evaluation indexes, which are defined as follows:

[0089]

[0090]

[0091] Among them, TP represents the number of sample pieces whose true label is True and whose predicted label is also True, FN represents the number of sample pieces whose true label is True and whose predicted label is False; TN represents the number of sample pieces whose true label is False and whose predicted label is also False, and FP represents the number of sample pieces whose true label is False and whose predicted label is True.

[0092] In order to fully verify the performance of the method (Our model) of the present application under small sample, the number of samples is set to 16 to generate the training set and the verification set, so as to simulate the small sample environment, wherein the number of samples in the test set remains consistent with that in the original test set. The present application uses Roberta-large as a unified pre-training language model, and compares with the existing prompt learning methods PET, P-tuning and standard fine-tuning, and the results are shown in Tables 1 and 2:

[0093] (1) Fine-tuning: standard fine-tuning method.

[0094] (2) PET: a fine-tuning method by manually constructing a prompt template.

[0095] (3) P-tuning: a method of automatically searching for continuous prompts for fine-tuning

[0096] Table 1 Experimental results on single sentence classification task

[0097]

[0098] Table 2 Experimental results on sentence pair classification task

[0099]

[0100] In summary, the above-mentioned embodiment of the small sample text classification method based on knowledge contrast enhanced prompt first initializes a continuous prompt vector with BiLSTM, then uses a pre-trained language model as a knowledge base to automatically generate a set of positive and negative prompt templates, and then combines contrast learning and masked language model for joint training, finally obtains effective continuous prompt embedding for accurate masked prediction, and provides an effective method that can automatically construct continuous prompt templates, and extensive experiments are conducted on 14 datasets, the results show that the accuracy of the method is improved by more than 3.5% compared with the optimal contrast model, solving the two problems of continuous prompt sensitivity to initial parameters and easy overfitting in small sample environment.

[0101] Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.< / y> < / x> < / y> < / x>

Claims

1. A small sample text classification method based on knowledge contrast enhanced prompts, characterized in that, The method comprises the following steps: S1: initialize continuous prompts Initialize the continuous prompt embedding with learnable and contextualized using bidirectional long short-term memory network; S2: knowledge template generation Use a pre-trained language model as a knowledge base, and automatically generate positive and negative prompt templates required for contrastive learning by setting a set of opposite labels; S3: joint training optimization Use the generated positive and negative prompt templates to jointly train the continuous prompt embedding using contrastive learning and masked language model, and obtain the trained continuous prompt; S4: mask prediction Use the continuous prompt embedding trained in step S3 to perform mask prediction on the input sentence to obtain the label word probability of filling the mask label position, and then complete the text classification task; In the step S1, the specific process is as follows: S11: Given a set of inputs (x, y), convert the text classification task into a fill-in-the-blank task by designing a template function T(·) and a label mapping function M(·), wherein T(·) is used to convert the sentence x into a model input T(x) with a mask mark, and M(·) is used to map the label to the corresponding label word M(y) in the model vocabulary; S12: Obtain the probability p of the mask token position label word by using the mask language model MLM The calculation formula is as follows: p(y | x) = p MLM ([MASK] = M(y) | T(x)) where p MLM represents the output probability of the masked language model, and p(y|x) represents the probability of the output label y given the condition x. S13: Use a bidirectional long short-term memory network with a multilayer perceptron as a prompt encoder to initialize the virtual mark, and the formula is as follows: wherein, and respectively represent the output of the forward LSTM network and the backward LSTM network for the i-th dummy token t i , h i represents the prompt embedding obtained after passing the prompt encoder for each dummy token. S14: The model input T(x) after continuous prompt embedding initialization is represented as: T(x) = {e(x), h1,..., h i ,e([MASK]),h i+1 ,…,h m+1} Where e(x) and e([MASK]) are the original embedding vectors of the sentence x and [MASK] in the MLM.

2. The small sample text classification method based on knowledge contrast enhanced prompts according to claim 1, characterized in that: In the step S11, the prompt template generated by the template function is a discrete template word or a continuous virtual mark.

3. The small sample text classification method based on knowledge contrast and enhanced prompts according to claim 1, characterized in that: In the step S2, the specific processing process is as follows: S21: Pre-train the language model on a large corpus to obtain the language model, which is used as a knowledge base; S22: Use the language model as a prompt template generator, add placeholders in the input, and set opposite label words to generate a set of positive and negative prompt templates.

4. The small sample text classification method based on knowledge contrast and enhanced prompts according to claim 3, characterized in that: In the step S22, the specific process is as follows: convert the input (x, y) through the conversion function G(·) as follows to serve as the input of the language model: <x>→<x><X>M(y)<Y> wherein, denotes the opposite label of the original label y, the first transformation formula is used to generate the positive prompt template, and the second transformation formula is used to generate the negative prompt template.

5. The small sample text classification method based on knowledge contrast and enhanced prompts according to claim 3, characterized in that: In the step S3, the specific processing process is as follows: S31: based on the generated positive and negative prompt templates and the initialized continuous prompts, construct the input (T(x), T + (x), T - (x)) of the masked language model; S32: The masked language model takes the hidden states of each input at the masked token positions as their prompt embedding vectors (v, v + ,v - ), and performs contrastive training optimization on consecutive prompts, trains the model through a contrastive loss function, so that the consecutive prompt embedding v i is close to the positive prompt embedding and far from the negative prompt embedding S33: Combine the mask language model loss The joint training optimization is performed on the continuous prompts to obtain trained continuous prompts.

6. The small sample text classification method based on knowledge contrast and enhanced prompts according to claim 5, characterized in that: In the step S32, the contrast loss function in the contrast training optimization is calculated as follows: L = -log (1 + exp (-y * x)) where N is the training batch size, τ represents a temperature coefficient to control the discrimination degree of the model to positive and negative samples, sim(v i ,v j ) is a cosine similarity, v i , represent the prompt embedding vectors of the i th th sentence, respectively.

7. The small sample text classification method based on knowledge contrast and enhanced prompts according to claim 5, characterized in that: In the step S33, the joint training optimization formula is as follows: where argmin(·) is the minimum index function, is the continuous prompt embedding obtained by joint training.

Images