Prompt learning small sample classification method, system, device and medium based on pre-training language model

Abstract

The application discloses a prompt learning small sample classification method, system, device and medium based on a pre-training language model, and the method comprises the following steps: preprocessing data samples according to a pre-established prompt template; inputting the preprocessed data samples into a pre-training language model to obtain label word prediction probabilities at the mask positions, and using the label word prediction probabilities to calculate calibration parameters; constructing a classification model and training the classification model, wherein the classification model combines a traditional fine-tuning method and prompt learning, constructs a new model to fully learn downstream task knowledge and use the knowledge learned by pre-training; using a cross-entropy loss function to calculate a loss value during training, and using the loss value to update the classification model; and using the trained classification model to classify new samples. The application trains a classification model under small sample data by fusing the prompt learning and traditional fine-tuning method, effectively learns specific knowledge of a downstream task, and improves classification precision.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and particularly relates to a prompt learning small sample classification method, system, device and medium based on a pre-training language model. BACKGROUND

[0002] In recent years, artificial intelligence related technologies have developed rapidly, and in the field of natural language processing, a series of pre-training language models (BERT, RoBERTa, GPT, T5, etc.) have emerged, greatly promoting the development of natural language processing technologies. Due to the powerful ability of pre-training language models, they have become the main method to solve many natural language processing tasks.

[0003] The usual practice is to add a linear classifier at the head of the language model and then fine-tune the whole model to adapt to the downstream task, while a recent method, prompt learning, uses prompts to perform various downstream tasks and is considered to be able to release the potential of language models. Pre-training models acquire general language knowledge through word filling or continuation writing during the pre-training phase, while prompt learning constructs an input template to let the language model fill in the words, maps the filled words to specific labels of the classification task, and thus models the classification task as a cloze task. This approach reduces the gap between the pre-training phase and the downstream task phase, and achieves good results, especially in small sample training scenarios.

[0004] However, recent studies have shown that the word filling prediction of pre-training language models is biased, as it tends to predict words with high word frequency during the pre-training phase, resulting in unfair predictions, and prompt learning performance is not stable (more dependent on artificially constructed templates and label words). In addition, in the small sample scenario, prompt learning mainly uses the knowledge learned by the pre-training language model during the pre-training phase, and less knowledge can be learned in the downstream task, resulting in a significant decrease in the accuracy of small sample training compared to full sample training. Therefore, how to train a bias-free, stable and high-precision model in a small sample scenario is a difficult problem that needs to be solved. SUMMARY

[0005] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a prompt learning small sample classification method, system, device and medium based on a pre-training language model, which calculates model calibration parameters using training data to alleviate the prediction bias of the model and reduce the performance difference caused by different templates and label words, and then trains the classification model in a small sample data by fusing prompt learning and traditional fine-tuning, effectively learning the specific knowledge of the downstream task and improving the classification accuracy.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] In a first aspect, the present application provides a prompt learning small sample classification method based on a pre-trained language model, comprising the following steps,

[0008] The data samples are preprocessed according to a pre-established prompt template; the prompt template comprises an input sample and a masked word, and the preprocessed data sample refers to a data sample wrapped with the prompt template, so that the wrapped data sample contains a label word;

[0009] The preprocessed data sample is input into a pre-trained language model to obtain a label word prediction probability at the masking position, and a calibration parameter is calculated using the label word prediction probability;

[0010] A classification model is constructed and trained, the classification model is a combination of a traditional fine-tuning method and prompt learning, a new model is constructed to fully learn downstream task knowledge and utilize pre-trained knowledge; a cross-entropy loss function is used to calculate a loss value during training, and the classification model is updated using the loss value;

[0011] The trained classification model is used to classify new samples.

[0012] As a preferred technical solution, the data sample contains N categories, each sample is labeled with a category, and the number of samples in different categories is K, forming a K-way-N-shot small sample dataset, which contains a total of K*N training samples.

[0013] As a preferred technical solution, the prediction word probability is used to calculate the calibration parameter, which specifically comprises the following steps:

[0014] The preprocessed data sample is input into a pre-trained language model to obtain a label word logits corresponding to the masked word position, and the label word is one-to-one corresponding to the label, i.e. obtaining logits of the classification label;

[0015] A scaling coefficient λ is calculated to scale the calibrated logits back to the size of the original output logits of the model:

[0016]

[0017] Wherein, z is the logits of each label word at the masking position; N represents the number of categories of the sample, K represents the number of different category samples, and M represents the pre-trained language model; W v represents the word embedding of the predicted label word, which is also used to predict the word on the hidden layer state; h mask is the hidden layer state of the last layer of the model at the masking position; the diag function is a function of expanding a vector into a diagonal matrix; i represents the input sample number, and j represents the index of different labels, represents the logits of the jth label of the ith sample output by the model i represents the logits vector of the ith sample output by the model.

[0018] As a preferred technical solution, in the classification model, the output probability of the calibrated prompt learning calculated using the calibration parameter is:

[0019] p MLM =Softmax(Wz)

[0020] The prompt is constructed as a feature extractor, and the hidden layer state at the mask output by the pre-trained language model is taken as the feature, and a classifier f is constructed for classification:

[0021] p CLS =Softmax(f(h mask ))

[0022] And the two outputs are weighted and fused:

[0023] p(y∣∣x prompt )=α·p MLM +(1-α)·p CLS

[0024] α is a balance factor, and the above Softmax function is represented as:

[0025]

[0026] x c and x j are the label output logits corresponding to the index c and j of the label; since p MLM and p CLS both use h mask , the hidden layer state at the mask is shared in both ways, implicit multi-task learning is used, the classification task is divided into two sub-classification tasks, and the classification model has better generalization performance.

[0027] As a preferred technical solution, the training of the classification model includes the following steps:

[0028] Each batch of data is input into the constructed classification model to obtain the prediction probability distribution p i of each class;

[0029] Using a cross-entropy loss function, the loss value is calculated based on the label corresponding to each sample and the probability distribution;

[0030] The loss value is used for back propagation and the gradient descent method is used to update the parameters of the entire classification model;

[0031] wherein the cross-entropy loss function is represented as:

[0032]

[0033] where p = [p0, ···, p C-1 ] represents the predicted probabilities of all classes, p i represents the predicted probability of the i-th class, y = [y0, ···, y C-1 ] is the one-hot representation of the sample class, y i = 1 when the sample belongs to the i-th class, otherwise y i = 0; C is the number of classes.

[0034] As a preferred technical solution, the training of the classification model comprises the following steps:

[0035] A black-box optimization model is constructed, specifically:

[0036] The projection matrix A and the local optimization parameter z l are randomly initialized, l e L, L is the number of pre-trained model layers, each layer needs to maintain a set of local optimization parameters, and a local classifier f is constructed;

[0037] The output of the black-box optimization model is:

[0038] p(y||x prompt ) = a p MLM + (1-a) p CLS

[0039] For the training of the black-box optimization model, since the classifier is constructed locally, its gradient is completely visible, and gradient and evolutionary algorithms can be used alternately to optimize. The optimization process is as follows:

[0040] a) The training data is input into the model in the form of mini-batch and the cross-entropy loss is calculated, and the back propagation algorithm is used to update the parameters of the local classifier f;

[0041] b) All training data is input into the model and the cross-entropy loss is calculated, and the CMA-ES algorithm is used to select the optimal z l is saved and used for the CMA-ES algorithm in the next round;

[0042] c) Iterate 3 epochs, and iterate one round of step b), i.e. use the CMA-ES algorithm to optimize once for each layer l of the model, which constitutes one complete optimization training step;

[0043] The training step in c) is iterated for several rounds.

[0044] As a preferred technical solution, the classifier uses an AdamW optimizer to optimize parameters, and evaluates each complete optimization training in the iterative training process on a validation set to select the model with the highest accuracy on the validation set for saving.

[0045] In a second aspect, the application provides a prompt learning small sample classification system based on a pre-trained language model, which is applied to the prompt learning small sample classification method based on the pre-trained language model and includes an input data acquisition module, a model calibration and construction module, a model training module, and a model classification and prediction module.

[0046] The input data acquisition module is configured to preprocess data samples according to a pre-established prompt template; the prompt template includes an input sample and a masked word, and the preprocessed data sample refers to a data sample wrapped with the prompt template so that the wrapped data sample contains a label word.

[0047] The model calibration and construction module is configured to input the preprocessed data sample into a pre-trained language model to obtain a label word prediction probability at the mask, and calculate a calibration parameter using the label word prediction probability.

[0048] The model training module is configured to construct and train a classification model, wherein the classification model combines a traditional fine-tuning method and prompt learning to construct a new model to fully learn downstream task knowledge and utilize pre-trained knowledge; and the model training module is configured to calculate a loss value using a cross-entropy loss function during training, and update the classification model using the loss value.

[0049] The model classification and prediction module is configured to classify new samples using the trained classification model.

[0050] In a third aspect, the application provides an electronic device, which includes:

[0051] at least one processor; and

[0052] a memory communicatively connected to the at least one processor; wherein

[0053] the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the prompt learning small sample classification method based on the pre-trained language model.

[0054] In a fourth aspect, the application provides a computer-readable storage medium storing a program, and the program is executed by a processor to implement the prompt learning small sample classification method based on the pre-trained language model.

[0055] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0056] The present application is applied to a small sample learning scene, and the initial model has a large prediction deviation and different artificial templates bring a large performance difference. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0058] Figure 1 is an example of the prompt learning for sentiment analysis in the embodiment of the present application;

[0059] Figure 2 is a flowchart of the prompt learning small sample classification method based on the pre-trained language model in the embodiment of the present application;

[0060] Figure 3 is a whole schematic diagram of the prompt learning small sample classification method based on the pre-trained language model in the embodiment of the present application;

[0061] Figure 4 is a whole schematic diagram of the prompt learning small sample classification method based on the pre-trained language model in the embodiment of the present application applied to model black box optimization.

[0062] Figure 5 is a structure schematic diagram of the prompt learning small sample classification system based on the pre-trained language model in the embodiment of the present application;

[0063] Figure 6 is a structure schematic diagram of the electronic device for implementing the prompt learning small sample classification method based on the pre-trained language model in the embodiment of the present application. DETAILED DESCRIPTION

[0064] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application, so that those skilled in the art can better understand the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0065] Reference to "an embodiment" in this application means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. Those skilled in the art will appreciate that the embodiments described herein are merely examples of the application and are not the only way in which the application can be practiced.

[0066] Large-scale pre-training language models have achieved great success in various fields of natural language processing. Few-shot learning is a paradigm of machine learning, which fine-tunes a model with a small amount of training samples to fit downstream tasks and obtain a model with high precision. Humans can easily establish cognition of new things with one or a few examples, while machine learning algorithms usually need a large amount of training data to train the model to obtain better generalization ability. In the field of machine learning, the cost of data annotation is high, especially in some fields, the annotated data is very scarce and difficult to obtain, and people hope that the machine learning model can be trained to have strong generalization ability and high precision with a small amount of samples.

[0067] The emergence of GPT-2 and GPT-3 has made it possible to use the prompt-based tuning method in downstream tasks, and this method can be successfully extended to MLM models (such as BERT, RoBERTa). Prompt-based tuning performs better than traditional fine-tuning in the few-shot scenario, because this method can narrow the gap between the pre-training stage and the downstream task stage of the model. Figure 1 An example of prompt learning for sentiment analysis is shown as follows: Figure 1 As shown, a template "[X] It was [MASK]." can be constructed, where "[X]" represents the input sample, and [MASK] is the masked word, and the model needs to predict the word at this position to predict the label, for example, in Figure 1In this embodiment, the selected label words are "great" and "terrible", and when the model predicts the word "great", the label is predicted to be positive, and vice versa. Similarly, this method and the method of the present application can be extended to Chinese or other languages, and will not be described one by one.

[0068] However, the initial model of this method has a large label word prediction bias, and different artificially constructed templates and label words bring large performance differences. Based on this, as shown in Figure 2 The embodiment proposes a small sample classification method based on pre-training language model prompt learning, including the following steps:

[0069] S1, pre-processing data samples according to prompt templates

[0070] S11, selecting templates and label words according to the characteristics of the classification task and the classification sample domain;

[0071] The templates and label words can be selected manually, and the external knowledge of human beings can be used to construct prompts. In addition, LM-BFF is an advanced technology for automatically generating templates and label words through T5 model, which can be used to automatically generate to avoid the bias of manual selection. In addition, the label words can not be limited to single words, and each class of label can correspond to multiple labels.

[0072] S12, wrapping the data sample with the prompt template;

[0073] As shown in Figure 1 The data wrapped with the prompt template should contain "[MASK]" for model prediction, and the sample should be segmented and the words should be converted into corresponding word table indexes, and an attention mask should be constructed to prepare for inputting the model.

[0074] Among them, the artificially constructed prompt template and the label word corresponding to each label are defined (the label word can also be automatically generated by using T5 model and the like), and taking sentiment analysis as an example, the prompt template is as follows:

[0075] Hello, my dog is cute. It was [MASK]. Among them, "Hello, my dog is cute" is the input sample, [MASK] is the masked word, and the language model needs to predict the label at this position, and the label word can be "great" and "terrible", which corresponds one-to-one to the classification labels "positive" and "negative".

[0076] Further, the small sample dataset is preprocessed. Generally, the training dataset is N-way-k-shot, containing K*N samples, N representing the number of label categories of the classification task, and K representing the number of samples of each label category. The template defined in step S1 packages these data.

[0077] S2, calculating calibration parameters using data samples and the model;

[0078] S21, inputting the preprocessed small dataset into the model to generate label word logits of each label category at the masking position:

[0079] z = M(x|prompt) = W v(y) *h mask

[0080] S22, calculating a scaling coefficient λ to scale the calibrated logits back to the size scale of the original logits output by the model:

[0081]

[0082] S23, calculating calibration parameters W:

[0083]

[0084] wherein z in the above formula is logits of each label word at the masking position; M represents a pre-trained language model; W v represents a word embedding of a predicted label word, which is also used to predict words on the hidden layer state; h mask is a hidden layer state of the last layer of the model at the masking position; the diag function is a function of expanding a vector into a diagonal matrix; i represents an input sample number, j represents an index of different labels, represents logits of the jth label of the ith sample output by the model, and z i represents a logits vector of the ith sample output by the model.

[0085] S3, constructing a model and training

[0086] S31, constructing a model

[0087] Prompt learning can better use the knowledge learned in the pre-training stage, but it does not learn enough specific knowledge of the downstream task, such as Figure 3 As shown, by combining the traditional fine-tuning method and prompt learning, a new model is constructed to fully learn the knowledge of the downstream task and effectively use the knowledge learned in the pre-training. The output probability of the calibrated prompt learning calculated using the calibration parameters in S2 is:

[0088] p MLM= Softmax(Wz)

[0089] The prompt is constructed as a feature extractor, taking the hidden layer state at the mask output by the pre-trained language model as the feature, and a classifier f is constructed for classification:

[0090] p CLS = Softmax(f(h mask ))

[0091] And the two outputs are weighted and fused:

[0092] p(y∣∣x prompt ) = α·p MLM + (1-α)·p CLS

[0093] Wherein, α ∈ (0, 1) is a hyperparameter for balancing the weights of the two, for a relatively simple task, α can be set to 0.5, and for a relatively complex task (such as sentence pair task), the value of α can be increased;

[0094] α is a balancing factor, and the above Softmax function is represented as:

[0095]

[0096] x c and x j are the label outputs logits corresponding to the indexes c and j of the labels; since p MLM and p CLS both use h mask , the hidden layer state at the mask is shared in both ways, implicit multi-task learning is used, the classification task is divided into two sub-classification tasks, which can make the model have better generalization performance.

[0097] S32, model training

[0098] The small sample dataset is trained using the mini-batch method and the cross entropy loss function (Cross Entropy Loss) to calculate the loss, specifically:

[0099] a) input each batch of data into the model in S31 to obtain the prediction probability distribution p i of each class;

[0100] b) use the cross entropy function to calculate the loss value corresponding to the label and the probability distribution of each sample;

[0101] c) the loss value is used for back propagation and the gradient descent method is used to update the parameters of the entire model.

[0102] The cross entropy loss function is represented as:

[0103]

[0104] where p = [p0, ···, p C-1 ] represents the predicted probability of all classes, p i represents the predicted probability of the i-th class, y = [y0, ···, y C-1 ] is the one-hot representation of the sample class, y i = 1 when the sample belongs to the i-th class, otherwise y i = 0; C is the number of classes.

[0105] In the iterative training process of the embodiment, the model uses the AdamW optimizer to optimize the parameters, the maximum number of steps is set to 500, and the model with the highest accuracy on the validation set is selected and saved every 50 steps in the iterative training process; the batch size is selected as [2, 4, 8], the learning rate is selected as [1e-5, 2e-5], and a is selected as [0.5, 0.7]; the above hyperparameters are increased according to the situation to increase the hyperparameter search space, and the grid-search hyperparameters are traversed to select the model trained by the optimal set of hyperparameters on the validation set for use; the training uses a linearly decaying learning rate to gradually decrease to 0.

[0106] S4, the to-be-predicted data sample is preprocessed according to S1 and input into the trained classification model to obtain a predicted classification result.

[0107] In another embodiment of the application, based on the same idea as the above embodiment, the method can be used in black-box optimization of a pre-trained language model; a large language model (LLM) is usually pre-trained by an Internet vendor and needs to be used by consumers in the form of an API (such as GPT3) for a fee. In order to protect the model from being stolen, the model weights and gradients are invisible to consumers calling the API, and black-box optimization can be used to train the model without the model weights and gradients being visible. The specific method is to optimize a set of parameters through an evolutionary algorithm, and the set of parameters is projected to a higher dimension as a prefix prompt of the model, so as to achieve the purpose of training the model.

[0108] As shown in Figure 4 , the method of the application can be applied to black-box optimization in this scenario, and because most of the weights of the model cannot be optimized, the method of the application can achieve better calibration effect, and the steps of the embodiment other than the following steps of constructing a model and training (corresponding to step S3 of the above embodiment of the prompt learning small sample classification method based on a pre-trained language model) are the same as those of the above embodiment, specifically:

[0109] S31, constructing a black-box optimization model

[0110] Randomly initialize projection matrix A and local optimization parameters z l l∈L, L is the number of pre-training model layers, each layer needs to maintain a set of local optimization parameters, and construct a local classifier f

[0111] As in the above embodiment, the black box optimization model output is:

[0112] p(y∣∣x prompt )=α·p MLM +(1-α)·p CLS

[0113] S32, black box optimization model training:

[0114] Since the classifier is constructed locally, its gradient is completely visible, and gradient and evolutionary algorithms can be used alternately to optimize:

[0115] a) The training data is input into the model in the form of a mini-batch and the cross-entropy loss is calculated, and the back propagation algorithm updates the parameters of the local classifier f

[0116] b) All training data is input into the model and the cross-entropy loss is calculated, and the CMA-ES algorithm is used to select the optimal z l Save and use the CMA-ES algorithm for the next round

[0117] c) a step is iterated for 3 epochs, and b step is iterated for one round, i.e. the CMA-ES algorithm is used to optimize each layer l of the model once, constituting a complete optimization training step

[0118] d) The training step in c is iterated for several rounds

[0119] In the iterative training process of the embodiment, the local classifier uses the AdamW optimizer to optimize the parameters, the learning rate is selected as 1e-5, the CMA-ES evolutionary algorithm population is selected as 20, and each complete optimization training in the iterative training process is evaluated on the validation set, and the model with the highest accuracy on the validation set is selected and saved.

[0120] Based on the same idea as the prompt learning small sample classification method based on the pre-trained language model in the above embodiment, the application also provides a prompt learning small sample classification system based on a pre-trained language model, which can be used to execute the prompt learning small sample classification method based on the pre-trained language model. For ease of illustration, in the structural diagram of the prompt learning small sample classification system embodiment based on the pre-trained language model, only the parts related to the embodiments of the application are shown, and those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and can include more or fewer components than the illustrated structure, or combine certain components, or different component arrangements.

[0121] As Figure 5 shown, the embodiment of the present application provides a prompt learning small sample classification system 100 based on a pre-trained language model, comprising an input data acquisition module 101, a model calibration and construction module 102, a model training module 103 and a model classification prediction module 104.

[0122] The input data acquisition module 101 is used to pre-process data samples according to a pre-established prompt template; the prompt template comprises an input sample and a masked word, and the pre-processed data sample refers to a data sample wrapped with the prompt template, so that the wrapped data sample contains a label word.

[0123] The model calibration and construction module 102 is used to input the pre-processed data sample into a pre-trained language model to obtain a label word prediction probability at the masking position, and calculate a calibration parameter using the label word prediction probability.

[0124] The model training module 103 is used to construct and train a classification model, wherein the classification model combines a traditional fine-tuning method and prompt learning to construct a new model to fully learn downstream task knowledge and utilize pre-trained knowledge; a cross-entropy loss function is used to calculate a loss value during training, and the classification model is updated using the loss value.

[0125] The model classification prediction module 104 is used to classify new samples using the trained classification model.

[0126] It should be noted that the prompt learning small sample classification system based on the pre-trained language model of the present application corresponds to the prompt learning small sample classification method based on the pre-trained language model of the present application, and the technical features and advantages described in the embodiment of the prompt learning small sample classification method based on the pre-trained language model are applicable to the embodiment of the prompt learning small sample classification system based on the pre-trained language model, and the specific content can be referred to the description in the method embodiment of the present application. Therefore, it is declared here.

[0127] In addition, in the embodiment of the prompt learning small sample classification system based on the pre-trained language model of the above embodiment, the logical division of each program module is only an example, and in actual application, the above functions can be completed by different program modules according to needs, for example, considering the configuration requirements of the corresponding hardware or the convenience of software implementation, that is, the internal structure of the prompt learning small sample classification system based on the pre-trained language model is divided into different program modules to complete all or part of the functions described above.

[0128] As Figure 6As shown, in another embodiment of the present application, an electronic device 200 for a prompt learning small sample classification method based on a pre-trained language model is provided, which can include a first processor 201, a first memory 202 and a bus, and can further include a computer program stored in the first memory 202 and executable on the first processor 201, such as a prompt learning small sample classification program 203 based on a pre-trained language model.

[0129] The first memory 202 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the first memory 202 can include both an internal storage unit and an external storage device of the electronic device 200. The first memory 202 can be used not only to store application software and various data installed in the electronic device 200, such as the code of the prompt learning small sample classification program 203 based on a pre-trained language model, but also to temporarily store data that has been output or will be output.

[0130] The first processor 201 can be composed of an integrated circuit in some embodiments, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more combinations of a central processing unit (CPU), a microprocessor, a digital processing chip, a graphics processor, and various control chips, etc. The first processor 201 is the control unit of the electronic device, which connects various components of the entire electronic device through various interfaces and lines, and executes various functions and processes data of the electronic device 200 by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202.

[0131] Figure 6 Only an electronic device with components is shown, and those skilled in the art can understand that, Figure 5The structure shown does not constitute a limitation on the electronic device 200, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0132] The prompt learning small sample classification program 203 stored in the first memory 202 in the electronic device 200 is a combination of multiple instructions, which, when running in the first processor 201, can implement:

[0133] Preprocess the data sample according to the pre-established prompt template; the prompt template includes an input sample and a masked word, and the preprocessed data sample refers to wrapping the data sample with the prompt template, so that the wrapped data sample contains a label word;

[0134] Input the preprocessed data sample into the pre-trained language model to obtain a label word prediction probability at the mask, and calculate a calibration parameter using the label word prediction probability;

[0135] Construct and train a classification model, which is a combination of traditional fine-tuning methods and prompt learning to construct a new model to fully learn downstream task knowledge and utilize pre-trained knowledge; a cross-entropy loss function is used to calculate a loss value during training, and the classification model is updated using the loss value;

[0136] Use the trained classification model to classify new samples.

[0137] Further, the modules / units of the electronic device 200, if implemented in the form of software function units and sold or used as independent products, can be stored in a non-volatile computer readable storage medium. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0139] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0140] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principles of the present application shall be equivalent replacement methods and shall be within the scope of protection of the present application.

Claims

1. A cue-based few-shot classification method based on a pre-trained language model, characterized in that, Includes the following steps, Data samples are preprocessed according to a pre-established prompt template; the prompt template includes an input sample and masked words, and the preprocessing of data samples refers to packaging data samples with the prompt template so that the packaged data samples contain tag words; The preprocessed data samples are input into the pre-trained language model to obtain the predicted probability of the occluded label words, and the calibration parameters are calculated using the predicted probability of the label words. A classification model is constructed and trained, which combines traditional fine-tuning methods with cue learning to build a new model that fully learns downstream task knowledge and utilizes the knowledge learned during pre-training; during training, the cross-entropy loss function is used to calculate the loss value, and the loss value is used to update the classification model; Use the trained classification model to classify the new samples; The calculation of calibration parameters using predicted word probabilities specifically includes the following steps: The preprocessed data samples are input into the pre-trained language model to obtain the logits of the corresponding positions of the masked words. The logits correspond one-to-one with the labels, thus obtaining the logits of the classification labels. Calculate the scaling factor λ to scale the calibrated logits back to the original output logits size of the model: Where z is the logits of each labeled word at the occlusion point; N represents the category of the sample, K represents the number of samples of different categories, and M represents the pre-trained language model; W v Word embeddings representing predicted labels are also used to predict words in hidden states; h mask This represents the hidden state of the last layer of the model at the occlusion point; The `diag` function expands a vector into a diagonal matrix; `i` represents the input sample index, and `j` represents the index of the different labels. This represents the model output logits for the j-th label of the i-th sample, z. i This represents the model output logits vector for the i-th sample; Training a classification model involves the following steps: Construct a black-box optimization model, specifically as follows: Randomly initialize the projection matrix A and the local parameters to be optimized z. l , l∈L, where L is the number of layers in the pre-trained model. Each layer needs to maintain a set of local optimization parameters and construct a local classifier f; The output of the black-box optimization model is: p(y∣∣x prompt )=α·p MLM +(1-α)·p CLS For black-box optimization model training, since the classifier is built locally and its gradients are fully visible, gradient descent and evolutionary algorithms can be used alternately for joint optimization. The optimization process is as follows: a) The training data is input into the model in mini-batch form and the cross-entropy loss is calculated. The backpropagation algorithm updates the parameters of the local classifier f. b) Input all training data into the model and calculate the cross-entropy loss. Use the CMA-ES algorithm to select the optimal z. l Save and use it for the next round of the CMA-ES algorithm; c) Iterate for 3 epochs, b) iterate one round of steps, that is, each layer l of the model is optimized once using the CMA-ES algorithm, which constitutes a complete optimization training step; Iterate the training steps in c) for several rounds.

2. The cue-based few-shot classification method based on a pre-trained language model according to claim 1, characterized in that, The data samples contain N categories, and each sample is labeled with its category. The number of samples in different categories is K, forming a small sample dataset of K-way-N-shot, which contains a total of K*N training samples.

3. The cue-based few-shot classification method based on a pre-trained language model according to claim 1, characterized in that, In the classification model, the output probability of the calibrated cue learning is calculated using calibration parameters as follows: p MLM =Softmax(Wz) We construct a cue as a feature extractor, taking the hidden state of the occlusion point from the output of the pre-trained language model as features, and construct a classifier f for classification: p CLS =Softmax(f(h mask )) And then weighted and merged the two outputs: p(y∣∣x prompt )=α·p MLM +(1-α)·p CLS α is the balance factor, and the above Softmax function is expressed as: x c and x j Output logits for the labels corresponding to indices c and j; since p MLM and p CLS Both outputs used h mask The two methods share the hidden state at the occlusion point, and implicit multi-task learning is used to split the classification task into two sub-classification tasks, so that the classification model has better generalization performance.

4. The cue-based few-shot classification method based on a pre-trained language model according to claim 1, characterized in that, Training a classification model involves the following steps: Each batch of data is input into the constructed classification model to obtain the predicted probability distribution p for each category. i ; The cross-entropy loss function is used to calculate the loss value based on the label and probability distribution corresponding to each sample. The loss value is used for backpropagation and the parameters of the entire classification model are updated using gradient descent. The cross-entropy loss function is expressed as: Where p = [p0, ..., p] C-1 ] represents the predicted probability for all categories, p i Let y = [y0, ..., y] represent the predicted probability of the i-th category. C-1 ] is the one-hot representation of the sample class, where y represents the class when the sample belongs to the i-th class. i =1, otherwise y i =0; C is the number of categories.

5. The cue-based few-shot classification method based on a pre-trained language model according to claim 1, characterized in that, The classifier uses the AdamW optimizer to optimize the parameters, and each complete optimization training is evaluated on the validation set during iterative training. The model with the highest accuracy on the validation set is selected and saved.

6. A cue-based few-shot classification system for learning language based on a pre-trained language model, characterized in that, The cue-based few-shot classification method based on a pre-trained language model, as described in any one of claims 1-5, includes an input data acquisition module, a model calibration and construction module, a model training module, and a model classification prediction module; The input data acquisition module is used to preprocess data samples according to a pre-established prompt template; the prompt template includes an input sample and a masked word, and the preprocessed data sample refers to packaging the data sample with the prompt template so that the packaged data sample contains the tag word; The model calibration and construction module is used to input the preprocessed data samples into the pre-trained language model, obtain the predicted probability of the label words at the occlusion point, and use the predicted probability of the label words to calculate the calibration parameters. The model training module is used to build and train a classification model. The classification model combines traditional fine-tuning methods with cue learning to build a new model that fully learns downstream task knowledge and utilizes the knowledge learned during pre-training. During training, the cross-entropy loss function is used to calculate the loss value, and the loss value is used to update the classification model. The model classification prediction module is used to classify new samples using a trained classification model.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the cue-based few-shot classification method based on a pre-trained language model as described in any one of claims 1-5.

8. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the prompting learning few-shot classification method based on a pre-trained language model as described in any one of claims 1-5.

Images