Student model training method and text classification system based on pre-trained language model

Abstract

A method for training a student model based on a pre-trained language model (PLM) and a text classification system are disclosed. The method includes: constructing cue-based training samples; adjusting the pre-trained language model using the cue-based training samples to obtain a cue-adjusted teacher model; and training the student model using the processed training samples, wherein during training, the student model simultaneously learns the classification probability vectors output by the cue-adjusted teacher model and the original teacher model. This invention requires the student model to learn from two teacher models simultaneously, thereby alleviating the overfitting problem of the student model in small-sample scenarios by adding a distillation path that learns from the original PLM teacher model with unsupervised data. Furthermore, by transferring the intermediate layer representation of the PLM through knowledge probes and stabilizing the performance of knowledge distillation through comparative learning, the student model can learn higher-order dependencies from the intermediate layer representation of the teacher model, improving the accuracy and efficiency of knowledge distillation.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning, and more particularly to a compressed model training method and a text classification system based on a pre-trained language model. Background Technology

[0002] To achieve high-precision prediction results in specific natural language processing tasks, it is usually necessary to train a pre-trained language model (PLM) using a large amount of labeled data. However, a large amount of labeled data can make training too costly. To address this, few-shot learning techniques have been developed that allow pre-trained language models to be trained with a small number of training samples, thereby achieving high prediction accuracy with lower training costs.

[0003] However, to learn knowledge from massive corpora, PLMs have an extremely large parameter size; the existing GPT-3 model has as many as 175 bytes of parameters. This makes PLMs unsuitable for resource-constrained or latency-sensitive scenarios. Therefore, an improved deep learning language model is needed that is applicable to resource-constrained or latency-sensitive scenarios. Summary of the Invention

[0004] One technical problem this disclosure aims to solve is to provide a student model training method and text classification system based on a pre-trained language model. This invention utilizes both a few-shot-adjusted PLM and the original PLM as teacher models for knowledge distillation in the student model to be trained, thereby avoiding overfitting of the student model caused by sparse labeled data. Furthermore, this invention can also use knowledge probes to derive intermediate layer information from the teacher model as a source of supervision for the intermediate layers of the student model, and utilize contrastive learning as a training aid, thereby obtaining a small-scale student model that has fully learned the knowledge from the PLM based on few-shot training.

[0005] According to a first aspect of this disclosure, a method for training a student model based on a pre-trained language model is provided, comprising: adding prompt information and masked text placeholders to samples to obtain processed training samples; adjusting the pre-trained language model using the processed training samples to obtain a prompt-adjusted teacher model, wherein the un-prompted pre-trained language model is the original teacher model; and training a student model using the processed training samples, wherein during the training process, the student model learns the classification probability vectors output by the prompt-adjusted teacher model and the original teacher model.

[0006] Optionally, training a student model using the processed training samples includes: obtaining the corresponding prediction result of the student model for the masked text placeholder; adjusting the network parameters of the student model using a first loss function, wherein the first loss function calculates the loss based on whether the corresponding prediction result of the masked text placeholder is the same as the label.

[0007] Optionally, during training, the student model learns the classification probability vectors output by the prompted and adjusted teacher model and the original teacher model, including: adjusting the network parameters of the student model with a second loss function, the second loss function representing the similarity between the classification probability vector output by the student model and the classification probability vector output by the prompted and adjusted teacher model; and adjusting the network parameters of the student model with a third loss function, the third loss function representing the similarity between the classification probability vector output by the student model and the classification probability vector output by the original teacher model.

[0008] Optionally, the method further includes: adding prompt information and mask text placeholders to the second sample to obtain a processed second training sample, wherein the second training sample is an unlabeled sample, and the third loss function characterizes the difference between the classification probability vector output by the student model for the second training sample and the classification probability vector output by the original teacher model for the second training sample.

[0009] Optionally, the student model learning the classification probability vectors output by the prompted and adjusted teacher model and the original teacher model during training includes: the student model learning the classification probability vectors output by the intermediate layers of the prompted and adjusted teacher model and the original teacher model during training.

[0010] Optionally, during training, the student model learns the classification probability vectors output by the intermediate layers of the prompted and adjusted teacher model and the original teacher model, including: adjusting the network parameters of the student model with a fourth loss function, the fourth loss function representing the difference in similarity between the classification probability vectors output by the intermediate layers of the student model and the classification probability vectors output by the intermediate layers of the prompted and adjusted teacher model when the inputs have different true labels; and adjusting the network parameters of the student model with a fifth loss function, the third loss function representing the difference in similarity between the classification probability vectors output by the intermediate layers of the student model and the classification probability vectors output by the intermediate layers of the original teacher model when the inputs have different true labels.

[0011] Optionally, the classification probability vector output by each intermediate layer of the student model is multiplied by the classification probability vector output by each intermediate layer of the teacher model, and the average of the products is calculated to characterize the similarity between the intermediate layer outputs of the student model and the teacher model.

[0012] Optionally, the student model is trained using the processed training samples, and the student model learns the classification probability vectors output by the cue-adjusted teacher model and the original teacher model during the training process, including: training the student model using a weighted sum of a first loss function and loss functions representing the similarity between the classification probability vectors output by the cue-adjusted teacher model and the original teacher model and the classification probability vector output by the student model, respectively, as the total loss function.

[0013] According to a second aspect of this disclosure, a text classification system is provided, comprising: an input acquisition unit for acquiring input from a user; an intent determination unit including a student model acquired by the method described in the first aspect, the student model being used to classify the user's intent based on the input; and an operation unit for performing subsequent operations based on the classified user intent.

[0014] According to a third aspect of this disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method described in the first aspect above.

[0015] According to a fourth aspect of this disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0016] Therefore, a few-shot knowledge distillation scheme based on prompt-tuned PLM is proposed. This scheme requires the student model to learn simultaneously from both the prompt-tuned PLM and the original PLM teacher model. This alleviates the overfitting problem of the student model in few-shot scenarios by adding a distillation path that learns from the original PLM teacher model with unsupervised data. Furthermore, the intermediate layer representations of the PLM are transferred through knowledge probes, and the performance of knowledge distillation is stabilized through contrastive learning. This enables the student model to learn higher-order dependencies from the intermediate layer representations of the teacher model, improving the accuracy and efficiency of the student model's knowledge learning. Attached Figure Description

[0017] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, in which the same reference numerals generally represent the same components.

[0018] Figure 1 Examples of prompts and label selection for sentiment analysis of comments are shown.

[0019] Figure 2A schematic flowchart of a student model training method based on a pre-trained language model according to an embodiment of the present invention is shown.

[0020] Figure 3 Examples of soft and hard targets and temperature factor-modulated soft targets are shown.

[0021] Figure 4 The diagram shows the overall schematic of the present invention, which trains a student model based on two teacher models.

[0022] Figure 5 A schematic diagram of the composition of a text classification system according to an embodiment of the present invention is shown.

[0023] Figure 6 A schematic diagram of a computing device is shown, which can be used to implement the above-described student model training method based on a pre-trained language model according to an embodiment of the present invention. Detailed Implementation

[0024] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0025] Large-scale pre-trained language models have achieved tremendous success in various fields of NLP (Natural Language Processing). Instead of training language models from scratch, a general-purpose PLM is first obtained on a large general corpus through unsupervised agent tasks. Then, in downstream tasks, the general-purpose PLM is fine-tuned on supervised data, leveraging existing language knowledge in the general corpus to achieve target classification. This two-stage model paradigm has been widely adopted in many practical language application scenarios.

[0026] Few-shot learning is a paradigm in machine learning that aims to achieve high accuracy by fine-tuning a model with only a small number of training samples. The ability to learn and generalize from a limited number of samples is a key differentiator between artificial intelligence and human intelligence. Humans can easily build understanding of new concepts using just one or a few examples, while machine learning algorithms typically require thousands of labeled samples to ensure generalization. In fields like machine vision and natural language processing, data labeling is expensive; conversely, labeled data is scarce in new scenarios. This limits the application of deep learning algorithms. Few-shot learning is significant and challenging in machine learning. Inspired by humans' rapid learning ability, the goal is for machine learning models, after learning from a large amount of data in a certain category, to quickly learn new categories with only a small number of samples. This is the problem that few-shot learning aims to solve.

[0027] Given the unique nature of small sample tasks, the downstream task of stage two can be fine-tuned and reconstructed into a "cloze test problem," that is, using PET (Pattern Exploring Training).

[0028] Since BERT, prompt-based fine-tuning (MLM) has become a common practice in NLP for downstream tasks. The GPT-3 model, with its 175-byte parameter set, offers a novel approach to using language models (LM) for downstream tasks: by using natural language prompts and demonstrations as context, GPT-3 can handle many tasks with just a few samples without updating the parameters in the underlying model. The sheer size of GPT-3 is a key factor in its success, and the concepts of prompts and demonstrations provide new insights into how to better utilize language models. A prompt is a piece of text inserted into the input sample, thus transforming the original prediction task into an MLM problem. For example, suppose we want to perform sentiment classification on the movie review "No reason to watch," we can append the prompt "It was," resulting in "Noreason to watch. It was [MASK]." The "[MASK]" character maps the predicted output of the pre-trained model MLMhead to the actual class label. For the example above, if the probability of predicting "great" is higher, it corresponds to the "positive" category; if the probability of predicting "terrible" is higher, it corresponds to the "negative" category. With PLM possessing a vast amount of language knowledge, PLM will have a higher probability of determining that the character "[MASK]" corresponds to "terrible" rather than "great".

[0029] Figure 1 Examples of prompts and label selection for sentiment analysis of comments are shown. For example... Figure 1As shown, to determine the sentiment category (e.g., positive praise or negative criticism) of the sentence "Wonderfulmovieineveryaspect," the input text can be directly cueed with "It is [MASK].", and the model can predict either "good" (positive label) or "terrible" (negative label). In other words, the cue template can be constructed as "It is + sentiment attribute word," and a Verbalizer can be used to select two words from the vocabulary corresponding to positive and negative sentiment as labels, in this case, "good" and "terrible." Thus, the original training sample "Wonderfulmovieineveryaspect." can be transformed into a processed training sample with a masked cue: "Wonderful movie in every aspect. It is [MASK]." This training sample is then fed into the PLM for training, for example, to calculate the loss function and adjust it based on whether the PLM model predicts "[MASK]" as "good" or "terrible."

[0030] exist Figure 1 In the example shown, positive and negative labels can be manually selected. For example, one could choose "good" and "terrible" as shown in the illustration, or other words from the vocabulary (e.g., the general vocabulary) used to express emotional attributes, such as "great" and "bad". Additionally, in Figure 1 In the example, the prompt "It is" can also be designed by humans.

[0031] Additionally, although examples of English text and prompts are shown in the figure, prompts, masks, and labels can also be used to construct samples for Chinese and perform subsequent classification.

[0032] While existing PLMs can be fine-tuned based on cues learned from few samples to quickly acquire the ability to classify target tasks, such as classifying comment sentiment as positive or negative, the resulting fine-tuned PLM has a very large parameter scale, making it unsuitable for resource-constrained or latency-sensitive scenarios.

[0033] In machine learning, knowledge distillation (KD) is a process that transfers knowledge from a large model to a smaller model. While large models (such as very deep neural networks or collections of many models) have a greater knowledge capacity than small models, this capacity may not be fully utilized. Even if a model utilizes only a small portion of its knowledge capacity, the computational cost of evaluating it is the same. On the other hand, small models are more difficult to train than large models. Knowledge distillation transfers knowledge from a large model to a smaller model without sacrificing its effectiveness. Because small models have lower evaluation costs, they can be deployed on less powerful hardware (such as mobile devices).

[0034] However, existing knowledge distillation techniques are difficult to apply to few-shot learning scenarios because scarce labeled data can cause student models to overfit, and existing knowledge distillation methods cannot train the target model based on prompts.

[0035] To address this issue, this invention proposes a few-shot knowledge distillation scheme based on prompt-tuned PLM. This scheme requires the student model to learn simultaneously from both the prompt-tuned PLM and the original PLM teacher model. This alleviates the overfitting problem of the student model in few-shot scenarios by adding a distillation path that learns from the original PLM teacher model with unsupervised data. Furthermore, by transferring the intermediate layer representations of the PLM through knowledge probes and stabilizing the performance of knowledge distillation through contrastive learning, the student model can learn higher-order dependencies from the intermediate layer representations of the teacher model, improving the accuracy and efficiency of knowledge learning in the student model.

[0036] In one embodiment, the present invention can be implemented as a student model training method based on a pre-trained language model. Figure 2 A schematic flowchart of a student model training method based on a pre-trained language model according to an embodiment of the present invention is shown.

[0037] In step S210, prompt information and masked text placeholders are added to the samples to obtain processed training samples.

[0038] In the student model training of this invention, a small-sample training dataset is required, for example, given an N-way-K-shot training dataset X. Here, N represents that the model can output N categories, and K represents the number of samples in each category. Therefore, the N-way-K-shot training dataset X contains N×K samples, and in the case of small-sample training, the value of N×K will be very small.

[0039] The sample could be, for example, N×K sentences with sentiment preferences, and could be as follows: Figure 1The input sample is constructed by adding the corresponding prompt message "itis" and the mask text placeholder "[MASK]", and a corresponding label (i.e., the real label) is generated for each sample based on the actual sentiment preference contained in the sentence.

[0040] In step S220, the pre-trained language model is adjusted using the processed training samples to obtain a prompted-adjusted teacher model, wherein the un-prompted pre-trained language model is the original teacher model. Then, in step S230, the student model is trained using the processed training samples, and during the training process, the student model learns the classification probability vectors output by the prompted-adjusted teacher model and the original teacher model.

[0041] In this invention, in addition to constructing an N-way-K-shot training dataset X, a large-scale PLM (teacher model) and a smaller PLM (student model) are also required. Compared to the teacher model, the student model can have a similar but fewer substructures. For example, the teacher model can have N... T+1 There are N Transformer structures, and the student model can have N... S+1 There are N Transformer structures, where N S+1 <N T+1 And preferably, N S+1 <<N T+1 (The Transformer itself is a deep learning model that uses self-attention to improve the training speed of the model. Existing pre-trained language models include multiple Transformer structures.) The goal of training is to compress the performance of the teacher model, which is fine-tuned with hints on small sample data, into the student model through knowledge distillation.

[0042] The given content and training objective can be described using mathematical notation. Specifically, the training dataset X = {(x i ,y i )}(Here, y i The input text x i Category tags, among which It is a tag set. and During parameter tuning, there is also a small validation set of the same size as X (used for parameter tuning). Using Θ T The parameters represent a PLM that has been prompted for adjustment (also known as prompted for fine-tuning). Model Θ T It is from its pre-trained initialization Θ T’ The adjustment was obtained through the hint. In other words, Θ can be used here. T’To represent the parameters of the original teacher model. The goal of this invention is to obtain a parameter derived from Θ. S The parameters represent a much smaller PLM, while making Θ S The performance can be as close as possible to Θ. T .

[0043] To achieve this goal, after constructing the small sample training dataset X in step S210, it is necessary to first process the large-scale original PLMΘ in step S220. T’ Received PLMΘ after prompting adjustment T Here, the Masked Language Model (MLM) task can be used to obtain the prompted and adjusted PLMΘ. T Specifically, the training samples fed into the original PLM can have a form such as "Wonderful movie in every aspect. It is [MASK].", and the model needs to output the label classification corresponding to [MASK]. For example, when the classification N=2 (in this case, the word corresponding to [MASK] includes only one positive word, such as "good", and one negative word, such as "terrible"), the model will output the probability of whether the word corresponding to [MASK] is "good" or "terrible". If the probability of the model determining it as "good" is greater than the probability of it as "terrible", then the classification result is good.

[0044] In MLM, for each word in the vocabulary, the prediction target vector is a one-hot vector. A one-hot vector, also known as a "one-hot vector," means that in a set of predictable words, only the coefficient corresponding to the classification label ("good" in this example) is 1; the coefficients for predicting other words are all 0. Therefore, when using one-hot vectors to construct the loss function, no loss will occur only if the model predicts the label itself, for example, if the model outputs "good." However, if the model outputs any other word besides "good," the same loss will occur.

[0045] The calculated loss can be used to adjust the parameters Θ of the original PLM based on the backpropagation algorithm. T’ After adjustments were made and the N-way-K-shot training dataset X was input, the adjusted PLMΘ was obtained. T .

[0046] The student model can be trained using a small-sample training dataset X in a similar manner to training the original teacher model using a small-sample training dataset X. To this end, training the student model using the processed training samples includes: obtaining the student model's corresponding prediction results for the masked text placeholders; and adjusting the network parameters of the student model using a first loss function, whereby the loss is calculated based on whether the corresponding prediction results of the masked text placeholders are the same as the masked words. In other words, the first loss function can also be constructed using one-hot vectors in a masked language modeling (MLM) task.

[0047] In one embodiment, the PET method can be followed, where l(y) is the label word for category y. Is it using input x i and PLMΘ T In the case of Θ, predict the score of l(y) at the masked language marker. T x i The probability of being assigned to category y is defined as follows:

[0048]

[0049] Here, further Represented as all N categories The probability vector. It is x i The corresponding N-dimensional one-hot true vector. The classification loss of the student model (corresponding to the first loss function) can be directly derived as follows:

[0050]

[0051] Where CE(·,·) represents the cross-entropy loss between two vectors.

[0052] In this invention, the original PLMΘ is used. T’ And PLMΘ adjusted as prompted T Both serve as teacher models for knowledge distillation. Specifically, during the training of the student model using a small sample training dataset X, the student model learns the classification probability vectors output by the prompted and adjusted teacher model and the original teacher model. In other words, knowledge distillation can be achieved by the student model learning the classification probability vectors output by the teacher model.

[0053] The classification model is ultimately set with a softmax layer, whose output values ​​correspond to the probability values ​​of the respective classes. During knowledge distillation, since there is already a teacher model with strong generalization ability, the student model can directly learn the generalization ability of the teacher model. A straightforward and efficient way to transfer generalization ability is to use the class probabilities (i.e., the classification probability vectors) output by the softmax layer as the "soft target".

[0054] Conventional neural network training methods define a loss function with the goal of making the predicted values ​​as close as possible to the true values ​​(corresponding to the hard-target, also known as the "hard goal"). The loss function is to minimize the sum of the neural network's losses. This training process involves maximizing the likelihood of the ground truth. In knowledge distillation, however, a training process is involved using the class probabilities of the teacher model as soft labels to train the student model.

[0055] Figure 3 Examples of soft and hard targets and temperature-modulated soft targets are shown. Assume... Figure 3 This corresponds to the output of a 10-class classification model. Figure 3 The left side corresponds to hard targets, including one-shot labels from the original dataset. Except for the positive label of class 2, which is 1, the negative labels of the other 9 classes are all 0. Figure 3 The middle corresponds to soft targets, such as the class probabilities output by the softmax layer of the teacher model. Each class is assigned a probability. The positive label probability corresponding to the second class is the highest (close to 0.6), but the negative labels of the other nine classes also have a certain probability. For example, the probability of the third class is close to 0.2, although these probabilities are all lower than the probability of the positive label.

[0056] Because the output of the softmax layer contains a wealth of information about the teacher model's inductive reasoning, in addition to positive examples, the negative labels also carry a significant amount of information, such as the probability of some negative labels being much higher than that of others (e.g., Figure 3 The third category shown in the middle represents the sample that the teacher model considers to have a certain similarity to the negative label during reasoning. Therefore, the knowledge distillation training method provides the student model with more information per sample than traditional training methods. In other words, when using soft-target training, the student model can quickly learn the reasoning process of the teacher model.

[0057] Therefore, during training, the student model learns the classification probability vectors output by the prompted and adjusted teacher model and the original teacher model, including: adjusting the network parameters of the student model with a second loss function, the second loss function representing the difference between the classification probability vector output by the student model and the classification probability vector output by the prompted and adjusted teacher model; and adjusting the network parameters of the student model with a third loss function, the third loss function representing the difference between the classification probability vector output by the student model and the classification probability vector output by the original teacher model.

[0058] Here, the classification probability vectors involved in the second and third loss functions can be the classification probability vectors output by the softmax layer of each model, that is, the classification probability vectors output by the last layer of the model.

[0059] Furthermore, the "soft target" provided by the teacher model, after being prompted for adjustment, is for the small sample training dataset X, as stated above. Specifically, the same sample x from the small sample training dataset X can be used as the target. i The teacher model and student model, which have been adjusted according to the prompts, are fed into the system respectively. The teacher model and student model, which have been adjusted according to the prompts, calculate their respective classification probability vectors. A second loss function can be constructed based on the cross-entropy between the two classification probability vectors, and the adjustment direction of the second loss function is to reduce the cross-entropy between the two vectors.

[0060] In one embodiment, the labeled knowledge distillation loss (corresponding to the second loss function) can be defined as follows:

[0061]

[0062] Here, α>0 is the temperature factor. As mentioned earlier, the class probabilities output by the teacher model's softmax layer can be used as the soft-target to help the student model quickly learn the teacher model's reasoning process. However, because the softmax function normalizes the probabilities of Logits across classes and amplifies the differences between Logits values, when the probability distribution entropy output by the softmax is relatively small, the values ​​of the negative labels are very close to 0, contributing very little to the loss function. At this point, a temperature factor α is needed to amplify the information carried by the negative labels. In the entire knowledge distillation process, the temperature factor can be increased first, and then restored to a "low temperature" during the testing phase; this is the origin of the term "distillation."

[0063] Back Figure 3 ,in, Figure 3The middle section shows the class probabilities output by the softmax layer of the teacher model, which is equivalent to a temperature factor α = 1. During the distillation process, the temperature factor can be increased, thereby increasing the probabilities corresponding to other negative labels. Figure 3 The right side shows the soft targets as the temperature factor α increases (greater than 1). Clearly, the probability of a positive label remains the highest at this point, but the probability of a negative label increases.

[0064] Because this invention uses only labeled small sample datasets, it faces the challenge of a lack of training data and a rather limited amount of supervision signals. Therefore, it is advisable to consider learning directly from a pre-trained teacher model that has not undergone any fine-tuning, and to use unlabeled data to alleviate the overfitting problem that student models often cause with small sample training. When using unlabeled data, the predicted similarity between the student PLM and the teacher PLM can be used as a supervision signal, allowing the student PLM to learn the linguistic knowledge contained in the teacher PLM.

[0065] The second loss function above corresponds to labeled knowledge distillation loss, meaning that both the student model and the prompted, fine-tuned teacher model are trained on small sample data. Specifically, the model input for the second loss function is the N-way-K-shot training dataset X as described above. The third loss function, however, corresponds to unlabeled knowledge distillation loss, meaning that both the student model and the prompted, fine-tuned teacher model are trained on larger, unlabeled sample data (e.g., the unlabeled dataset described below). in

[0066] Therefore, the training method of the present invention may include adding prompt information and masked text placeholders to the second sample to obtain a processed second training sample, wherein the second training sample is an unlabeled sample. That is, since the size of X is very small (i.e., N×K), it is assumed here that there exists a larger unlabeled dataset. in This serves as an auxiliary dataset for knowledge distillation (i.e., corresponding to the second training samples). Therefore, the third loss function characterizes the difference between the classification probability vector output by the student model for the second training samples and the classification probability vector output by the original teacher model for the second training samples.

[0067] Specifically, the parameters of the original teacher model are Θ, as described above. T’ Therefore, in one embodiment, it is possible to further define based on Θ T’ and The unlabeled knowledge distillation loss (corresponding to the third loss function) is as follows:

[0068]

[0069] In scenarios with few samples, it is necessary to extract as much information as possible from the model. Besides the MLM head, intermediate layer representations can also provide useful clues for knowledge distillation. Therefore, during training, the student model learning the classification probability vectors output by the cue-adjusted teacher model and the original teacher model can include: during training, the student model learning the classification probability vectors output by the intermediate layers of the cue-adjusted teacher model and the original teacher model.

[0070] Because of the capacity gap between the models, directly narrowing the differences in intermediate layer representations between teachers and students will not yield better results. To alleviate the model capacity gap problem, this invention transfers intermediate layer knowledge by calculating the correlation between intermediate layer representations. Furthermore, contrastive learning can be used to make the correlations under different labels less pronounced. Contrastive learning is a type of unsupervised learning that focuses on learning common features between similar instances and distinguishing differences between dissimilar instances. The goal of contrastive learning is to learn an encoder that encodes similar data for similar classes and makes the encoding results for different classes of data as different as possible. In this invention, contrastive learning can be achieved by selecting samples with different real labels.

[0071] At this point, during the training process, the student model learns the classification probability vectors output by the intermediate layers of the prompted and adjusted teacher model and the original teacher model, including: adjusting the network parameters of the student model with a fourth loss function, the fourth loss function representing the difference in the correlation between the classification probability vectors output by the intermediate layers of the student model and the classification probability vectors output by the intermediate layers of the prompted and adjusted teacher model when the inputs have different true labels; and adjusting the network parameters of the student model with a fifth loss function, the third loss function representing the difference in the correlation between the classification probability vectors output by the intermediate layers of the student model and the classification probability vectors output by the intermediate layers of the original teacher model when the inputs have different true labels.

[0072] In one embodiment, the classification probability vector output by each intermediate layer of the student model is multiplied by the classification probability vector output by each intermediate layer of the teacher model, and the average of the products is calculated to characterize the correlation between the intermediate layer outputs of the student model and the teacher model.

[0073] This invention uses knowledge probes to transfer knowledge in the intermediate layer. Knowledge probes, a series of pseudo-MLM heads, connect the information in the intermediate layer with the actual labels, thereby assisting in the transfer of intermediate layer information. After prompt-based fine-tuning and distillation of the teacher and student models, their parameters can be frozen and the knowledge probes trained with real labels. This ultimately yields information containing the intermediate layer information, which serves as the intermediate layer supervision source for the student model.

[0074] Specifically, in freezing Θ T and Θ S After that, it is Θ T and Θ S Each transformer's encoding layer (except the last layer) trains an MLM-based probe classifier on the ground-labeled words. In total, there are N... T One probe is used for the teacher model, N S One probe is used for the student model (at this time, the teacher model has N probes). T+1 There are N Transformer structures, and the student model has N... S+1 A Transformer structure can be used to train a probe classifier for the encoding layer of each transformer except the last one. Indicated based on Θ T and the j-th probe (j = 1, ..., N) T x i Assigned to N classes The probability vector. Similarly, the outcome probability from the student model is represented as... Where k = 1, ..., N S The exponential matching score between two models can be defined as:

[0075]

[0076] To stabilize performance on small-scale data, contrastive learning can be used as an auxiliary training objective. For a given instance, a set of instances with different labels can be randomly selected as negative examples, and the results of all knowledge probes can be regarded as different data augmentations for these instances. This yields a batch of negative examples. Therefore, this invention proposes labeled contrastive prompt distillation (CPD) loss (corresponding to the fourth loss function) to transfer intermediate knowledge across models:

[0077]

[0078] Similarly, for the original teacher model Θ T’ and unlabeled datasets It is possible to suggest distillation loss in unlabeled comparisons (corresponding to the fifth loss function):

[0079]

[0080] Here, the difference from the fourth loss function lies in the negative samples. Extraction cannot be directly based on real labels (because real labels are unavailable). As a simple heuristic, the same hints and tag words can be used directly in Θ. T’ Reasoning The labels (which can be viewed as zero-shot learning) are used.

[0081] Therefore, training the student model using the processed training samples, and the student model learning the classification probability vectors output by the cue-adjusted teacher model and the original teacher model during the training process, may include: using a weighted sum of a first loss function and loss functions representing the similarity between the classification probability vectors output by the cue-adjusted teacher model and the original teacher model and the classification probability vector output by the student model, respectively, as the total loss function, to train the student model.

[0082] In a preferred embodiment of the present invention, the loss function characterizing the similarity of the classification probability vectors output by the student model and the teacher model may include second and third functions directly based on the similarity of the final classification probability vectors output by the softmax, and may also include fourth and fifth functions describing the similarity of the classification probability vectors output by the intermediate layer. Combining the above knowledge distillation objectives with weighted summation, the following optimal loss function is obtained:

[0083]

[0084] Where λ1 and λ2 are equilibrium hyperparameters, thus obtaining the distilled student model.

[0085] Figure 4 This diagram illustrates the overall structure of the invention, which trains a student model based on two teacher models. Figure 4 As shown, the left side illustrates the overall framework of the cue-distiller of this invention, and the right side shows an example of cue-enhancing data (i.e., enhancements can be made using unlabeled datasets).

[0086] As shown on the left side of the figure, the student model has a similar structure to the teacher model, but with fewer Transformers and fewer Transformer encoder layers (shown as the Trm Layer). The two teacher models have identical network structures; the only difference is that the teacher model, after being cue-tuned based on a small sample of the N-way-K-shot training dataset X, has different parameters Θ. T Compared to the original teacher model, Θ T’ There have been some minor adjustments.

[0087] When training the student model, it is necessary to construct a task-specific MLM loss, namely the first loss function L as described above. MLM (X), this is based on Figure 4The data in the upper right corner was used for training and inference.

[0088] Knowledge distillation of the student model by the two teacher models can begin with knowledge distillation based on, for example, the similarity of the classification probability vectors output by the Softmax layer. The similarity between the adjusted teacher and student models, as described above, can be calculated using the classification vector similarity. Figure 4 The data in the upper right corner is used for training and inference, which corresponds to the second loss function L as described above. KD (X). Regarding the similarity between the original teacher model and the student model, the calculation of the classification vector similarity above can be targeted at... Figure 4 The data obtained from training and inference using unlabeled data in the lower right corner corresponds to the third loss function described above.

[0089] Furthermore, knowledge can be extracted from the intermediate layers. In this case, a knowledge probe can be trained for each intermediate layer (TrmLayer) of the three models. Knowledge distillation of the intermediate layers is achieved based on the overall similarity of the classification probability vectors of each layer and by utilizing the difference in values ​​under positive and negative labels. Similarly, the similarity between the adjusted teacher model and the student model is calculated based on the labeled training dataset X, resulting in the fourth loss function L. CPD (X). The similarity between the original teacher model and the student model is still obtained based on inference trained on unlabeled data, which corresponds to the fifth loss function as described above.

[0090] As described above Figure 2 and Figure 4 This invention describes a student model training method based on a pre-trained language model. After obtaining the student model using the above method, because the student model has fewer parameters and learns the knowledge inherent in the large-scale pre-trained language model through multi-pipeline knowledge distillation, it is suitable for deployment in practical application scenarios.

[0091] Therefore, the present invention can also be implemented as a text classification system. Figure 5 A schematic diagram of the composition of a text classification system according to an embodiment of the present invention is shown.

[0092] As shown in the figure, the system 500 may include an input acquisition unit 510, a classification determination unit 520, and an operation unit 530.

[0093] The input acquisition unit 510 is used to acquire text input from the user. The text input acquired here can be text entered by the user himself, such as a movie review posted by the user, or text converted from user input, such as the recognition result of user voice input.

[0094] The classification determination unit 520 may include a student model obtained by the method described above, which is used to classify based on the text input. The operation unit 530 can then be used to perform operations based on the classification result.

[0095] This text classification system can be applied in various scenarios. For example, in intelligent robot interaction scenarios, it can obtain user input from text boxes, determine the user's intent in real time, and enable the operating unit to provide appropriate text feedback or other operations based on the identified intent. Another example is the ability to read and classify massive amounts of comments on a particular artwork, providing a general sentiment index of users' opinions on the work, which can then be used as a basis for recommendations to other users. Furthermore, it can classify text to determine whether it contains promotional material or inappropriate information, and then delete or report such content in subsequent operations.

[0096] Therefore, the operation unit 530 may perform at least one of the following operations based on the classification results: providing feedback based on the intent classification results of the input text; performing statistics based on the sentiment classification results of the input text; and reporting based on the attribute classification of the input text.

[0097] Figure 6 A schematic diagram of a computing device is shown, which can be used to implement the above-described student model training method based on a pre-trained language model according to an embodiment of the present invention.

[0098] See Figure 6 The computing device 600 includes a memory 610 and a processor 620.

[0099] Processor 620 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 620 may include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 620 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0100] Memory 610 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 620 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 610 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 610 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0101] The memory 610 stores executable code, which, when processed by the processor 620, enables the processor 620 to execute the student model training method based on the pre-trained language model described above.

[0102] The student model training and text classification system based on a pre-trained language model according to the present invention has been described in detail above with reference to the accompanying drawings.

[0103] This invention improves the few-shot learning performance of large-scale PLMs using prompt-based learning. To enable online deployment of PLMs in resource-constrained environments, this invention employs knowledge distillation to compress large-scale PLMs. Specifically, this invention proposes the Prompt Distiller, the first implementation of few-shot knowledge distillation for prompt-fine-tuning PLMs, enabling the student model to learn simultaneously from both a pre-trained and prompt-fine-tuned teacher model. Considering the different knowledge carrying capacities of the teacher and student models, this invention further designs a contrastive learning technique to learn higher-order dependencies from the intermediate layer representations of the teacher model.

[0104] To address the problems existing in related technologies, this invention mainly proposes the following solutions and improvements:

[0105] 1. The loss function for knowledge distillation has been improved to accommodate the prompt-based fine-tuning model;

[0106] 2. A distillation pipeline for learning from teacher PLM using unsupervised data has been added to alleviate overfitting caused by lack of labels in small sample scenarios;

[0107] 3. Transfer the intermediate layer representation of PLM through knowledge probes, and treat different probe results as different data augmentations, thereby stabilizing the performance of distillation through comparative learning.

[0108] Furthermore, the method according to the present invention can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of the present invention.

[0109] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the various steps of the method described above according to the present invention.

[0110] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0112] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a student model based on a pre-trained language model, comprising: Add prompts and masked text placeholders to the samples to obtain processed training samples; Add prompts and masked text placeholders to the second sample to obtain a processed second training sample, which is an unlabeled sample. The pre-trained language model is adjusted using the processed training samples to obtain a prompted-adjusted teacher model, wherein the pre-trained language model without prompt adjustment is the original teacher model; and Obtain the prediction result of the student model for the masked text placeholder; The network parameters of the student model are adjusted using a first loss function, which calculates the loss based on whether the prediction result of the mask text placeholder is the same as the label. The network parameters of the student model are also adjusted using a second loss function, which characterizes the similarity between the classification probability vector output by the student model and the classification probability vector output by the teacher model after prompting and adjustment. as well as The network parameters of the student model are adjusted using a third loss function, which represents the difference between the classification probability vector output by the student model for the second training sample and the classification probability vector output by the original teacher model for the second training sample. The network parameters of the student model are adjusted using a fourth loss function, which characterizes the difference in similarity between the classification probability vector output by the intermediate layer of the student model and the classification probability vector output by the intermediate layer of the teacher model (adjusted with prompting) when the inputs have different true labels; and The network parameters of the student model are adjusted using a fifth loss function, which represents the difference in similarity between the classification probability vector output by the intermediate layer of the student model and the classification probability vector output by the intermediate layer of the original teacher model when the inputs have different true labels.

2. The method as described in claim 1, wherein, The similarity between the intermediate layer outputs of the student model and the teacher model is characterized by multiplying the classification probability vector output of each intermediate layer of the student model with the classification probability vector output of each intermediate layer of the teacher model, and taking the average of the products.

3. The method as described in claim 1, wherein, The student model is trained using the processed training samples, and during training, the student model simultaneously learns the classification probability vectors output by the prompted and adjusted teacher model and the original teacher model, including: The student model is trained using a weighted sum of a first loss function and a loss function representing the similarity between the classification probability vectors output by the cue-adjusted teacher model and the original teacher model and the classification probability vectors output by the student model, respectively.

4. A text classification system, comprising: The input acquisition unit is used to acquire input text from the user; The classification determination unit includes a student model obtained by the method as described in any one of claims 1-3, the student model being used to classify based on the input text; and An operation unit is configured to perform operations based on the classification results, wherein the operations include at least one of the following: Feedback is provided based on the intent classification result of the input text; Statistical analysis was performed based on the sentiment classification results of the input text; and The report is generated based on the attribute classification of the input text.

5. A computing device, comprising: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 3.

6. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as claimed in any one of claims 1 to 3.

Images