A knowledge selection method applied to knowledge distillation

Abstract

The application discloses a knowledge selection method applied to knowledge distillation, and comprises the following steps: 1) a teacher model is trained by using data of a text classification task; 2) different types of knowledge are extracted by using the trained teacher model, and the knowledge is classified; 3) a student model is initialized based on the trained teacher model; 4) a knowledge selection module is trained by using a REINFORCE algorithm; and 5) the final student model is trained by using the trained knowledge selection module, and the knowledge selection module selects the type of knowledge that the student model should learn according to the characteristics of the corresponding teacher model and student model during the training. The method can learn the type of knowledge suitable for the student model to learn in the knowledge distillation process, so that the type of knowledge suitable for the student model to learn can be selected according to the state of the teacher model and the student model in each training step in the knowledge distillation process.

Description

TECHNICAL FIELD

[0001] The present application relates to a knowledge distillation technology, in particular to a knowledge selection method applied to knowledge distillation. BACKGROUND

[0002] Pre-trained language models (PLMs) are usually stacked by bidirectional Transformers, which can achieve excellent performance by training on large-scale monolingual corpus and fine-tuning on specific task data. Although PLMs can achieve excellent performance on many natural language processing tasks, they suffer from difficulties in deployment, high computational cost, and long inference time due to their large size. Knowledge distillation is a method for compressing PLMs and is the most commonly used method. It represents a smaller PLM (student model) that is trained and updated under the guidance of a larger and better-performing PLM (teacher model).

[0003] PLMs are stacked by multiple bidirectional Transformers, and the output of each Transformer is the input of the next Transformer. Typically, the output of each Transformer is also referred to as the intermediate layer feature. When performing text classification, the input text sequence is converted into a vector containing text information using an encoder, and a classifier is added at the last layer to calculate the input text category probability distribution.

[0004] Early text classification frameworks use convolutional neural networks to encode input text sequences, and then use recurrent neural networks (RNNs) to convert intermediate vectors into final feature vectors corresponding to the text, which are then input into a classifier for classification. The successful application of PLMs with attention mechanisms in text classification has brought text classification to a new level. After the PLM reads the text, the attention mechanism simulates the process of human reading the text. When inputting the text feature vector, the model does not treat all words as identical as in the recurrent neural network, but assigns different weights to different words. Words with high relevance to the current time will have higher attention weights. It can be seen that the introduction of the attention mechanism alleviates the shortcomings of the recurrent neural network.

[0005] In actual processes, whether it is text classification or other deep neural network-based tasks, human behavior is constantly simulated, and innovation and breakthroughs are constantly made. In the process of knowledge distillation, it is assumed that if a model first learns the knowledge of the corpus, and then uses the knowledge learned by the model, it can help other models learn better.

[0006] Although there are some knowledge selection methods for knowledge distillation now, such as dynamically adjusting the weight of the loss, there are still the following shortcomings:

[0007] 1) The knowledge type is not directly modeled, such as dynamically adjusting the loss weight. When adjusting the loss weight, the knowledge and the loss are not directly related, and the problem of knowledge selection in knowledge distillation is not directly solved;

[0008] 2) The loss weight adjustment modeling method is simple, and the existing weight adjustment method is to directly model the confidence of the model and some related factors, but the selection of knowledge in the world is affected by multiple factors, and it is not determined by some factors alone;

[0009] 3) Few knowledge types are considered, and the existing method does not systematically classify and divide all knowledge, but solves the knowledge selection problem for a certain knowledge. For example, the existing method only studies whether the reply knowledge should be learned by the student model in the training process, and ignores other types of knowledge. SUMMARY

[0010] In view of the problem of multiple knowledge types in the training of the existing knowledge distillation, which may cause the student model to not learn the correct knowledge, the technical problem to be solved by the present application is to provide a knowledge selection method applied to knowledge distillation, which can determine what type of knowledge the student model should learn according to the learning state of the teacher model and the student model to obtain better performance, and more fully and correctly use the knowledge of the teacher model in the training process of knowledge distillation.

[0011] The present application provides a knowledge selection method applied to knowledge distillation, comprising the following steps:

[0012] 1) training a teacher model using text classification task data;

[0013] 2) using the trained teacher model to extract different types of knowledge, and classifying the knowledge;

[0014] 3) initializing a student model based on the trained teacher model;

[0015] 4) Train the knowledge selection module using REINFORCE algorithm;

[0016] 5) Train the final student model using the trained knowledge selection module, which selects the type of knowledge the student model should learn according to the features of the corresponding teacher model and student model.

[0017] Step 1) is to train the teacher model using the available text classification task data, denoted as:

[0018] TeacherModel = ModelTrain(S, B)

[0019] where ModelTrain is the model training function, S is the input text set, B is the label of the input text, and represents a single text classification model training.

[0020] Step 2) is as follows:

[0021] 201) Suppose the text classification task is a binary classification task, and the input text is S = (s0, s1, s2, … s i …, s m ), where s i represents the i-th sentence in the text S; the corresponding label of the text classification is B = (b0, b1, b2, … b i …, b m ), where b i represents the corresponding classification label of the sentence s i , and its value is 0 or 1;

[0022] Use the teacher model to classify the sentence s i in the given text S, and generate three different types of knowledge;

[0023] 202) Extract the probability distribution P T (x i ) calculated by the teacher model for the input sentence x i as the reply knowledge;

[0024] 203) Extract the last layer representation of the teacher model as the feature knowledge L T (x i );

[0025] 204) Calculate the relative relationship between the outputs of each layer of the teacher model as the relative knowledge R T (x i ).

[0026] Step 4) is to train a knowledge selection module using the REINFORCE algorithm, and the specific steps are as follows:

[0027] 401) In each step of training, the representation vectors of the last layer of the teacher model and the student model are extracted respectively, and then connected as the feature input F(x) of the knowledge selection module, representing the learning state and data features of the teacher model and the student model in the corresponding knowledge distillation process;

[0028] 402) The knowledge selection module decides the type of knowledge that the student model should learn in the current state according to the input features, and lets the student model learn;

[0029] 403) After the knowledge distillation is completed, the score of the student model on the verification set is calculated, and then the difference between the score and the score of the initial state of the student model on the verification set is used as the reward value R obtained by the decision of the corresponding knowledge selection module;

[0030] 404) The reward value R and the loss of the decision made by the knowledge selection module in step 402) are used to update the parameters of the knowledge selection module;

[0031] 405) Repeat steps 401) to 404) until the model parameters of the knowledge selection module converge.

[0032] In step 5), the knowledge selection module trained in step 4) is used to decide the type of knowledge that the student model should learn in each step of the training process, and the specific steps are as follows:

[0033] 501) Extract the output representation vectors of the last layer of the teacher model and the student model in the current training step, and splice them into features F(x) to input the knowledge selection module to obtain the type of knowledge that the student model should learn in the current training step;

[0034] 502) According to the decision provided by the knowledge selection module, the parameters of the student model are updated, and finally the knowledge selection is realized.

[0035] The present application has the following advantages and benefits:

[0036] 1. The present application comprehensively divides the knowledge types in the knowledge distillation process, and the commonly used knowledge types in the knowledge distillation process are 1) reply knowledge, which is derived from the probability distribution of the last output of the teacher model; 2) feature knowledge, which is derived from the vector representation of the output of the last layer of the teacher model; 3) relative knowledge, which is derived from the relative relationship between the layers of the teacher model.

[0037] 2. The present application uses a neural network to select a specific knowledge type, which can automatically learn relevant features according to the input, so as to decide what type of knowledge the student model should learn in the current training step, and does not manually extract features like some existing methods, having good self-adaptive ability.

[0038] 3. The application can select multiple knowledge types, and can expand the knowledge space and reward function according to existing needs, and has the advantages of easy operation and easy expansion.

[0039] 4. In addition to the above, the application has strong flexibility, such as a teacher model that can use a relatively strong integrated model, so that the knowledge type that is more suitable for the student model to learn can be selected from the integrated model. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 a knowledge selection module structure of the application;

[0041] Figure 2 a flowchart of the overall process of the application. DETAILED DESCRIPTION

[0042] The application will be further described below in conjunction with the accompanying drawings of the specification.

[0043] The application provides a knowledge selection method applied to knowledge distillation, which can provide different knowledge types for a student model in the knowledge distillation process, and the specific process is as shown in Figure 2 , and includes the following steps:

[0044] 1) A teacher model is trained using data of a text classification task, and the teacher model has the characteristics of strong performance but large size. Here, the teacher model is set to be a pre-trained model BERT with 12 layers;

[0045] 2) The trained teacher model is used to extract different types of knowledge, and the knowledge is classified;

[0046] 3) The student model is initialized based on the trained teacher model;

[0047] In order to have a better initialization state for the student model, the parameters of the teacher model are used to initialize the student model. Here, the student model is set to be a BERT with 6 layers, so the first 6 layers of the teacher model are used to initialize the student model;

[0048] 4) The REINFORCE algorithm is used to train the knowledge selection module (structure as shown in Figure 1 );

[0049] First, in each training step, the hidden layer of the last layer of the teacher model and the hidden layer of the last layer of the student model are input to the KSM, and then the KSM selects a specific knowledge type for the student model to learn. After learning, the score of the student model on the validation set is calculated, and the difference between the score and the score of the student model on the validation set at the initial state is calculated, and the calculated difference is the reward corresponding to the KSM knowledge selection.

[0050] 5) Using the trained knowledge selection module to train the final student model, the knowledge selection module selects the type of knowledge that the student model should learn according to the characteristics of the corresponding teacher model and student model during training.

[0051] Step 1) is to train the teacher model using the available text classification task data, denoted as:

[0052] TeacherModel = ModelTrain (S, B)

[0053] Where ModelTrain is a model training function, S is a set of input texts, and B is the label of the input text, indicating that a single text classification model is trained, and the teacher model has the characteristics of good performance and large size;

[0054] A 12-layer BERT is fine-tuned as a knowledge provider, that is, a teacher model, using task-specific data.

[0055] During training, in order to improve the representation learning ability of the model, position encoding and multi-head attention are used to capture the relationship between different words in the sentence in the representation space, so that the model can learn more dimensional knowledge; Specifically:

[0056] 101) Calculate the position encoding of each word and add it to the word embedding vector to generate the input vector required by the model, the calculation method is as follows:

[0057]

[0058]

[0059] PE(.) is the position encoding function, pos is the position of the current word, i represents the dimension of the position encoding vector, d model is a basic parameter of the translation model, representing the size of the hidden layer at each position.

[0060] 102) Map the model vectors Q, K, and V into h subsets (h heads), and perform dot product and concatenation operations, the specific calculation is as follows:

[0061] head i =Attention(QW i Q ,KW I K ,VW I V )

[0062] Mult[Head(Q,K,V)=Concat(head1,…,headh )W o

[0063] head i is the vector representation of the i-th head, W is the translation model parameter, Attention(.) is the attention mechanism function, and Concat(.) is the vector concatenation function.

[0064] 103) Using a fully connected network, the vector representation after the attention mechanism operation is mapped to a new space, which is implemented as follows:

[0065] FFN(x) = max(0, xW1 + b1)W2 + b2 i )W2 + b2

[0066] where W1, W2, b1, b2 are the parameters of the translation model, which are automatically learned by the model.

[0067] 104) The model is trained using the corresponding cross-entropy loss function;

[0068] 105) Test the model, using F1 value as the measurement index.

[0069] Step 2), the specific steps are:

[0070] 201) Assuming that the text classification task is a binary classification task, the input text is S = (s0, s1, s2, …s i …,s m ), and s i represents the i-th sentence in the text S; the label corresponding to the text classification is B = (b0, b1, b2, …b i …,b m ), and b i represents the classification label corresponding to the sentence s i , whose value is 0 or 1;

[0071] Using the teacher model to classify the given text S sentence s i , generating three different types of knowledge;

[0072] 202) Extract the probability distribution P i (x T ) calculated by the teacher model for the input sentence x i as the reply knowledge;

[0073] 203) Extract the last layer representation of the teacher model as the feature knowledge L T (x i );

[0074] 204) Calculate the relative relationship between the outputs of each layer of the teacher model as the relative knowledge R T (xi ).

[0075] Using the teacher model, sentences s in the given text S. i The classification process involves extracting three different categories of knowledge: the first category is response knowledge, which is the probability distribution of sentence categories calculated by the teacher model, denoted as P. T (x i The second category is feature knowledge, which is the output representation of the last layer of the teacher model, denoted as L. T (x i The third category is relative knowledge, which is the relative correlation of the output representation of each layer of the teacher model, denoted as R. T (x i During the knowledge distillation process, all of this knowledge will be selected and provided to the student model for learning;

[0076] Step 3) Initializing the student model based on the trained teacher model ensures the student model has a good initial state. The parameters of the teacher model are used to initialize the student model. Here, the student model is set to a 6-layer BERT; therefore, the parameters of the first 6 layers of the teacher model are used to initialize the student model.

[0077] 4) A knowledge selection module, hereinafter referred to as KSM, is trained using the REINFORCE algorithm, specifically as follows:

[0078] 401) In each step of training, extract the representation vectors of the last layer of the teacher model and the student model respectively, and then concatenate them as the feature input F(x) of the knowledge selection module, which represents the learning state and data features of the teacher model and the student model in the corresponding knowledge distillation process;

[0079] 402) The knowledge selection module determines the type of knowledge that the student model should learn in the current state based on the input features, and then allows the student model to learn.

[0080] 403) After knowledge distillation is completed, calculate the score of the student model on the validation set, and then use the difference between this score and the score of the student model's initial state on the validation set as the reward value R for obtaining the decision of the corresponding knowledge selection module.

[0081] 404) Update the parameters of the knowledge selection module using the reward value R and the loss of the decision made by the knowledge selection module in step 402);

[0082] 405) Repeat steps 401) to 404) until the knowledge selection module model parameters converge.

[0083] Step 4) Firstly, in each training step, the hidden layer of the last layer of the KSM teacher model and the hidden layer of the last layer of the student model are inputted, and then the KSM selects a specific type of knowledge for the student model to learn. After learning, the score of the student model on the validation set is calculated, and the difference between the score and the score of the student model on the validation set at the initial state is calculated. The calculated difference is the reward corresponding to the KSM knowledge selection.

[0084] Step 5) is to use the knowledge selection module trained in step 4) to decide the type of knowledge that the student model should learn at each step of the training process. The specific steps are as follows:

[0085] 501) Extract the output representation vectors of the last layer of the teacher model and the student model in the current training step, and concatenate them into a feature F(x) to input the knowledge selection module to obtain the type of knowledge that the student model should learn in the current training step;

[0086] 502) According to the decision provided by the knowledge selection module, update the parameters of the student model, and finally realize knowledge selection.

[0087] The trained KSM is used to train the final student model. During training, the KSM will select the type of knowledge that the student model should learn according to the features of the corresponding teacher model and student model.

[0088] For example, when distilling a 6-layer BERT student model, the KSM will select the type of knowledge that the student model should learn in each training step. For example, at the beginning, the feature knowledge is selected to let the student model learn some internal knowledge first. When there is a certain amount of knowledge, the student model is allowed to learn the prediction probability distribution knowledge corresponding to the text, i.e. the reply knowledge, and finally the student is allowed to learn the relative relationship containing structural relationship.

[0089] Table 1

[0090] Dataset Original knowledge distillation method Fine-tuning based method Inventive method MRPC 85.5 / 79.8 85.1 / 79.2 87.9 / 82.2 QQP 70.0 / 88.4 69.4 / 88.2 71.1 / 89.1 MNLI-m / mm 82.4 / 81.6 81.1 / 79.8 83.1 / 82.1

[0091] The application uses a REINFORCE algorithm to train a knowledge selection module, which can select a suitable knowledge type for the student model to learn according to the learning characteristics of the teacher model and the student model in the knowledge distillation process, so that the student model can learn more suitable knowledge. Further, the performance of the student model distilled in the end of the knowledge distillation is improved. The application carries out experiments on three text classification data sets MRPC, QQP and MNLI, and reports the corresponding accuracy and F1 score in Table 1. In addition, two common pre-training model compression methods are reproduced, one is the original knowledge distillation method, which directly uses the output distribution of the teacher model to train the student model, without the knowledge selection process; the other is the method designed based on fine-tuning, which directly uses the parameters of the teacher model to initialize the parameters of the student model, and then fine-tunes the student model using data, without the knowledge distillation process. From the experimental results, the method of the application is obviously better than the other two methods, most obviously, the method of the application can achieve an accuracy score of 87.9 on the MRPC data set, which is 2.4 accuracy scores higher than the original knowledge distillation method. In summary, the method of the application can effectively improve the performance of knowledge distillation, and finally prove that the application effectively realizes knowledge selection in the knowledge distillation process.

Claims

1. A knowledge selection method applied to knowledge distillation, characterized in that The method comprises the following steps: 1) training a teacher model using data of a text classification task; 2) extracting different types of knowledge using the trained teacher model and classifying the knowledge; 3) initializing a student model based on the trained teacher model; 4) training a knowledge selection module using the REINFORCE algorithm; 5) training a final student model using the trained knowledge selection module, wherein the knowledge selection module selects the type of knowledge that the student model should learn according to the characteristics of the corresponding teacher model and student model during training; Step 4) trains a knowledge selection module using the REINFORCE algorithm, and the specific steps are as follows: 401) In each step of training, the representation vectors of the last layer of the teacher model and the student model are extracted respectively, and then they are connected to serve as the feature input F(x) of the knowledge selection module, which represents the learning state and data characteristics of the teacher model and the student model in the corresponding knowledge distillation process; 402) The knowledge selection module decides the type of knowledge that the student model should learn in the current state according to the input features, and makes the student model learn; 403) After the knowledge distillation is completed, the score of the student model on the verification set is calculated, and then the difference between the score and the score of the student model in the initial state on the verification set is used as the reward value R obtained by the decision of the corresponding knowledge selection module; 404) The reward value R and the loss L of the decision made by the knowledge selection module in step 402) are used to update the parameters of the knowledge selection module; 405) Steps 401) to 404) are repeated until the model parameters of the knowledge selection module converge; In step 5), the knowledge selection module trained in step 4) is used to decide the type of knowledge that the student model should learn in each step of the training process, and the specific steps are as follows: 501) The output representation vectors of the last layer of the teacher model and the student model in the current training step are extracted and spliced to become the feature F(x), which is input into the knowledge selection module to obtain the type of knowledge that should be learned by the student model in the current training step; 502) According to the decision provided by the knowledge selection module, the parameters of the student model are updated, and finally the knowledge selection is realized.

2. The knowledge selection method for knowledge distillation according to claim 1, wherein: Step 1) is to train a teacher model using the available text classification task data, which is represented as: TeacherModel=ModelTrain(S,B) Where ModelTrain is a model training function, S is a set of input texts, and B is a label of the input text, which represents a single training of a text classification model.

3. The knowledge selection method for knowledge distillation according to claim 1, wherein: The specific steps based on step 2) are as follows: 201) Let the text classification task be a binary classification task, and the input text be S = (s0, s1, s2, …s i n-1), where si represents the ith sentence in the text S. m i ​ The label corresponding to the text classification is B = (b0, b1, b2,... b i ...), where b m represents the classification label corresponding to the sentence s i , and the value is 0 or 1. i ​ using the teacher model on the given text S i produces three different categories of knowledge; 202) extract the teacher model's probabilities for the input sentence x i The computed probability distribution P T (x i ) as the reply knowledge; 203) extract the last layer representation of the teacher model as the feature knowledge L T (x i ); 204) computing relative relationships between outputs between each layer of the teacher model as relative knowledge R T (x i ).

Images