A Relation Classification Method Combining Distant Supervision and Supervised

Abstract

The present invention provides a method for combining remote supervision and supervised relationship classification. A teacher model is obtained through supervised data training. When a student model is trained using remote supervision data, entities are randomly masked. At the same time, the supervision signal of the teacher model is introduced using distillation technology to achieve the purpose of noise reduction. After obtaining the student model, the student model is fine-tuned on the supervised data, and the supervised data is further used to enhance the student model.

Description

Technical Field

[0001] The present invention relates to a relationship classification method combining remote supervision and supervision. Background Art

[0002] With the rapid development of the internet, a vast amount of information is generated daily. To more efficiently organize and utilize this information, Google proposed the concept of knowledge graphs in 2012. Knowledge graphs aggregate information, data, and links on the internet into knowledge, making information resources easier to compute and understand. This objective knowledge typically exists in knowledge graphs as structured triples: <head entity, relation, tail entity>. As a key technology for building knowledge graphs, relation classification has garnered widespread attention from both academia and industry in recent years. Relation classification aims to determine what kind of relationship between target entities is expressed in text (sentences or documents).

[0003] Currently, there are two main approaches to relational classification technology: supervised learning and distant supervision. Relational classification techniques based on supervised learning offer high accuracy, but they require a large number of high-quality, manually annotated samples. This leads to high annotation costs and long annotation times. To address the data shortage, distant supervision has been proposed. It can be used to automatically annotate large amounts of training data. Distant supervision achieves automatic annotation by aligning relational facts in a knowledge base with entities in the text. However, due to its strong assumptions, it inevitably introduces annotation noise. To leverage the respective strengths of supervised and distant supervision techniques, it is necessary to effectively combine these two technologies.

[0004]

Paper 1

[0005] [Paper 2] Dual Supervision Framework for Relation Extraction withDistant Supervision and Human Annotation. [2]

[0006] This paper uses two independent prediction networks to train with supervised and distant supervised data respectively to prevent the accuracy of the supervised model from decreasing due to incorrect labeling of distant supervision. This paper proposes an additional loss term to penalize the divergence between the supervised model and the distant supervised model, which enables the supervised model to learn from the distant supervised labels.

[0007]

Paper 3

[0008] This paper first applies a pre-denoising module to filter out some anonymous instances from all documents. It then pre-trains the document encoder using three pre-training tasks on a document-level distant supervision dataset. These pre-training tasks include entity mention-to-entity matching, relation discovery, and fact alignment. Finally, it fine-tunes the model on a manually annotated dataset.

[0009] The first paper uses supervised data to train a binary classifier that determines whether a sentence expresses a relationship to provide a supervisory signal of attention weights to the remote supervision model, which plays a role in noise reduction. However, only supervised data is used to train the binary classifier, and the relationship labels of the supervised data are not fully utilized. The second paper uses two independent networks to prevent the influence of remote supervision data noise on the supervised model, and at the same time uses divergence penalties to enable the supervised model to learn from the remote supervision labels. However, this method ignores the ability to extract features learned from a large amount of remote supervision data and fails to fully utilize the remote supervision data. The third paper designs several pre-training tasks to pre-train the document encoder on the remote supervision data, providing powerful feature extraction capabilities while reducing noise. However, the pre-training tasks designed by this method are only for document-level relationship classification and lack universality. Supervised data was not utilized during noise reduction, and the noise reduction effect is relatively general. Summary of the Invention

[0010] Purpose of the invention: The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and provide a relationship classification method that combines remote supervision and supervised classification, including the following steps:

[0011] Step 1: Obtain relation classification text data, including supervised relation classification text data and remotely supervised relation classification text data;

[0012] Step 2: train a relation classification teacher model on supervised relation classification text data;

[0013] Step 3: Train the relation classification student model on the remotely supervised relation classification text data, and perform noise reduction on the remotely supervised data through entity masking and distillation techniques;

[0014] Step 4: Fine-tune the student model trained in step 3 on supervised relational classification text data;

[0015] Step 5: Use the teacher model trained in step 4 to perform relationship prediction.

[0016] Step 1 includes:

[0017] Step 1-1, obtain supervised relation classification text data through manual annotation;

[0018] Step 1-2: Obtain automatically labeled relation classification text data through remote supervision technology.

[0019] Steps 1-2 include:

[0020] Step 1-2-1, obtain triple information in the knowledge graph. The triple information contains entity pairs and the relationship between entity pairs, forming a triple set R = {(h1, t1, r1), ..., (h i ,t i ,r i ),...,(h n ,t n ,r n )}, where n is the number of triples collected, h i represents the head entity in the i-th triplet, r i represents the relationship in the i-th triple, t i represents the tail entity in the i-th triple;

[0021] Step 1-2-2, obtain text from the Internet or other sources, and build an unstructured corpus D = {(text1,h1,t1),...,(text i ,h i ,t i ),...,(text m ,h m ,t m )}, where m represents the number of collected texts, text i Represents the i-th text;

[0022] Step 1-2-3: Use the distant supervision method to mark the relationship between the head and tail entity pairs (h, r) in each text in the text corpus D. Based on the assumption of distant supervision, the annotation is performed: if the entity pair appears in the triple R, it is annotated as the relationship indicated in R; if the triple does not appear in R, it is annotated as no relationship;

[0023] The assumption of remote supervision is too strong and does not take into account the text in which the entity pairs are located. Therefore, the obtained labeled data contains a lot of noise. The dataset obtained by remote supervision is denoted as Among them, m represents the number of remote supervision annotation samples, It represents the label of the i-th labeled sample obtained by remote supervision, which is encoded in a one-hot manner. The corresponding position of the labeled relationship is 1, and the other positions are 0.

[0024] Step 2 includes:

[0025] Step 2-1, initialize the parameters θ of the teacher model T ;

[0026] Step 2-2, input supervised relationship classification text data Among them, n represents the number of supervised samples, satisfying n<<m, that is, the number of supervised samples is much smaller than the number of remote supervised samples, text i represents the text of the i-th sample, h i and t i represents the head entity and tail entity in the i-th sample, represents the label of the i-th manually labeled sample, encoded using a one-hot encoding method;

[0027] Step 2-3, the teacher model performs forward calculation: the encoder encodes the input manually labeled sample x T ={w1,...,w i ,...,h,...,t,...,w n}, get the representation of each word token E = {e1,...,e i ,...,e h ,...,e t ,...,e n}. Among them, w i represents the i-th word in the context, h and t represent the head and tail entities respectively, e i represents the representation of the i-th word after being encoded by the encoder, e h and e t Respectively represent the representation of the head entity and the tail entity after encoding by the encoder;

[0028] Step 2-4, get e h and e t , input them into the bilinear layer and get the output z on each relationship category i , the calculation formula is: i =e h W i e t Where W i Represents the parameter matrix corresponding to the i-th relationship category, i = {1, 2, ..., c}, c represents the total number of relationship categories; then the softmax function is used to obtain the predicted probability of each relationship: Among them, exp represents the exponential function, represents the probability value of the i-th relationship;

[0029] Steps 2-5, calculate the probability distribution and manually annotated labels y T Cross entropy loss loss T : The cross entropy loss is used as the prediction loss of the teacher model; where is the true label of the i-th relation, represents the predicted probability of the teacher model on the i-th relationship;

[0030] Steps 2-6, calculate the gradient and back-propagate to update the parameters θ of the teacher model T ;

[0031] Steps 2-7, after iterative training, the teacher model is obtained.

[0032] Step 3 includes:

[0033] Step 3-1, initialize the parameters θ of the student model S ;

[0034] Step 3-2, input distantly supervised relation classification text data;

[0035] Step 3-3, use the teacher model trained in step 2 to label the input remote supervision sample x ds ={w1,...,w i ,h,...,t,...,w n} to make predictions and get the output results {z1,...,z i ,...,z c}, z i Represents the prediction result of the teacher model on the i-th relationship before normalization; the probability value on each relationship is calculated by the softmax function Compared to steps 2-4, there is an additional parameter T, which is used to represent the distillation temperature. The larger T is, the smoother the probability distribution is. The probability distribution calculated here is called the soft target;

[0036] Step 3-4, for the input remote supervision data sample x ds The entity part in is randomly masked. The random masking variable M follows the Bernoulli distribution. The probability of masking is P(M=1)=q. q is a hyperparameter used to control the proportion of masked entities. The input remote supervision data sample is expressed as:

[0037]

[0038] Among them, [E1] and [E2] are two special symbols, used to mask the head entity and the tail entity respectively; entity masking can make the student model pay more attention to contextual information instead of simply relying on entities for judgment, alleviate the model's preference for distant supervision assumptions, and thus improve the student model's noise resistance.

[0039] In steps 3-5, the student model performs forward calculations: the input sample is encoded by the encoder, and the encoded result is represented as E:

[0040]

[0041] e [E1] and e [E2] They represent the encoded representation of the masked head and tail entities, respectively, e i Represents the representation of the i-th word after being encoded by the encoder;

[0042] Step 3-6, obtain the entity representation in 3-5, when M=1, take e [E1] and e [E2] ; When M = 0, take e h and e t ; Input the entity representation into the bilinear layer and softmax layer to obtain the probability distribution P predicted by the student model S ;

[0043] Step 3-7, calculate the probability distribution P S and the distant supervision label y ds Cross entropy loss loss ds ;

[0044] Steps 3-8 calculate the difference between the probability distribution predicted by the student model and the probability distribution predicted by the teacher model.

[0045] Step 3-9, add the losses in steps 3-6 and 3-7 as the final prediction loss of the student model S ;

[0046] Step 3-10, calculate the gradient and back-propagate to update the parameters θ of the student model S ;

[0047] Steps 3-11, after iterative training, the student model is obtained.

[0048] Steps 3-7 include: calculating the probability distribution P using the following formula S and the distant supervision label y ds Cross entropy loss loss ds : in, is the distant supervision label of the i-th relation, represents the predicted probability of the student model on the i-th relationship.

[0049] Steps 3-8 include: selecting KL (Kullback-Leibler divergence) as the divergence measurement method, and the specific calculation formula is: loss KL Represents the KL divergence value of the probability distribution of the student model and the teacher model prediction. This loss uses the teacher model's prediction results to guide the student model, making the student model fit the teacher model's prediction results, thereby achieving the purpose of noise reduction;

[0050] Steps 3-9 include: loss s The calculation formula is: loss S =α*loss ds +(1-α)*loss KL , where α is a hyperparameter used to control the weight of the two loss terms;

[0051] Beneficial effects: From a technical perspective, the technical solution of the present invention proposes for the first time the use of entity masking and knowledge distillation to combine remote supervision and supervised relational classification models; entity masking reduces the student model's dependence on entity information, increases attention to text context content, and improves the student model's noise resistance; knowledge distillation technology is used to guide the student model using a teacher model trained with supervised data, thereby reducing the annotation noise of remote supervision data; the student model is fine-tuned on supervised data, and clean supervised data can be used to further correct the student model.

[0052] From an application perspective, the technical solution of this invention can automatically mine structured triples from massive amounts of text data, significantly reducing the labor cost required to populate knowledge graphs. By effectively combining noisy data from remote supervision with clean, manually annotated data, it improves the performance of relational classification models while reducing reliance on supervised data, thus helping to lower the labor cost of developing relational classification models. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0054] Figure 1 It is a flow chart of the method of the present invention.

[0055] Figure 2 It is the process of automatically annotating data through remote supervision technology.

[0056] Figure 3 It is the process of training a relation classification teacher model.

[0057] Figure 4 It is the process of training the student model for relation classification. DETAILED DESCRIPTION

[0058] Example

[0059] The overall process of the present invention is as follows Figure 1 、 Figure 4 Shown, including:

[0060] Step 1: Obtain relation classification text data. Specifically including:

[0061] Step 1-1, obtain supervised relationship classification text data through manual annotation. In this invention, the DocRED manually annotated dataset is used, which has a total of 5053 documents, 63,427 instances, and 96 relationships. Among them, 33.3% of the relationships are science-related, 11.5% of the relationships are art-related, 8.3% of the relationships are time-related, and 4.2% of the relationships are related to personal life. The supervised relationship classification text data is divided into a training set, a validation set, and a test set. The training set has 3053 documents, and the validation set and test set have 1000 documents each.

[0062] Step 1-2: Obtain automatically labeled relational classification text data through remote supervision technology. The process is as follows: Figure 2 Shown, including:

[0063] Step 1-2-1, obtain triple information from Wikidata, which contains entity pairs and the relationship between entity pairs;

[0064] Step 1-2-2: obtain some documents or sentences from Wikipedia and build an unstructured text corpus;

[0065] In step 1-2-3, a remote supervision method is used to label the relationships between entity pairs in documents or sentences. If the entity pair appears in Wikidata, it is labeled as a relationship in Wikidata, otherwise it is marked as no relationship, and a remotely supervised relationship classification text dataset is obtained. In this invention, the DocRED remote supervision dataset is used, which has a total of 101,873 documents, including 1,508,320 instances, involving 97 types of relationships (including None relationships). The relationship categories here are consistent with the relationship categories in step 1-1.

[0066] Step 2: Train the relation classification teacher model on the supervised relation classification text data. The specific training process is as follows: Figure 3 shown.

[0067] Step 2-1, initialize the parameters θ of the teacher model T In order to utilize the powerful encoding capability of the pre-trained model and the general knowledge learned on a large-scale corpus, the present invention uses Bert as the encoder. The parameters of the Bert part are initialized using the parameters of the pre-trained model; the bilinear layer of the teacher model is initialized using random initialization;

[0068] Step 2-2, input the supervised relation classification text data training set;

[0069] Step 2-3: The teacher model performs forward computation. The encoder encodes the input sample, and the result of the encoding is represented as E.

[0070] Step 2-4, get the head and tail entity representation e h and e t , input them into the bilinear layer and get the output z on each relationship category i , the calculation formula is: i =e h W i e t Where W i Represents the parameter matrix corresponding to the i-th relationship category, i = {1, 2, ..., c}, c represents the total number of relationship categories, which is 97 in this example; then the softmax function is used to obtain the predicted probability of each relationship: Among them, exp represents the exponential function, represents the probability value of the i-th relationship;

[0071] Steps 2-5, calculate the probability distribution and manually annotated labels y TCross entropy loss loss T : The cross entropy loss is used as the prediction loss of the teacher model; where is the true label of the i-th relation, represents the predicted probability of the teacher model on the i-th relationship;

[0072] Steps 2-6, calculate the gradient and back-propagate to update the parameters θ of the teacher model T The present invention uses Adam as the optimizer, with a learning rate of 2e-5 and a batch size of 30;

[0073] Steps 2-7: iterative training, the training epoch is set to 300, and the model corresponding to the epoch with the highest F1 on the validation set is selected as the teacher model.

[0074] Step 3-1, initialize the parameters θ of the student model S ,The network structures of the student model and the teacher model are the same,and also use the same initialization method;

[0075] Step 3-2, input distantly supervised relation classification text data;

[0076] Step 3-3, use the teacher model trained in step 2 to label the input remote supervision sample x ds ={w1,...,w i ,...,h,...,t,...,w n} to make predictions and get the output results {z1,...,z i ,...,z c}, z i Represents the prediction result of the teacher model on the i-th relationship before normalization; the probability value on each relationship is calculated by the softmax function The parameter T is used to represent the temperature of distillation. T used in the present invention is 1. Here, the sentence "Steve Jobs returns to Apple." and the entity pair "Steve Jobs" and "apple" are used as examples for explanation. For this example, the probability values ​​predicted by the teacher model on the pre-defined relationships are all lower than 0.05, while the predicted value on the special relationship category None is about 0.9, that is, the probability vector predicted by the teacher model is: [0.01,…,0.02,…,0.90]. This shows that the teacher model correctly judged that in this sentence, there is no pre-defined relationship between the entity pair "Steve Jobs" and "apple";

[0077] Step 3-4, for the input remote supervision data sample x dsThe entity part in [E1] is randomly masked. The randomly masked variable M follows a Bernoulli distribution with a masking probability of P(M=1)=q, where q is 0.7. Similarly, taking the sentence "Steve Jobs returns to Apple" and the entities "Steve Jobs" and "Apple" as an example, when M=1, the entity is masked, and the masked text becomes "[E1] returns to [E2]." When M=0, the entity is not masked, and the original text remains "Steve Jobs returns to Apple."

[0078] In steps 3-5, the student model performs forward computation. The input sample is encoded by the encoder, and the result obtained by encoding is represented as E.

[0079] Step 3-6, obtain the entity representation in 3-5, input the entity representation into the bilinear layer and softmax layer, and obtain the probability distribution P predicted by the student model S , P S is a 97-dimensional vector;

[0080] Step 3-7, calculate the probability distribution P s and the distant supervision label y ds Cross entropy loss loss ds : in, is the distant supervision label of the i-th relation, represents the predicted probability of the student model on the i-th relation; since the triple information collected from Wikidata contains <Steve Jobs, founded, Apple>, when using distant supervision technology to label it, it will be considered that the sentence "Steve Jobs returned to Apple." expresses "Steve Jobs founded Apple". Therefore, in this example, the obtained distant supervision label is wrong, and the calculated cross entropy loss is also wrong;

[0081] Steps 3-8 calculate the difference between the probability distribution predicted by the student model and the probability distribution predicted by the teacher model: For the above example, the teacher model's prediction is correct, and the loss calculated here is more reasonable than the loss in the previous step;

[0082] Step 3-9, add the losses in steps 3-6 and 3-7 as the final prediction loss of the student model S , the calculation formula is: loss S =α*loss ds +(1-α)*loss KL, where α is a hyperparameter that controls the weight of the two loss terms. α is set to 0.4. As can be seen, by weighting the two loss terms, the probability distribution of the remote supervision label and the teacher model prediction is comprehensively considered. In the above example, the remote supervision label is wrong. At this time, the correct prediction result of the teacher model can correct the overall loss term, thereby guiding the student model to perform correct training;

[0083] Step 3-10, calculate the gradient and back-propagate to update the parameters θ of the student model S The present invention uses Adam as the optimizer, with a learning rate of 2e-5 and a batch size of 30;

[0084] Step 3-11: iterative training, set the training epoch to 20, and select the model corresponding to the epoch with the highest F1 on the validation set as the student model.

[0085] Step 4: Fine-tune the student model trained in step 3 on the supervised relational classification text data. The number of rounds of fine-tuning is 20;

[0086] Step 5: End the model training and use the trained student model to predict samples in the test set. As shown in Table 1, the comparison results between the method described in this invention and existing mainstream methods on the DocRED dataset show that the method described in this invention has a significant advantage in the F1-Score evaluation criterion on both the validation set and the test set.

[0087] Table 1

[0088]

[0089] The present invention provides a method for combining remote supervision and supervised relationship classification. There are many methods and approaches for implementing this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A method for combining distant supervision and supervised relationship classification, characterized in that: The following steps are involved: Step 1, obtaining relation classification text data, including supervised relation classification text data and remotely supervised relation classification text data; Step 2, training a relation classification teacher model on supervised relation classification text data; Step 3: Train the relation classification student model on the remotely supervised relation classification text data, and denoise the remotely supervised data through entity masking and distillation techniques; Step 4, fine-tune the student model trained in step 3 on the supervised relational classification text data; Step 5, using the student model trained in step 4 to perform relationship prediction; Step 2 includes: Step 2-1, initialize the parameters θ of the teacher model T ; Step 2-2, input supervised relationship classification text data Among them, n represents the number of supervised samples, text i represents the text of the i-th sample, h i and t i represents the head entity and tail entity in the i-th sample, represents the label of the i-th labeled sample, encoded in a one-hot manner; Step 2-3, the teacher model performs forward calculation: the encoder encodes the input labeled sample x T ={w1,...,w i ,...,h,...,t,...,w n }, and get the representation of each word token E = {e1,...,e i ,...,e h ,...,e t ,...,e n }, where w i represents the i-th word in the context, h and t represent the head and tail entities respectively, and e i represents the representation of the i-th word after being encoded by the encoder, e h and e t They represent the representation of the head entity and the tail entity after being encoded by the encoder respectively; Step 2-4, get e h and e t , input them into the bilinear layer, and get the output result z on each relationship category i , the calculation formula is: i =e h W i e t , where W i Represents the parameter matrix corresponding to the i-th relationship category, i = {1, 2, ..., c}, c represents the total number of relationship categories; and then the softmax function is used to obtain the predicted probability of each relationship: Among them, exp represents the exponential function, represents the probability value of the i-th relationship; Step 2-5, calculate the probability distribution and the labeled label y T The cross entropy loss loss T : The cross entropy loss is used as the prediction loss of the teacher model; where is the true label of the i-th relation, represents the predicted probability of the teacher model on the i-th relationship; Step 2-6, calculate the gradient and back-propagate to update the parameters θ of the teacher model T ; Steps 2-7, after iterative training, the teacher model is obtained; Step 3 includes: Step 3-1, initialize the parameters θ of the student model S ; Step 3-2, input distant supervised relation classification text data; Step 3-3: Use the teacher model trained in step 2 to label the input remote supervision sample x ds ={w1,...,w i ,h,...,t,...,w n } to make predictions and obtain the output results {z1,...,z i ,...,z c }, z i Represents the prediction result of the teacher model on the i-th relationship before normalization; the probability value on each relationship is calculated by the softmax function The parameter T is used to represent the distillation temperature; the calculated probability distribution is called soft target; Step 3-4, for the input remote supervision data sample x ds The entity part in is randomly masked, and the variable M of random masking follows Bernoulli distribution. The probability of masking is P(M=1)=q, where q is a hyperparameter used to control the proportion of masked entities. The input remote supervision data sample is expressed as: Among them, [E1] and [E2] are two special symbols, which are used to cover the head entity and the tail entity respectively; Step 3-5, the student model performs forward calculation: the input sample is encoded by the encoder, and the encoded result is represented as E: e [E1] and e [E2] They represent the encoded representation of the masked head and tail entities respectively, e i Represents the representation of the i-th word after being encoded by the encoder; Step 3-6, obtain the entity representation in 3-5, when M = 1, take e [E1] and e [E2] ; When M = 0, take e h and e t ; Input the entity representation into the bilinear layer and the softmax layer to obtain the probability distribution P predicted by the student model S ; Step 3-7, calculate the probability distribution P S and the distant supervision label y ds The cross entropy loss loss ds : Step 3-8, calculate the difference between the probability distribution predicted by the student model and the probability distribution predicted by the teacher model; Step 3-9, add the losses in steps 3-6 and 3-7 as the final prediction loss of the student model S ; Step 3-10, calculate the gradient and back-propagate to update the parameters θ of the student model S ; Step 3-11, after iterative training, the student model is obtained.

2. The method according to claim 1, characterized in that Step 1 includes: Step 1-1, obtain supervised relational classification text data by annotation; Step 1-2, obtain automatically labeled relational classification text data through remote supervision technology.

3. The method according to claim 2, characterized in that Steps 1-2 include: Step 1-2-1, obtain the triple information in the knowledge graph. The triple information contains entity pairs and the relationship between entity pairs, forming a triple set R = {(h1, t1, r1), ..., (h i ,t i ,r i ),...,(h n ,t n ,r n )}, where n is the number of triples collected, h i represents the head entity in the i-th triple, r i represents the relationship in the i-th triple, t i represents the tail entity in the i-th triple; Step 1-2-2, obtain text from the Internet and build an unstructured corpus D = {(text1,h1,t1),...,(text i ,h i ,t i ),...,(text m ,h m ,t m )}, where m represents the number of collected texts, text i Represents the i-th text; Step 1-2-3, using the remote supervision method, for each text in the text corpus D, mark the relationship between the head and tail entity pairs (h, r) that appear in it, and label them based on the assumption of remote supervision: if the entity pair appears in the triple R, it is labeled as the relationship indicated in R, if the triple does not appear in R, it is labeled as no relationship; The dataset obtained through remote supervision annotation is denoted as Among them, m represents the number of remote supervision annotation samples, Represents the label of the i-th labeled sample obtained by remote supervision, which is encoded in a one-hot manner, with the corresponding position of the labeled relationship being 1 and the remaining positions being 0.

4. The method according to claim 3, characterized in that Step 3-7 includes: calculating the probability distribution P using the following formula S and the distant supervision label y ds The cross entropy loss loss ds : in, is the remote supervision label of the ith relation, represents the predicted probability of the student model on the i-th relationship.

5. The method according to claim 4, characterized in that Steps 3-8 include: Select KL divergence as the measurement method, and the specific calculation formula is: loss KL It represents the KL divergence value of the predicted probability distribution of the student model and the teacher model.

6. The method according to claim 5, characterized in that Steps 3-9 include: loss S The calculation formula is: loss s =α*loss ds +(1-α)*loss KL , where α is a hyperparameter.

Images