Text processing method and device based on knowledge distillation, equipment and medium

Abstract

The application discloses a text processing method and device based on knowledge distillation, equipment and medium. The text processing method based on knowledge distillation comprises the following steps: acquiring a natural language text, and acquiring a text processing model, the text processing model is obtained by knowledge distillation of a plurality of teacher models as a student model, a distillation loss is obtained by weighting the sequence labeling difference of the teacher model and the student model on the sample text based on the first weight of the teacher model, and a feature loss is obtained by weighting the output result difference of the teacher model and the student model on the corresponding network layer based on the second weight of the teacher model; and the natural language text is processed based on the text processing model, and at least a sequence labeling result of the natural language text is obtained. The above scheme can improve the accuracy of natural language text processing on the basis of improving the reliability and robustness of the text processing model, especially when the processing task contains sequence labeling.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a text processing method, apparatus, device and medium based on knowledge distillation. Background Technology

[0002] Natural Language Processing (NLP) is an important branch of computer science and artificial intelligence, and it is widely used in various industries. NLP provides theories and methods for enabling effective communication between humans and computers using natural language, allowing computers to understand, process, and generate language like humans do.

[0003] Currently, intent recognition and slot filling are two crucial subtasks in natural language understanding. Intent recognition focuses on predicting user intent based on user input, while slot filling focuses on extracting semantic concepts from user input as constraints for natural language, i.e., assigning semantic labels to each word in the input. Intent recognition and slot filling are generally performed using deep learning models. However, due to the high complexity and computational cost of most deep learning models, ensuring their reliability and robustness is difficult. Therefore, improving the reliability and robustness of text processing models while further enhancing the accuracy of natural language text processing, especially when tasks involve sequence labeling, has become a pressing issue. Summary of the Invention

[0004] The main technical problem addressed in this application is to provide a text processing method, apparatus, electronic device, and storage medium based on knowledge distillation, which can improve the reliability and robustness of the text processing model and further enhance the accuracy of natural language text processing, especially when the processing task involves sequence labeling.

[0005] To address the aforementioned issues, the first aspect of this application provides a text processing method based on knowledge distillation, comprising: acquiring natural language text and acquiring a text processing model, wherein the text processing model is used as a student model to perform knowledge distillation on several teacher models; during the knowledge distillation process, the student model adjusts network parameters based on training loss, wherein the training loss includes at least one of distillation loss and feature loss; the distillation loss is obtained by weighting the sequence labeling differences between the teacher model and the student model on the sample text based on a first weight of the teacher model; and the feature loss is obtained by weighting the output differences between the teacher model and the student model on the corresponding network layers based on a second weight of the teacher model; and processing the natural language text based on the text processing model to obtain at least the sequence labeling results of the natural language text.

[0006] To address the aforementioned problems, a second aspect of this application provides a text processing apparatus based on knowledge distillation, comprising: an acquisition module and a processing module. The acquisition module acquires natural language text and a text processing model. The text processing model, acting as a student model, performs knowledge distillation on several teacher models. During knowledge distillation, the student model adjusts network parameters based on a training loss, which includes at least one of distillation loss and feature loss. The distillation loss is obtained by weighting the sequence label differences between the teacher and student models on the sample text using a first weight of the teacher models. The feature loss is obtained by weighting the output differences between the teacher and student models at corresponding network layers using a second weight of the teacher models. The processing module processes the natural language text based on the text processing model to obtain at least the sequence label results of the natural language text.

[0007] To address the aforementioned problems, a third aspect of this application provides an electronic device including a memory and a processor coupled to each other. The memory stores program instructions, and the processor executes the program instructions to implement the knowledge distillation-based text processing method described in the first aspect.

[0008] To address the aforementioned problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used in the knowledge distillation-based text processing method described in the first aspect above.

[0009] The above scheme obtains natural language text and a text processing model. This text processing model, acting as a student model, performs knowledge distillation on several teacher models. During knowledge distillation, the student model adjusts network parameters based on training loss, which includes at least one of distillation loss and feature loss. The distillation loss is obtained by weighting the sequence label differences between the teacher and student models on the sample text using the teacher models' first weights. The feature loss is obtained by weighting the output differences between the teacher and student models at corresponding network layers using the teacher models' second weights. The scheme then processes the natural language text using the text processing model to obtain at least the sequence label results. This process involves obtaining the text processing model through knowledge distillation on several teacher models by the student model, and during the knowledge distillation process, the student model adjusts network parameters based on training loss. The training loss adjusts the network parameters. Since the training loss includes at least one of distillation loss and feature loss, it helps improve the performance of the student model. On the other hand, the distillation loss is obtained by weighting the differences in sequence labeling between the teacher model and the student model on the sample text based on the first weight of the teacher model. The feature loss is obtained by weighting the differences in output results between the teacher model and the student model at the corresponding network layers based on the second weight of the teacher model. Based on the differences in sequence labeling on the sample text and the differences in output results at the corresponding network layers, the distillation loss and feature loss are obtained respectively, which helps improve the reliability and rationality of knowledge transfer, thereby improving the training effect of the student model. On this basis, the text processing model is used to process natural language text to obtain the sequence labeling results of natural language text, further improving the accuracy of the sequence labeling results of natural language text. Therefore, it can improve the reliability and robustness of the text processing model and further improve the accuracy of natural language text processing, especially the accuracy when the task involves sequence labeling. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the text processing method based on knowledge distillation in this application;

[0011] Figure 2 This is a schematic diagram of the framework of an embodiment of the text processing method based on knowledge distillation in this application;

[0012] Figure 3 This is a schematic diagram of a framework for an embodiment of distillation loss during knowledge distillation;

[0013] Figure 4 This is a schematic diagram of the framework of an embodiment of the text processing apparatus based on knowledge distillation in this application;

[0014] Figure 5 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0015] Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0016] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0017] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0018] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. "Several" means at least one. The terms "first," "second," etc., in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the text processing method based on knowledge distillation according to this application. Specifically, it may include the following steps:

[0020] Step S11: Obtain natural language text and obtain the text processing model.

[0021] In this embodiment, the text processing model, acting as a student model, performs knowledge distillation on several teacher models. The teacher models can fine-tune or train the network model according to the specific task. The network models can include, but are not limited to, BERT, ERNIE, XLNet, ALBERT, etc., or the student model can train its own teacher model with a more complex network structure. During knowledge distillation, the student model adjusts the network parameters based on the training loss, which includes at least one of distillation loss and feature loss. The distillation loss is obtained by weighting the sequence label differences between the teacher model and the student model on the sample text using the teacher model's first weight. The feature loss is obtained by weighting the output differences between the teacher model and the student model at the corresponding network layers using the teacher model's second weight.

[0022] In one implementation scenario, the network layers in the teacher model are grouped into several first network groups, and the network layers in the student model are grouped into several second network groups. Based on the similarity of the output features between each first network group and each second network group, a correspondence between the first and second network groups is determined. Then, based on this correspondence, the first network layer in the first network group and the second network layer in the second network group are selected as corresponding network layers. Furthermore, in each round of training of the student model, the above steps for determining corresponding network layers can be repeated to ensure the accuracy of the corresponding network layers as much as possible. Alternatively, the corresponding network layers can be determined in the first round of training of the student model, eliminating the need to repeat this step in subsequent training rounds and saving training time. This method, by grouping the teacher and student models separately and calculating the similarity of the output features of each group to determine the corresponding network layers, helps improve the accuracy of the corresponding network layers, thereby improving the training effect of the student model.

[0023] In one implementation scenario, during each round of training of the student model, the first weight of the teacher model can be set to a fixed value. For example, the first weight can be set to 0.4, 0.6, 0.8, etc. The first weight can be determined according to the actual situation, and no specific limitation is made here.

[0024] In another implementation scenario, unlike the aforementioned implementation, in each round of training of the student model, the first weight of the teacher model can be determined based on the first loss of the teacher model's sequence labeling on the sample text, and the first weight is negatively correlated with the first loss. For example, this can be represented as follows:

[0025]

[0026]

[0027] in, This represents the sequence labeling of the k-th teacher model on the sample text, where s represents the SoftMax function, C is the number of classes, and y... c The first loss is calculated by comparing the sequence labels of the teacher model on the sample text with the actual sequence labels of the sample text. The calculated first loss is summed to obtain the loss values ​​of all teacher models, and then normalized to obtain the first weight. That is, the first weight of the k-th teacher model, and the first weight is negatively correlated with the first loss. The above method, by determining the first weight of the teacher model based on the first loss of the teacher model in sequence labeling on the sample text, helps to improve the correlation between the teacher model and the first weight, and further improves the accuracy of the first weight.

[0028] In one implementation scenario, sequence label differences can include third-party label differences. To obtain third-party label differences, we can first obtain the sequence labels of the teacher model and the student model on the sample text, as shown below:

[0029]

[0030] Where z = [z 1 ,..., C ] represents the sequence labels on the sample text, s represents the SoftMax function, C is the number of classes, and t represents the distillation temperature, which can be a hyperparameter. After obtaining the sequence labels of the teacher model and the student model on the sample text, the third label difference can be obtained based on the sequence labels of the teacher model and the student model on the sample text. Then, the distillation loss is obtained by weighting the third label difference between the teacher model and the student model on the sample text using the first weight of the teacher model. For example, it can be expressed as follows:

[0031]

[0032] Among them, L KD Indicates distillation loss, Indicates the first weight. This represents the sequence labeling of the teacher model on the sample text. This represents the sequence labeling of the student model on the sample text, and then the distillation loss is calculated.

[0033] In another implementation scenario, unlike the aforementioned implementation, the sequence labeling difference includes at least one of a first labeling difference and a second labeling difference. The first labeling difference is obtained based on the first prediction score of the target labeling sequence predicted by the teacher model in the student model, and the target labeling sequence is obtained by filtering based on the second prediction scores of each predicted labeling sequence output by the teacher model. The second labeling difference is obtained based on the prediction probability distribution of each character in the sample text by the teacher model and the student model. It should be noted that the sequence labeling difference may include only one of the first and second labeling differences, or it may be further combined to form the final sequence labeling difference. The selection method of the sequence labeling difference can be determined according to the actual situation and is not specifically limited here. The above method, by obtaining the first and second labeling differences through the teacher model and the student model respectively, helps to improve the accuracy of the first and second labeling differences. Furthermore, by obtaining at least one of the first and second labeling differences, the distillation loss is obtained, improving the accuracy of the distillation loss and further improving the training effect of the student model.

[0034] In a specific implementation scenario, to obtain the first annotation difference, the predicted annotation sequences output by the teacher model can be sorted in descending order of the second predicted scores. The predicted annotation sequence preceding a preset position is then selected as the target annotation sequence; the preset position can be 3, 5, etc., and is not specifically limited here. The first annotation difference is then obtained by averaging the first predicted scores of the target annotation sequences within the student model. For example, this can be represented as follows:

[0035]

[0036] Among them, L Top-K Indicates the difference in the first annotation, and 'k' represents the preset order. Represents the target labeled sequence. This represents the first predicted score of the student model, from which the first annotation difference is calculated.

[0037] Alternatively, the first annotation difference can be obtained by weighting the contribution of the target annotation sequence to the student model and its first predicted score in the student model. The contribution of the target annotation sequence to the student model is positively correlated with the second predicted score of the target annotation sequence. For example, this can be represented as follows:

[0038]

[0039]

[0040] Where k represents the preset order position. This represents the target label sequence, meaning we first calculate the contribution w of the target label sequence to the student model. t (y|x), further weighting the first prediction score of the student model by the contribution of the target annotation sequence to the student model, yields the first annotation difference L. Top-WK It should be noted that any method can be chosen to determine the first annotation difference based on the usage scenario. The above method, which processes the first annotation difference based on the first predicted score of the target annotation sequence in the student model, helps to improve the accuracy of the first annotation difference, thereby improving the training effect of the student model.

[0041] In a specific implementation scenario, to obtain the second annotation difference, the sub-difference on the corresponding character can be obtained based on the predicted probability distribution of the same character in the sample text by the teacher model and the student model. The predicted probability distribution includes the probability values ​​of the character in several label categories, and the predicted probability is a local probability predicted from a global perspective. Specifically, the probability values ​​of the character in several label categories are calculated according to the forward-backward algorithm, thereby conveying global structural knowledge in the form of local probability distributions and improving the algorithm performance of character-level tasks. For example, the expression can be represented as follows:

[0042]

[0043] ∝α(y k )×β(y k )

[0044] Where α represents prediction from front to back, and β represents prediction from back to front, the expression can be represented as follows:

[0045]

[0046]

[0047] For example, β(y) can be made n ) = 1. Furthermore:

[0048]

[0049]

[0050] Where x = {x1,..., n} represents the input natural language text sequence. Representing the labels on the sample text, r = {r1,..., n Let} represent the context representation of x after passing through the network structure, and Y(x) represent the set of all possible label sequences. Furthermore, in conditional random fields, and b y′, The emission score and transition score are represented. Further, based on the sub-differences on each character in the sample text, a second annotation difference is statistically obtained. For example, the expression can be represented as follows:

[0051]

[0052] Among them, L pos Indicates the difference in the second annotation, q t ( i =|x) represents the predicted probability distribution of the teacher model in the sample text, q s ( i =|x) represents the predicted probability distribution of the student model in the sample text. Based on the predicted probability distribution of the same characters in the sample text by the teacher model and the student model, sub-differences on corresponding characters are obtained. Then, based on the sub-differences on each character in the sample text, the second annotation difference is statistically obtained. This method, by obtaining the sub-differences on corresponding characters through the predicted probability distribution of the same characters in the sample text by the teacher model and the student model, and then statistically obtaining the second annotation difference based on these sub-differences, improves the accuracy of the second annotation difference and further enhances the training effect of the student model.

[0053] In one implementation scenario, during each round of training of the student model, the second weight of the teacher model can be set to a fixed value. For example, the second weight can be set to 0.4, 0.6, 0.8, etc. The second weight can be determined according to the actual situation, and no specific limitation is made here.

[0054] In another implementation scenario, during each training round of the student model, the second weights of the teacher model are determined based on the second loss of the teacher model's sequence labeling and / or intent recognition on the sample text, and the second weights are negatively correlated with the second loss. This approach, using the second loss of the teacher model's sequence labeling and / or intent recognition on the sample text to determine the second weights, helps improve the correlation between the teacher model and the second weights, further enhancing the accuracy of the second weights.

[0055] In a specific implementation scenario, to obtain the second loss, during the student model's processing of sample text, the feature vector output from the hidden layer of the student model can be obtained, along with the text processing result of the output layer of the teacher model processing the feature vector. This text processing result includes the predicted annotation sequence and the predicted text intent. Then, based on the difference between the predicted annotation sequence and the sample annotation sequence of the sample text, and / or the difference between the predicted text intent and the sample text intent of the sample text, the second loss is obtained. For example, the expression can be as follows:

[0056]

[0057]

[0058]

[0059] in, This represents the text processing result of the output layer in the k-th teacher model, which processes the feature vectors. This represents the processed vector of the feature vector output from the hidden layer in the student model, for example... This represents the feature vector output by the hidden layer in the student model. This represents the vector formed by average pooling the feature vectors output from the hidden layers in the student model. Further, based on the difference between the predicted labeled sequence and the sample labeled sequence of the sample text, and / or the difference between the predicted text intent and the sample text intent of the sample text, a second loss, y, is obtained. c The sample annotation sequence representing the sample text. This represents the second loss. Further, the second weights of the teacher model are determined based on the second loss in which the teacher model performs sequence labeling and / or intent recognition on the sample text. This represents the second weight. The above method, by obtaining the text processing results of the feature vector processed by the output layer in the teacher model and the feature vector output by the hidden layer in the student model, respectively, helps to determine the second loss, thereby improving the accuracy of the second loss and further enhancing the training effect of the student model.

[0060] In one implementation scenario, the output difference includes a first output difference, which is obtained by measuring the difference between the feature vectors output by the corresponding network layers when the teacher model and the student model process the same sample text. For example, the expression can be represented as follows:

[0061]

[0062] in, This represents the feature vectors output by the corresponding network layers when the teacher model processes sample text. Let r(·) represent the feature vectors output by the corresponding network layers when the student model processes sample text. r(·) is a function used to align the size of the feature vectors output by the student and teacher models. L represents the set of hidden layers, thus yielding the first output difference. Furthermore, the second weight based on the teacher model The feature loss L is obtained by weighting the differences in the output results of the teacher model and the student model at the corresponding network layers. inter The above method measures the difference between the feature vectors output by the corresponding network layers when the teacher model and the student model process the same sample text to obtain the first output difference, thereby improving the accuracy of the first output difference.

[0063] In one implementation scenario, the output difference includes a second output difference. To obtain the second output difference, we can first obtain the first similarity matrix when the teacher model processes several sample texts and the second similarity matrix when the student model processes several sample texts. Then, based on the difference between the first and second similarity matrices, we obtain the second output difference. The corresponding network layers include the first network layer in the teacher model and the second network layer in the student model. The first similarity matrix includes the similarity between the feature vectors output by the first network layer, and the second similarity matrix includes the similarity between the feature vectors output by the second network layer. For example, we can first obtain the third similarity matrix when the teacher model processes several sample texts and perform normalization to obtain the first similarity matrix. The expression can be represented as follows:

[0064]

[0065]

[0066] in, Let l represent the third similarity matrix and l represent the first network layer. Further, after obtaining the third similarity matrix, normalization is performed to obtain the first similarity matrix. Furthermore, we can obtain the second similarity matrix when the student model processes several sample texts. Specifically, we can first obtain the fourth similarity matrix when the student model processes several sample texts, and then normalize it to obtain the fourth similarity matrix. The expression can be represented as follows:

[0067]

[0068]

[0069] in, Let m represent the fourth similarity matrix, and m represent the second network layer. Further, after obtaining the fourth similarity matrix, normalization is performed to obtain the second similarity matrix. Based on the difference between the first and second similarity matrices, the second output difference is obtained, which can be expressed as follows:

[0070]

[0071] Among them, L sim (G T G S ) represents the second output difference, L represents the set of hidden layers, and ||·|| F This represents the Frobenius norm. It's understandable that obtaining the second output difference doesn't require aligning the feature dimensions between the feature vectors output by the teacher and student models; it only requires maintaining pairwise similarity in their respective feature spaces. The above method, by obtaining the first and second similarity matrices of the teacher and student models respectively, and then obtaining the difference between the first and second similarity matrices, further improves the accuracy of the second output difference, thereby enhancing the training effect of the student model.

[0072] In one implementation scenario, the output difference includes a third output difference. To obtain the third output difference, we can first obtain the first text ranking when the teacher model processes several sample texts and the second text ranking when the student model processes several sample texts. Based on the difference between the first and second text rankings, we obtain the third output difference. The corresponding network layers include the first network layer in the teacher model and the second network layer in the student model. The first text ranking is obtained by ranking the sample texts based on the feature distance between several sample texts in the first network layer, and the second text ranking is obtained by ranking the sample texts based on the feature distance between several sample texts in the second network layer. For example, the expression can be represented as follows:

[0073]

[0074] Among them, L sim (G T G S () indicates the third output difference, and N represents the number of texts. Indicates the first text sorting. The second text ranking is represented by `rank()`, which sorts the samples by feature distance from smallest to largest. This method obtains the first and second text rankings when the teacher and student models process several sample texts, respectively. Based on these rankings, a third output difference is determined, further improving the accuracy of the third output difference and thus enhancing the training effect of the student model.

[0075] Step S12: Process the natural language text based on the text processing model to obtain at least the sequence labeling results of the natural language text.

[0076] In one implementation scenario, the text processing model is used as a student model to perform knowledge distillation on several teacher models. Furthermore, by processing natural language text based on the text processing model, sequence labeling results of natural language text can be obtained.

[0077] The above scheme obtains natural language text and a text processing model. This text processing model, acting as a student model, performs knowledge distillation on several teacher models. During knowledge distillation, the student model adjusts network parameters based on training loss, which includes at least one of distillation loss and feature loss. The distillation loss is obtained by weighting the sequence label differences between the teacher and student models on the sample text using the teacher models' first weights. The feature loss is obtained by weighting the output differences between the teacher and student models at corresponding network layers using the teacher models' second weights. The scheme then processes the natural language text using the text processing model to obtain at least the sequence label results. This process involves obtaining the text processing model through knowledge distillation on several teacher models by the student model, and during the knowledge distillation process, the student model adjusts network parameters based on training loss. The training loss adjusts the network parameters. Since the training loss includes at least one of distillation loss and feature loss, it helps improve the performance of the student model. On the other hand, the distillation loss is obtained by weighting the differences in sequence labeling between the teacher model and the student model on the sample text based on the first weight of the teacher model. The feature loss is obtained by weighting the differences in output results between the teacher model and the student model at the corresponding network layers based on the second weight of the teacher model. Based on the differences in sequence labeling on the sample text and the differences in output results at the corresponding network layers, the distillation loss and feature loss are obtained respectively, which helps improve the reliability and rationality of knowledge transfer, thereby improving the training effect of the student model. On this basis, the text processing model is used to process natural language text to obtain the sequence labeling results of natural language text, further improving the accuracy of the sequence labeling results of natural language text. Therefore, it can improve the reliability and robustness of the text processing model and further improve the accuracy of natural language text processing, especially the accuracy when the task involves sequence labeling.

[0078] Please see Figure 2 , Figure 2 This is a schematic diagram of the framework of an embodiment of the text processing method based on knowledge distillation in this application, as shown below. Figure 2 As shown, in the knowledge distillation process, the text can first be input into n teacher models and student models respectively. Then, the first weight of the teacher model is determined by the first loss based on the sequence labeling of the teacher model on the sample text, and the second weight of the teacher model is determined by the second loss based on the sequence labeling and / or intent recognition of the teacher model on the sample text. Finally, the distillation loss L is obtained by weighting the differences in sequence labeling between the teacher model and the student model on the sample text based on the first weight of the teacher model. KD The feature loss L is obtained by weighting the differences between the output results of the teacher model and the student model at the corresponding network layers based on the second weight of the teacher model. interFurthermore, during knowledge distillation, it is also necessary to obtain the cross-entropy loss between the sequence labels predicted by the student model on the sample text and the sample sequence labels of the sample text. The expression can be represented as follows:

[0079]

[0080] Among them, y c The sample sequence label represents the sample text. L represents the sequence labels predicted by the student model on the sample text. CE This represents the cross-entropy loss. Further, the total loss value is obtained through distillation loss, feature loss, and cross-entropy loss, and can be expressed as follows:

[0081] L=αL CE +βL KD +γL inter

[0082] Where L represents the total loss value, L CE Represents cross-entropy loss, L KD Indicates distillation loss, L inter This represents the feature loss, and α+β+γ=1. For example, α is 0.5, β is 0.3, and γ is 0.2. The values ​​of the weights can be determined according to the actual situation, and no specific limit is made here.

[0083] The above scheme, on the one hand, obtains a text processing model by performing knowledge distillation on several teacher models using a student model. During the knowledge distillation process, the student model adjusts its network parameters based on the training loss. Since the training loss includes at least one of the distillation loss and feature loss, it helps improve the performance of the student model. On the other hand, the distillation loss is obtained by weighting the differences in sequence labeling between the teacher and student models on the sample text using the first weight of the teacher model, and the feature loss is obtained by weighting the differences in the output results of the teacher and student models at the corresponding network layers using the second weight of the teacher model. Based on the differences in sequence labeling on the sample text and the differences in the output results at the corresponding network layers, the distillation loss and feature loss are obtained respectively, which helps improve the reliability and rationality of knowledge transfer, thereby improving the training effect of the student model. On this basis, the text processing model processes natural language text to obtain the sequence labeling results of the natural language text, further improving the accuracy of the sequence labeling results of the natural language text. Therefore, it can improve the reliability and robustness of the text processing model and further improve the accuracy of natural language text processing, especially when the task involves sequence labeling.

[0084] Please see Figure 3 , Figure 3 This is a schematic diagram of a framework for an embodiment of distillation loss during knowledge distillation, as shown below. Figure 3 As shown, n words are input into the teacher model and the student model respectively. The teacher model and the student model can predict the labeled sequence, i.e., label 1, label 2, ..., label n. Based on the predicted labeled sequence, the cross-entropy loss L between the sequence labels predicted by the student model on the sample text and the sample sequence labels of the sample text can be obtained. CE Furthermore, sequence label differences can be determined based on the predicted labeled sequences. Specifically, sub-differences on corresponding characters can be obtained based on the predicted probability distributions of the same characters in the sample text by the teacher model and the student model; then, based on the sub-differences on each character in the sample text, the second label difference L can be statistically obtained. pos The predicted label sequences output by the teacher model are sorted in descending order of the second predicted scores. The predicted label sequences preceding the preset order are selected as the target label sequences. The first label difference L is obtained by averaging the first predicted scores of the target label sequences in the student model. Top-K Alternatively, the first annotation difference L can be obtained by weighting the contribution of the target annotation sequence to the student model and its first predicted score in the student model. Top-WK It is understandable that sequence labeling differences include at least one of the first labeling difference and the second labeling difference. The first labeling difference includes L... Top-K L Top-WK One of them.

[0085] The above scheme, on the one hand, obtains a text processing model by performing knowledge distillation on several teacher models using a student model. During the knowledge distillation process, the student model adjusts its network parameters based on the training loss. Since the training loss includes at least one of the distillation loss and feature loss, it helps improve the performance of the student model. On the other hand, the distillation loss is obtained by weighting the differences in sequence labeling between the teacher and student models on the sample text using the first weight of the teacher model, and the feature loss is obtained by weighting the differences in the output results of the teacher and student models at the corresponding network layers using the second weight of the teacher model. Based on the differences in sequence labeling on the sample text and the differences in the output results at the corresponding network layers, the distillation loss and feature loss are obtained respectively, which helps improve the reliability and rationality of knowledge transfer, thereby improving the training effect of the student model. On this basis, the text processing model processes natural language text to obtain the sequence labeling results of the natural language text, further improving the accuracy of the sequence labeling results of the natural language text. Therefore, it can improve the reliability and robustness of the text processing model and further improve the accuracy of natural language text processing, especially when the task involves sequence labeling.

[0086] Please see Figure 4 , Figure 4This is a schematic diagram of the framework of an embodiment of the text processing apparatus based on knowledge distillation of this application. The text processing apparatus 40 based on knowledge distillation includes: an acquisition module 41 and a processing module 42. The acquisition module 41 is used to acquire natural language text and a text processing model; wherein the text processing model is obtained by knowledge distillation of several teacher models as a student model. During the knowledge distillation process, the student model adjusts network parameters based on training loss, and the training loss includes at least one of distillation loss and feature loss. The distillation loss is obtained by weighting the sequence label differences between the teacher model and the student model on the sample text based on the first weight of the teacher model, and the feature loss is obtained by weighting the output differences between the teacher model and the student model on the corresponding network layers based on the second weight of the teacher model. The processing module 42 is used to process the natural language text based on the text processing model to obtain at least the sequence label results of the natural language text.

[0087] The above scheme, on the one hand, obtains a text processing model by performing knowledge distillation on several teacher models using a student model. During the knowledge distillation process, the student model adjusts its network parameters based on the training loss. Since the training loss includes at least one of the distillation loss and feature loss, it helps improve the performance of the student model. On the other hand, the distillation loss is obtained by weighting the differences in sequence labeling between the teacher and student models on the sample text using the first weight of the teacher model, and the feature loss is obtained by weighting the differences in the output results of the teacher and student models at the corresponding network layers using the second weight of the teacher model. Based on the differences in sequence labeling on the sample text and the differences in the output results at the corresponding network layers, the distillation loss and feature loss are obtained respectively, which helps improve the reliability and rationality of knowledge transfer, thereby improving the training effect of the student model. On this basis, the text processing model processes natural language text to obtain the sequence labeling results of the natural language text, further improving the accuracy of the sequence labeling results of the natural language text. Therefore, it can improve the reliability and robustness of the text processing model and further improve the accuracy of natural language text processing, especially when the task involves sequence labeling.

[0088] In some public embodiments, during each round of training of the student model, the first weight of the teacher model is determined based on the first loss of the teacher model performing sequence labeling on the sample text, and the first weight is negatively correlated with the first loss.

[0089] Therefore, determining the first weight of the teacher model by using the first loss of sequence labeling on the sample text based on the teacher model helps to improve the correlation between the teacher model and the first weight, and further improve the accuracy of the first weight.

[0090] In some disclosed embodiments, the sequence labeling difference includes at least one of a first labeling difference and a second labeling difference; wherein, the first labeling difference is obtained based on the first prediction score of the target labeling sequence predicted by the teacher model in the student model, and the target labeling sequence is obtained by filtering based on the second prediction scores of each predicted labeling sequence output by the teacher model, and the second labeling difference is obtained based on the prediction probability distribution of each character in the sample text by the teacher model and the student model.

[0091] Therefore, obtaining the first and second annotation differences through the teacher model and the student model respectively helps to improve the accuracy of the first and second annotation differences. Furthermore, by obtaining at least one of the first and second annotation differences, the distillation loss is obtained, which improves the accuracy of the distillation loss and further enhances the training effect of the student model.

[0092] In some disclosed embodiments, the acquisition module 41 includes a first measurement submodule, which is used to sort the predicted label sequences output by the teacher model in descending order of the second predicted scores; select the predicted label sequence located before the preset order as the target label sequence; and obtain the first label difference by averaging the first predicted scores of the target label sequences in the student model, or by weighting the contribution of the target label sequences to the student model and the first predicted scores in the student model; wherein the contribution of the target label sequences to the student model is positively correlated with the second predicted scores of the target label sequences.

[0093] Therefore, processing the first annotation difference based on the first predicted score of the target annotation sequence in the student model helps to improve the accuracy of the first annotation difference, thereby improving the training effect of the student model.

[0094] In some disclosed embodiments, the acquisition module 41 includes a second measurement submodule, which is used to obtain sub-differences on corresponding characters based on the predicted probability distribution of the same characters in the sample text by the teacher model and the student model; wherein, the predicted probability distribution includes the probability values ​​of the characters in several label categories, and the predicted probability is a local probability predicted from a global perspective; based on the sub-differences on each character in the sample text, the second label difference is statistically obtained.

[0095] Therefore, by using the predicted probability distribution of the same characters in the sample text by the teacher model and the student model, the sub-differences on the corresponding characters are obtained. Then, based on the sub-difference statistics, the second annotation difference is obtained, thereby improving the accuracy of the second annotation difference and further improving the training effect of the student model.

[0096] In some public embodiments, during each round of training of the student model, a second weight of the teacher model is determined based on a second loss of the teacher model performing sequence labeling and / or intent recognition on sample text, and the second weight is negatively correlated with the second loss.

[0097] Therefore, determining the second weights of the teacher model through the second loss of sequence labeling and / or intent recognition on sample text by the teacher model helps to improve the correlation between the teacher model and the second weights, and further improve the accuracy of the second weights.

[0098] In some disclosed embodiments, the acquisition module 41 includes an acquisition submodule, which is used to acquire the feature vector output by the hidden layer in the student model during the student model's processing of sample text; acquire the text processing result of the output layer processing the feature vector in the teacher model; wherein the text processing result includes a predicted annotation sequence and a predicted text intent; and obtain a second loss based on the difference between the predicted annotation sequence and the sample annotation sequence of the sample text, and / or the difference between the predicted text intent and the sample text intent of the sample text.

[0099] Therefore, by obtaining the text processing results of the feature vector processed by the output layer in the teacher model and the feature vector output by the hidden layer in the student model respectively, and then determining the second loss, it is helpful to improve the accuracy of the second loss and further improve the training effect of the student model.

[0100] In some disclosed embodiments, the output difference includes a first output difference, which is obtained by measuring the difference based on the feature vectors output by the corresponding network layers when the teacher model and the student model process the same sample text.

[0101] Therefore, the first output difference is obtained by measuring the difference between the feature vectors output by the corresponding network layers when the teacher model and the student model process the same sample text, thereby improving the accuracy of the first output difference.

[0102] In some disclosed embodiments, the output difference includes a second output difference. The acquisition module 41 includes a third measurement submodule, which is used to acquire a first similarity matrix when the teacher model processes several sample texts and a second similarity matrix when the student model processes several sample texts. Based on the difference between the first similarity matrix and the second similarity matrix, the second output difference is obtained. The corresponding network layers include a first network layer in the teacher model and a second network layer in the student model. The first similarity matrix includes the similarity between the feature vectors output by the first network layer and the second similarity matrix includes the similarity between the feature vectors output by the second network layer.

[0103] Therefore, by obtaining the first and second similarity matrices of the teacher model and the student model respectively, and obtaining the difference between the first and second similarity matrices, the second output difference is obtained, which further improves the accuracy of the second output difference and thus improves the training effect of the student model.

[0104] In some disclosed embodiments, the output difference includes a third output difference. The acquisition module 41 includes a fourth measurement submodule, which is used to acquire the first text ranking when the teacher model processes several sample texts and the second text ranking when the student model processes several sample texts. Based on the difference between the first text ranking and the second text ranking, the third output difference is obtained. The corresponding network layers include the first network layer in the teacher model and the second network layer in the student model. The first text ranking is obtained by ranking several sample texts based on the feature distance between several sample texts in the first network layer, and the second text ranking is obtained by ranking several sample texts based on the feature distance between several sample texts in the second network layer.

[0105] Therefore, by obtaining the first and second text rankings when the teacher model and student model process several sample texts respectively, and then determining the third output difference based on the second and second text rankings, the accuracy of the third output difference is further improved, thereby further improving the training effect of the student model.

[0106] In some disclosed embodiments, the acquisition module 41 includes a determination submodule, which is used to group the network layers in the teacher model to obtain several first network groups and to group the network layers in the student model to obtain several second network groups; based on the similarity between each first network group and each second network group with respect to output features, the correspondence between the first network group and the second network group is determined; based on the correspondence, the first network layer in the first network group and the second network layer in the second network group are selected as corresponding network layers.

[0107] Therefore, by grouping the teacher model and the student model separately and calculating the similarity based on the output features of each group, the corresponding network layer can be determined, which helps to improve the accuracy of the corresponding network layer and thus improve the training effect of the student model.

[0108] Please see Figure 5 , Figure 5 This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other. The memory 51 stores program instructions, and the processor 52 is used to execute the program instructions to implement the steps in any of the above embodiments of the text processing method based on knowledge distillation. Specifically, the electronic device 50 may include, but is not limited to, desktop computers, laptops, servers, mobile phones, tablet computers, etc., and is not limited thereto.

[0109] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the above-described embodiments of the knowledge distillation-based text processing method. Processor 52 may also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 52 may be implemented using integrated circuit chips.

[0110] In the above scheme, the electronic device 50 can implement the steps in any of the above embodiments of the text processing method based on knowledge distillation. On the one hand, a text processing model is obtained by performing knowledge distillation on several teacher models using a student model. During the knowledge distillation process, the student model adjusts the network parameters based on the training loss. Since the training loss includes at least one of distillation loss and feature loss, it helps to improve the performance of the student model. On the other hand, the distillation loss is obtained by weighting the differences in sequence labeling between the teacher model and the student model on the sample text based on the first weight of the teacher model. The feature loss is obtained by weighting the differences in output results between the teacher model and the student model on the corresponding network layers based on the second weight of the teacher model. Based on the differences in sequence labeling on the sample text and the differences in output results on the corresponding network layers, the distillation loss and feature loss are obtained respectively, which helps to improve the reliability and rationality of knowledge transfer, thereby improving the training effect of the student model. On this basis, the natural language text is processed based on the text processing model to obtain the sequence labeling results of the natural language text, further improving the accuracy of the sequence labeling results of the natural language text. Therefore, it is possible to improve the reliability and robustness of the text processing model and further improve the accuracy of natural language text processing, especially when the processing task includes sequence labeling.

[0111] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor. The program instructions 61 are used to implement the steps in any of the above embodiments of the text processing method based on knowledge distillation.

[0112] The above scheme, implemented by the computer-readable storage medium 60, allows for the execution of the steps in any of the knowledge distillation-based text processing method embodiments. On one hand, a text processing model is obtained by performing knowledge distillation on several teacher models using a student model. During knowledge distillation, the student model adjusts network parameters based on training loss. Since the training loss includes at least one of distillation loss and feature loss, it helps improve the performance of the student model. On the other hand, the distillation loss is obtained by weighting the sequence label differences between the teacher and student models on the sample text using the first weight of the teacher model. The feature loss is obtained by weighting the output differences between the teacher and student models on the corresponding network layers using the second weight of the teacher model. Based on the differences in sequence labeling on the sample text and the differences in output results on the corresponding network layers, the distillation loss and feature loss are obtained respectively, which helps improve the reliability and rationality of knowledge transfer, thereby improving the training effect of the student model. On this basis, the text processing model is used to process natural language text, obtaining sequence labeling results for the natural language text, further improving the accuracy of the natural language text sequence labeling results. Therefore, it is possible to improve the reliability and robustness of the text processing model while further improving the accuracy of natural language text processing, especially when the processing task includes sequence labeling.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] If the technical solution of this application complies with the relevant laws and regulations regarding the collection, dissemination, and use of data.

Claims

1. A text processing method based on knowledge distillation, characterized in that, include: Natural language text is acquired, and a text processing model is acquired. The text processing model is used as a student model to perform knowledge distillation on several teacher models. During the knowledge distillation process, the student model adjusts network parameters based on training loss, where the training loss includes at least one of distillation loss and feature loss. The distillation loss is obtained by weighting the sequence label differences between the teacher model and the student model on the sample text based on a first weight of the teacher model. The feature loss is obtained by weighting the output differences between the teacher model and the student model on the corresponding network layers based on a second weight of the teacher model. The sequence label differences include at least one of first label differences and second label differences. The first label difference is obtained based on the first prediction score of the target label sequence predicted by the teacher model in the student model, and the target label sequence is obtained by filtering based on the second prediction scores of each predicted label sequence output by the teacher model. The second label difference is obtained based on the prediction probability distribution of each character in the sample text by the teacher model and the student model. The steps for measuring the first annotation difference include: The predicted label sequences output by the teacher model are sorted in descending order of the second predicted scores. Select the predicted label sequence that precedes the preset order position as the target label sequence; The first labeling difference is obtained by averaging the target labeling sequence on the first predicted score of the student model, or by weighting the contribution of the target labeling sequence to the student model and the first predicted score of the student model. The contribution of the target label sequence to the student model is positively correlated with the second prediction score of the target label sequence. The natural language text is processed based on the text processing model to obtain at least the sequence labeling results of the natural language text.

2. The method according to claim 1, characterized in that, In each round of training of the student model, the first weight of the teacher model is determined based on the first loss of the teacher model performing sequence labeling on the sample text, and the first weight is negatively correlated with the first loss.

3. The method according to claim 1, characterized in that, The second step in measuring the difference in annotations includes: Based on the predicted probability distribution of the same character in the sample text by the teacher model and the student model, the sub-differences on the corresponding character are obtained; wherein, the predicted probability distribution includes the probability values ​​of the character in several label categories, and the predicted probability is a local probability predicted from a global perspective; The second annotation difference is obtained by statistically analyzing the sub-differences on each character in the sample text.

4. The method according to claim 1, characterized in that, In each round of training of the student model, a second weight of the teacher model is determined based on a second loss of the teacher model performing sequence labeling and / or intent recognition on the sample text, and the second weight is negatively correlated with the second loss.

5. The method according to claim 4, characterized in that, The steps for obtaining the second loss include: During the process of the student model processing the sample text, the feature vector output by the hidden layer in the student model is obtained; Obtain the text processing result of the output layer in the teacher model processing the feature vector; wherein, the text processing result includes the predicted labeled sequence and the predicted text intent; The second loss is obtained based on the difference between the predicted annotation sequence and the sample annotation sequence of the sample text, and / or the difference between the predicted text intent and the sample text intent of the sample text.

6. The method according to claim 1 or 4, characterized in that, The output difference includes a first output difference, which is obtained by measuring the difference based on the feature vectors output by the corresponding network layers when the teacher model and the student model process the same sample text.

7. The method according to claim 1 or 4, characterized in that, The output difference includes a second output difference, and the measurement steps for the second output difference include: Obtain the first similarity matrix when the teacher model processes several sample texts and the second similarity matrix when the student model processes the several sample texts; The second output difference is obtained based on the difference between the first similarity matrix and the second similarity matrix; The corresponding network layer includes a first network layer in the teacher model and a second network layer in the student model. The first similarity matrix includes the similarity between the feature vectors output by the first network layer and the second similarity matrix includes the similarity between the feature vectors output by the second network layer.

8. The method according to claim 1 or 4, characterized in that, The output difference includes a third output difference, and the measurement steps for the third output difference include: Obtain the first text sorting when the teacher model processes several sample texts and the second text sorting when the student model processes the several sample texts; The third output difference is obtained based on the difference between the first text sorting and the second text sorting; The corresponding network layer includes a first network layer in the teacher model and a second network layer in the student model. The first text ranking is obtained by ranking the sample texts based on the feature distance between the sample texts in the first network layer, and the second text ranking is obtained by ranking the sample texts based on the feature distance between the sample texts in the second network layer.

9. The method according to claim 1 or 4, characterized in that, The steps for determining the corresponding network layer include: The network layers in the teacher model are grouped to obtain several first network groups, and the network layers in the student model are grouped to obtain several second network groups. Based on the similarity between each of the first network groups and each of the second network groups regarding output features, the correspondence between the first network groups and the second network groups is determined. Based on the correspondence, the first network layer in the first network group and the second network layer in the second network group are selected as the corresponding network layers.

10. A text processing device based on knowledge distillation, characterized in that, include: An acquisition module is used to acquire natural language text and a text processing model. The text processing model is obtained by performing knowledge distillation on several teacher models as a student model. During the knowledge distillation process, the student model adjusts network parameters based on training loss, and the training loss includes at least one of distillation loss and feature loss. The distillation loss is obtained by weighting the sequence label differences between the teacher model and the student model on the sample text based on a first weight of the teacher model. The feature loss is obtained by weighting the output differences between the teacher model and the student model on the corresponding network layers based on a second weight of the teacher model. The sequence label differences include at least one of first label differences and second label differences. The first label difference is obtained based on the first prediction score of the target label sequence predicted by the teacher model in the student model, and the target label sequence is obtained by filtering based on the second prediction scores of each predicted label sequence output by the teacher model. The second label difference is obtained based on the prediction probability distribution of each character in the sample text by the teacher model and the student model. The processing module is used to process the natural language text based on the text processing model, and at least obtain the sequence labeling results of the natural language text; The acquisition module includes a first measurement submodule, which is used to sort the predicted label sequences output by the teacher model in descending order of the second predicted scores; select the predicted label sequence located before the preset order as the target label sequence; and obtain the first label difference by averaging the first predicted scores of the student model based on the target label sequence, or by weighting the contribution of the target label sequence to the student model and the first predicted scores of the student model; wherein the contribution of the target label sequence to the student model is positively correlated with the second predicted score of the target label sequence.

11. An electronic device, characterized in that, The method includes a memory and a processor coupled to each other, the memory storing program instructions and the processor executing the program instructions to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The system stores program instructions that can be executed by a processor, the program instructions being used to implement the method according to any one of claims 1 to 9.

Images