Text processing method and device, equipment and medium

The text processing method using knowledge distillation trains a target text model with a simple structure to enhance both accuracy and efficiency in neural network-based text processing.

CN119067170APending Publication Date: 2024-12-03BEIJING XIAOMI MOBILE SOFTWARE CO LTD +1

Patent Information

Application Number
CN202310640343.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Increasing the structural complexity of neural network models to improve text processing accuracy leads to reduced efficiency.

Method used

A text processing method utilizing knowledge distillation to train a target text model by sampling the model structure of a first text model and adjusting parameters based on loss function values, allowing for a simple model structure with high performance.

Benefits of technology

Improves both the accuracy and efficiency of text processing by maintaining a lightweight model structure while achieving high performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067170A_ABST
    Figure CN119067170A_ABST
Patent Text Reader

Abstract

The invention relates to a text processing method and device, equipment and a medium. The method comprises the steps of obtaining a to-be-processed text; inputting the to-be-processed text into the target text model to obtain a text processing result; wherein the target text model is obtained by the following steps of: obtaining a sample text marked with a label; determining a to-be-trained first text model, and sampling a model structure of the first text model to obtain a second text model; according to the sample text, the first text model serves as a teacher model, the second text model serves as a student model, a model training process based on knowledge distillation is executed, a target text model is obtained, and the target text model is the trained second text model. According to the technical scheme, the target text model has high processing performance while having a simple model structure, so that during text processing, on one hand, the text processing accuracy can be improved, and on the other hand, the text processing efficiency can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of text processing technology, and in particular to a text processing method, device, equipment and medium. Background Art

[0002] With the continuous development of artificial intelligence, more and more neural network models are being used in the field of text processing. When processing text based on a neural network model, if you want to improve the accuracy of the text processing results, you need to improve the performance of the neural network model by increasing the structural complexity of the neural network model. However, the increase in the structural complexity of the neural network model will reduce the efficiency of the neural network model in processing text. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a text processing method, device, equipment and medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a text processing method is provided, the method comprising:

[0005] Get the text to be processed;

[0006] Inputting the text to be processed into the target text model to obtain a text processing result;

[0007] The target text model is obtained in the following way:

[0008] Get sample text annotated with labels;

[0009] Determine a first text model to be trained, and sample the model structure of the first text model to obtain a second text model;

[0010] According to the sample text, the first text model is used as a teacher model, and the second text model is used as a student model, and a model training process based on knowledge distillation is performed to obtain the target text model, wherein the target text model is the trained second text model.

[0011] Optionally, the step of taking the first text model as a teacher model and the second text model as a student model according to the sample text, performing a model training process based on knowledge distillation to obtain the target text model includes:

[0012] Inputting the sample text into the first text model and the second text model respectively, to obtain a first output result of the first text model and a second output result of the second text model;

[0013] Determine a first loss function value according to the first output result and the label, determine a second loss function value according to the second output result and the label, and determine a third loss function value according to the first output result and the second output result;

[0014] According to the first loss function value, the second loss function value and the third loss function value, the parameters of the first text model and the second text model are adjusted, and the second text model after the parameters are adjusted is determined as the target text model.

[0015] Optionally, determining a model performance gap value between the first text model and the second text model according to the first loss function value and the second loss function value;

[0016] The adjusting the parameters of the first text model according to the first loss function value, the second loss function value and the third loss function value comprises:

[0017] When the model performance gap value is less than a preset model performance gap threshold, the parameters of the first text model are adjusted according to the first loss function value, the second loss function value and the third loss function value.

[0018] Optionally, the method further comprises:

[0019] When the model performance gap value is greater than a preset model performance gap threshold, the gradient of the first text model is set to a preset threshold to adjust the parameters of the first text model according to the first loss function value and the second loss function value.

[0020] Optionally, when there are multiple second text models, the first text model is used as a teacher model and the second text model is used as a student model according to the sample text, and a model training process based on knowledge distillation is performed to obtain the target text model, including:

[0021] Inputting the sample text into the first text model and the second text model respectively, to obtain a first output result of the first text model and a second output result of the second text model;

[0022] Determine a first loss function value according to the first output result and the label, determine a second loss function value according to the second output result and the label, determine a third loss function value according to the first output result and the second output result, and determine a fourth loss function value according to the second output results of every two second text models;

[0023] According to the first loss function value, the second loss function value and the third loss function value, the parameters of the first text model are adjusted, and according to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, the parameters of the second text model are adjusted, and the second text model after the adjusted parameters is determined as the target text model.

[0024] Optionally, adjusting the parameters of the second text model according to the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value includes:

[0025] Determine the two second text models corresponding to the fourth loss function value as target text models;

[0026] According to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, the parameters of the target text model are adjusted, and according to the first loss function value, the second loss function value and the third loss function value, the parameters of other second text models except the target text model are adjusted.

[0027] Optionally, sampling the model structure of the first text model to obtain the second text model includes:

[0028] The model structure of the first text model is sampled by any of the following methods:

[0029] Randomly sampling the encoder structure of the first text model and randomly sampling the decoder structure of the first text model to obtain a second text model;

[0030] Randomly sampling the encoder structure of the first text model to obtain a target encoder structure, and using the decoder structure of the first text model as a target decoder structure, combining the target encoder structure and the target decoder structure to obtain a second text model;

[0031] The decoder structure of the first text model is randomly sampled to obtain a target decoder structure, and the encoder structure of the first text model is used as a target encoder structure. The target encoder structure and the target decoder structure are combined to obtain a second text model.

[0032] According to a second aspect of an embodiment of the present disclosure, there is provided a text processing device, including:

[0033] A first acquisition module is configured to acquire the text to be processed;

[0034] A processing module is configured to input the text to be processed into a target text model to obtain a text processing result;

[0035] The text processing device also includes:

[0036] A second acquisition module is configured to acquire sample text annotated with a label;

[0037] A sampling module is configured to determine a first text model to be trained, and sample the model structure of the first text model to obtain a second text model;

[0038] The training module is configured to perform a model training process based on knowledge distillation based on the sample text, using the first text model as a teacher model and the second text model as a student model to obtain the target text model, wherein the target text model is the trained second text model.

[0039] Optionally, the training module includes:

[0040] A first input unit is configured to input the sample text into the first text model and the second text model respectively, to obtain a first output result of the first text model and a second output result of the second text model;

[0041] A first determining unit is configured to determine a first loss function value according to the first output result and the label, determine a second loss function value according to the second output result and the label, and determine a third loss function value according to the first output result and the second output result;

[0042] The first adjustment unit is configured to adjust the parameters of the first text model and the second text model according to the first loss function value, the second loss function value and the third loss function value, and determine the second text model after the adjustment of the parameters as the target text model.

[0043] Optionally, the device further comprises:

[0044] A determination module, configured to determine a model performance gap value between the first text model and the second text model according to the first loss function value and the second loss function value;

[0045] Correspondingly, the first adjustment unit is further configured to adjust the parameters of the first text model according to the first loss function value, the second loss function value and the third loss function value when the model performance gap value is less than a preset model performance gap threshold.

[0046] Optionally, the device further comprises:

[0047] The second adjustment unit is configured to set the gradient of the first text model to a preset threshold when the model performance gap value is greater than a preset model performance gap threshold, so as to adjust the parameters of the first text model according to the first loss function value and the second loss function value.

[0048] Optionally, when there are multiple second text models, the training module includes:

[0049] A second input unit is configured to input the sample text into the first text model and the second text model respectively, to obtain a first output result of the first text model and a second output result of the second text model;

[0050] A second determining unit is configured to determine a first loss function value according to the first output result and the label, determine a second loss function value according to the second output result and the label, determine a third loss function value according to the first output result and the second output result, and determine a fourth loss function value according to the second output results of every two second text models;

[0051] The third adjustment unit is configured to adjust the parameters of the first text model according to the first loss function value, the second loss function value and the third loss function value, and adjust the parameters of the second text model according to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, and determine the second text model after adjusting the parameters as the target text model.

[0052] Optionally, the third adjustment unit includes:

[0053] A determination subunit, configured to determine the two second text models corresponding to the fourth loss function value as target text models;

[0054] The adjustment subunit is configured to adjust the parameters of the target text model according to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, and to adjust the parameters of other second text models except the target text model according to the first loss function value, the second loss function value and the third loss function value.

[0055] Optionally, the sampling module includes:

[0056] A first sampling subunit is configured to randomly sample the encoder structure of the first text model and randomly sample the decoder structure of the first text model to obtain a second text model;

[0057] A second sampling subunit is configured to randomly sample the encoder structure of the first text model to obtain a target encoder structure, and use the decoder structure of the first text model as a target decoder structure, and combine the target encoder structure and the target decoder structure to obtain a second text model;

[0058] The third sampling subunit is configured to randomly sample the decoder structure of the first text model to obtain a target decoder structure, and use the encoder structure of the first text model as the target encoder structure, combine the target encoder structure and the target decoder structure to obtain a second text model.

[0059] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0060] processor;

[0061] a memory for storing processor-executable instructions;

[0062] Wherein, the processor is configured to:

[0063] Execute the steps of any one of the methods described in the first aspect of the present disclosure.

[0064] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of any one of the methods described in the first aspect of the present disclosure are implemented.

[0065] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0066] According to the above technical solution, the target text model used to process the text to be processed is trained based on the model training method of knowledge distillation, so that the target text model can have a simple model structure while having high processing performance. Therefore, when processing text, on the one hand, it can improve the accuracy of text processing, and on the other hand, it can also improve the efficiency of text processing.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0069] Figure 1 is a flowchart of a text processing method according to an exemplary embodiment;

[0070] Figure 2 is a schematic diagram showing a method of sampling a model structure of a first text model according to an exemplary embodiment;

[0071] Figure 3 is a schematic diagram of an application of a second text model according to an exemplary embodiment;

[0072] Figure 4 is a block diagram of a text processing device according to an exemplary embodiment;

[0073] Figure 5 is a block diagram of a device for text processing according to an exemplary embodiment;

[0074] Figure 6 It is a block diagram of another apparatus for text processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0075] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0076] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0077] As mentioned in the background technology, with the continuous development of artificial intelligence, more and more neural network models are being used in the field of text processing. When processing text based on a neural network model, if you want to improve the accuracy of the text processing results, you need to improve the performance of the neural network model by increasing the structural complexity of the neural network model. However, the increase in the structural complexity of the neural network model will reduce the efficiency of the neural network model in processing text.

[0078] In view of this, the embodiments of the present disclosure provide a text processing method, apparatus, device and medium, wherein a target text model for processing a text to be processed is trained based on a model training method of knowledge distillation, so that the target text model can have a simple model structure while having high processing performance. Therefore, when processing text, on the one hand, the accuracy of text processing can be improved, and on the other hand, the efficiency of text processing can be improved.

[0079] The embodiments of the present disclosure are further explained below with reference to the accompanying drawings.

[0080] Figure 1 is a flowchart of a text processing method according to an exemplary embodiment. Figure 1 As shown, the text processing method may include steps S101-S102.

[0081] In step S101, the text to be processed is obtained.

[0082] The text to be processed may be a training corpus consisting of several text pairs, and each text pair includes a source language text and its corresponding target language text. The text to be processed may also be a training corpus consisting of several text pairs, and each text pair includes a text and a summary corresponding to the text. The text to be processed may also be a training corpus consisting of several text pairs, and each text pair includes a prompt text and a story text generated according to the prompt text.

[0083] It should be understood that this is only an illustrative description and does not constitute a limitation to the solution. In actual use, the text to be processed can be set according to actual conditions.

[0084] In step S102, the text to be processed is input into a target text model to obtain a text processing result. The target text model is obtained in the following manner:

[0085] Obtain sample text annotated with labels; determine a first text model to be trained, and sample the model structure of the first text model to obtain a second text model; according to the sample text, use the first text model as a teacher model and the second text model as a student model, perform a model training process based on knowledge distillation, and obtain the target text model, wherein the target text model is the trained second text model.

[0086] It should be understood that the text processing result is determined based on the text to be processed, and the embodiments of the present disclosure do not impose any restrictions on this. For example, in a machine translation scenario, the text processing result is to generate a target language text for the text to be processed, that is, by inputting the source language text to be processed into the target text model to obtain the target language text corresponding to the source language text. In a text summary generation scenario, the text processing result is to generate a summary of the text to be processed, that is, by inputting the text to be processed into the target text model to obtain a summary corresponding to the text. In a story generation scenario, the text processing result is to generate a story text for the text to be processed, that is, by inputting the prompt text to be processed into the target text model to obtain the story text corresponding to the prompt text.

[0087] It should also be understood that the sample text annotated with labels can also be set according to the actual application scenario, and the embodiments of the present disclosure do not impose any restrictions on this. For example, in a machine translation scenario, the sample text annotated with labels can be a source language text, and the label is used to characterize the text in the target language corresponding to the source language text. In a text summary generation scenario, the sample text annotated with labels can be a text to be processed, and the label is used to characterize the target summary corresponding to the text to be processed. In a story generation scenario, the sample text annotated with labels can be a prompt text, and the label is used to characterize the story text corresponding to the prompt text.

[0088] It should also be understood that the first text model can be a neural network model in the relevant technology, such as a recurrent neural network model, a convolutional neural network or a Transformer model, or a neural network model that is an improvement on the neural network model in the relevant technology, and the embodiments of the present disclosure do not impose any restrictions on this. It should also be understood that no matter what kind of neural network model the first text model is, its basic architecture belongs to the encoder-decoder architecture. Therefore, when sampling the model structure of the first text model to obtain the second text model, the encoder structure and / or decoder structure of the first text model can be randomly sampled to obtain the target encoder structure and the target decoder structure, and then the target encoder structure and the target decoder structure obtained by sampling are combined to obtain the second text model. That is, according to one embodiment of the present disclosure, the sampling of the model structure of the first text model to obtain the second text model may include:

[0089] The model structure of the first text model is sampled by any of the following methods:

[0090] The encoder structure of the first text model is randomly sampled, and the decoder structure of the first text model is randomly sampled to obtain a second text model; the encoder structure of the first text model is randomly sampled to obtain a target encoder structure, and the decoder structure of the first text model is used as the target decoder structure, and the target encoder structure and the target decoder structure are combined to obtain a second text model; the decoder structure of the first text model is randomly sampled to obtain a target decoder structure, and the encoder structure of the first text model is used as the target encoder structure, and the target encoder structure and the target decoder structure are combined to obtain a second text model.

[0091] For example, the first text model is set to a Transformer model, such as Figure 2 As shown, the Transformer model includes 6 encoders and 6 decoders, where the 6 encoders correspond to Figure 2The EL-1, EL-2, EL-3, EL-4, EL-5 and EL-6 decoders in Figure 2 DL-1, DL-2, DL-3, DL-4, DL-5 and DL-6 in . When sampling the model structure of the Transformer model to obtain the second text model, the encoder structure of the Transformer model can be randomly sampled to obtain the target encoder structure. For example, the target encoder structure obtained by random sampling can include EL-1, EL-2, EL-3 and EL-4; then the decoder structure of the Transformer model is randomly sampled to obtain the target decoder structure. For example, the target decoder structure obtained by random sampling can include DL-3, DL-4 and DL-5; finally, the target encoder structure and the target decoder structure are combined to obtain the second text model, such as Figure 2 The second text model -1 shown. The encoder structure of the Transformer model can also be randomly sampled to obtain a target encoder structure. For example, the target encoder structure obtained by random sampling may include EL-2, EL-3, EL-4, and EL-5; then the decoder structure of the Transformer model is used as the target decoder structure, that is, the target decoder structure may include DL-1, DL-2, DL-3, DL-4, DL-5, and DL-6; finally, the target encoder structure and the target decoder structure are combined to obtain a second text model, such as Figure 2 The second text model -2 shown. The encoder structure of the Transformer model can also be used as the target encoder structure, that is, the target encoder structure can include EL-1, EL-2, EL-3, EL-4, EL-5 and EL-6; then the decoder structure of the Transformer model is randomly sampled to obtain the target decoder structure. For example, the target decoder structure obtained by random sampling can include DL-1, DL-2, DL-3 and DL-4, and finally the target encoder structure and the target decoder structure are combined to obtain the second text model, as shown in FIG. Figure 2 The second text model shown is -3.

[0092] In this embodiment, the first text model can be sampled in different sampling modes to obtain second text models of different sizes and structures, thereby meeting the operating requirements of different devices. Figure 3 As shown, when the model structure of the second text model obtained by sampling is small, it can be deployed on a watch or bracelet; when the model structure of the second text model obtained by sampling is moderate, it can be deployed on a mobile phone or tablet; when the model structure of the second text model obtained by sampling is large, it can be deployed on a computer.

[0093] It should be understood that this is only an illustrative description and does not constitute a limitation to the solution.

[0094] In a possible implementation, the step of using the first text model as a teacher model and the second text model as a student model according to the sample text to perform a model training process based on knowledge distillation to obtain a target text model for processing the text may include:

[0095] Input the sample text into the first text model and the second text model respectively to obtain a first output result of the first text model and a second output result of the second text model; determine a first loss function value according to the first output result and the label, determine a second loss function value according to the second output result and the label, and determine a third loss function value according to the first output result and the second output result; adjust the parameters of the first text model and the second text model according to the first loss function value, the second loss function value and the third loss function value, and determine the second text model after the adjusted parameters as the target student model.

[0096] Among them, according to the first output result and the label, the first loss function value can be calculated based on the mean squared error loss function (Mean Squared Error), the hinge loss function (hinge loss) or the cross entropy loss function (CrossEntropyLoss), and the embodiment of the present disclosure does not impose any restrictions on this. In a possible implementation, the first loss function value is determined based on the cross entropy loss function, that is, by substituting the first output result and the label into the following formula to obtain the first loss function value:

[0097]

[0098] in, Represents the first loss function value, y i Represents the label value of the i-th sample text, y i ′ Represents the i-th first output result.

[0099] Similarly, according to the second output result and the label, the second loss function value can be determined by referring to the method for determining the first loss function value, which will not be elaborated in the embodiment of the present disclosure.

[0100] It should be understood that the principle of knowledge distillation is to transfer the knowledge of the teacher model to the student model, so that the student model has good performance while ensuring lightweight. Therefore, in the process of knowledge distillation, that is, in the process of model training, it is necessary to make the output of the student model as close as possible to the output of the teacher model. Therefore, in one possible implementation, the KL divergence (Kullback–Leibler divergence) can be used to measure the output difference between the teacher model and the student model, that is, the third loss function value can be determined based on the KL divergence. That is, by substituting the first output result and the second output result into the following formula to obtain the third loss function value:

[0101]

[0102] in, Represents the third loss function value, P T Represents the first output result, p s Indicates the second output result.

[0103] After obtaining the first loss function value, the second loss function value and the third loss function value, the first loss function value, the second loss function value and the third loss function value can be added to obtain the total loss function value in the model training process, and then the parameters of the first text model and the second text model can be adjusted according to the total loss function value. That is, the total loss function value can be expressed as:

[0104]

[0105] in, represents the total loss function value, represents the first loss function value, K represents the total number of the second text models, represents the k-th second loss function value, that is, the second loss function value corresponding to the k-th second text model. It represents the k-th third loss function value, that is, the third loss function value corresponding to the k-th second text model, and α represents the coefficient.

[0106] In addition, it should be understood that, since the second text model is obtained by sampling the model structure of the first text model, there are differences in the model structure between the second text model and the first text model, and the performance gap between the second text model and the first text model is also different according to the differences in the model structure. When the performance gap between the second text model and the first text model is too large, adjusting the parameters of the first text model according to the first loss function value, the second loss function value and the third loss function value will have a negative impact on the first text model, that is, during the training process, the performance of the first text model will be reduced. Therefore, in a possible implementation, the model performance gap value between the second text model and the first text model can also be detected in real time, and when the model performance gap value is less than a preset model performance gap threshold, the parameters of the first text model are adjusted according to the first loss function value, the second loss function value and the third loss function value. That is, according to an embodiment of the present disclosure, the model training method can also include:

[0107] Determining a model performance gap value between the first text model and the second text model according to the first loss function value and the second loss function value;

[0108] Accordingly, adjusting the parameters of the first text model according to the first loss function value, the second loss function value, and the third loss function value may include:

[0109] When the model performance gap value is less than a preset model performance gap threshold, the parameters of the first text model are adjusted according to the first loss function value, the second loss function value and the third loss function value.

[0110] When the model performance gap is greater than a preset model performance gap threshold, the inventors have found through multiple trainings that the gradient of the third loss function with respect to the first text model can be dynamically adjusted so that the third loss function does not affect the first text model during the training process. The gradient of the third loss function with respect to the first text model can be determined by obtaining the derivative of the third loss function with respect to the first text model parameters.

[0111] It should be understood that the gradient is a vector, and its mathematical meaning is the rate of change of the function f(x, y) at the differentiable point. The larger the gradient, the greater the rate of change of the function f(x, y) at the differentiable point. Therefore, by setting the gradient of the third loss function with respect to the first text model to zero, the parameters of the first text model are adjusted only according to the first loss function value and the second loss function value during the model training process. That is, according to one embodiment of the present disclosure, the model training method may further include:

[0112] When the model performance gap value is greater than a preset model performance gap threshold, the gradient of the first text model is set to a preset threshold to adjust the parameters of the first text model according to the first loss function value and the second loss function value.

[0113] In addition, the inventors also found that when the first text model is sampled multiple times to obtain multiple second text models, and the multiple second text models and the first text model are trained at the same time, if the second text models are allowed to learn from each other during the training process, the performance of the second text model can be further improved. Therefore, in a possible implementation, in addition to allowing the second text model to learn the first text model, the second text models can also be allowed to learn from each other to improve the performance of the second text model. In the mutual learning process of the second text model, the knowledge of the second text model with high performance is transferred to the second text model with low performance to improve the performance of the second text model with low performance. Therefore, during the training process, the fourth loss function value can also be determined according to the output of each two second text models, and then the parameters of the corresponding second text model can be adjusted according to the fourth loss function value, so that the output of the second text model with low model performance is as close as possible to the output of the second text model with high model performance. That is, according to an embodiment of the present disclosure, when there are multiple second text models, the first text model is used as a teacher model according to the sample text, and the second text model is used as a student model to perform a model training process based on knowledge distillation to obtain a target student model for processing text, which may include:

[0114] Input the sample text into the first text model and the second text model respectively to obtain a first output result of the first text model and a second output result of the second text model; determine a first loss function value according to the first output result and the label, determine a second loss function value according to the second output result and the label, determine a third loss function value according to the first output result and the second output result, and determine a fourth loss function value according to every two second output results of the second text models; adjust the parameters of the first text model according to the first loss function value, the second loss function value and the third loss function value, and adjust the parameters of the second text model according to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, and determine the second text model after the adjusted parameters as the target student model.

[0115] Among them, the method for determining the first loss function value and the second loss function value refers to the method for determining the first loss function value mentioned above, and the method for determining the third loss function value and the fourth loss function value refers to the method for determining the third loss function value mentioned above, which will not be repeated in the embodiments of the present disclosure.

[0116] After obtaining the first loss function value, the second loss function value and the third loss function value, the first loss function value, the second loss function value and the third loss function value can be added to obtain the total loss function value of the first text model during the training process, and then the parameters of the first text model can be adjusted according to the total loss function value of the first text model. That is, the total loss function value of the first text model can be expressed as:

[0117]

[0118] in, represents the total loss function value of the first text model, represents the first loss function value, K represents the total number of the second text models, represents the k-th second loss function value, that is, the second loss function value corresponding to the k-th second text model. It represents the k-th third loss function value, that is, the third loss function value corresponding to the k-th second text model, and α represents the coefficient.

[0119] Similarly, after obtaining the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, the first loss function value, the second loss function value, the third loss function value and the fourth loss function value can be added to obtain the total loss function value of the second text model during the model training process, and then the parameters of the second text model can be adjusted according to the total loss function value of the second text model. That is, the total loss function value of the second text model can be expressed as:

[0120]

[0121]

[0122]

[0123] in, Represents the total loss function value of the second text model, represents the first loss function value, represents the second loss function value corresponding to the k-th second text model, represents the third loss function value corresponding to the kth second text model, α and β represent coefficients, L kl represents the fourth loss function value corresponding to the kth second text model, K represents the total number of second text models, Indicates the kth ′ The second loss function value corresponding to the second text model, represents the second output result of the k-th second text model, Indicates the kth′ The second output result of the second text model, It means that the second output result of the second text model with low model performance is made close to the second output result of the second text model with high model performance.

[0124] In a possible implementation, adjusting the parameters of the second text model according to the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value may include:

[0125] The two second text models corresponding to the fourth loss function value are determined as target text models; the parameters of the target text model are adjusted according to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, and the parameters of other second text models except the target text model are adjusted according to the first loss function value, the second loss function value and the third loss function value.

[0126] For example, if the second text model includes a second text model-1, a second text model-2 and a second text model-3, and the fourth loss function value is determined by the outputs of the second text model-1 and the second text model-3, the second text model-1 and the second text model-3 are used as target text models, and the first loss function value, the second loss function value, the third loss function value and the fourth loss function value are used to adjust the parameters of the second text model-1 and the second text model-3, and the first loss function value, the second loss function value and the third loss function value are used to adjust the parameters of the second text model-2.

[0127] In summary, through the above-mentioned text processing method, since the target text model is trained based on the model training method of knowledge distillation, the target text model can have a simple model structure and high processing performance at the same time, and then when processing text, on the one hand, the accuracy of text processing can be improved, and on the other hand, the efficiency of text processing can be improved. At the same time, since the second text model can be sampled from the first text model during the model training process, the model structure and model parameters of the second text model do not need to be stored during the model training process, which can reduce the storage space occupied. In addition, since the first text model and the second text model are trained synchronously, multiple second text models and the first text model can be trained simultaneously during one training process, so that multiple second text models can be obtained in one training, thereby improving the model training efficiency.

[0128] Based on the same concept, the present disclosure also provides a text processing device. Figure 4 As shown, the text processing device 400 may include:

[0129] A first acquisition module 401 is configured to acquire a text to be processed;

[0130] The processing module 402 is configured to input the text to be processed into a target text model to obtain a text processing result;

[0131] The text processing device 400 may further include:

[0132] A second acquisition module is configured to acquire sample text annotated with a label;

[0133] A sampling module is configured to determine a first text model to be trained, and sample the model structure of the first text model to obtain a second text model;

[0134] The training module is configured to perform a model training process based on knowledge distillation based on the sample text, using the first text model as a teacher model and the second text model as a student model to obtain the target text model, wherein the target text model is the trained second text model.

[0135] In a possible implementation, the training module may include:

[0136] A first input unit is configured to input the sample text into the first text model and the second text model respectively, to obtain a first output result of the first text model and a second output result of the second text model;

[0137] A first determining unit is configured to determine a first loss function value according to the first output result and the label, determine a second loss function value according to the second output result and the label, and determine a third loss function value according to the first output result and the second output result;

[0138] The first adjustment unit is configured to adjust the parameters of the first text model and the second text model according to the first loss function value, the second loss function value and the third loss function value, and determine the second text model after the adjustment of the parameters as the target text model.

[0139] In a possible implementation, the text processing device 400 may further include:

[0140] A determination module, configured to determine a model performance gap value between the first text model and the second text model according to the first loss function value and the second loss function value;

[0141] Correspondingly, the first adjustment unit is further configured to adjust the parameters of the first text model according to the first loss function value, the second loss function value and the third loss function value when the model performance gap value is less than a preset model performance gap threshold.

[0142] In a possible implementation, the text processing device 400 may further include:

[0143] The second adjustment unit is configured to set the gradient of the first text model to a preset threshold when the model performance gap value is greater than a preset model performance gap threshold, so as to adjust the parameters of the first text model according to the first loss function value and the second loss function value.

[0144] In a possible implementation manner, when there are multiple second text models, the training module may include:

[0145] A second input unit is configured to input the sample text into the first text model and the second text model respectively, to obtain a first output result of the first text model and a second output result of the second text model;

[0146] A second determining unit is configured to determine a first loss function value according to the first output result and the label, determine a second loss function value according to the second output result and the label, determine a third loss function value according to the first output result and the second output result, and determine a fourth loss function value according to the second output results of every two second text models;

[0147] The third adjustment unit is configured to adjust the parameters of the first text model according to the first loss function value, the second loss function value and the third loss function value, and adjust the parameters of the second text model according to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, and determine the second text model after the adjustment of the parameters as the target text model.

[0148] In a possible implementation manner, the third adjustment unit may include:

[0149] A determination subunit, configured to determine the two second text models corresponding to the fourth loss function value as target text models;

[0150] The adjustment subunit is configured to adjust the parameters of the target text model according to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, and to adjust the parameters of other second text models except the target text model according to the first loss function value, the second loss function value and the third loss function value.

[0151] In a possible implementation, the sampling module may include:

[0152] A first sampling subunit is configured to randomly sample the encoder structure of the first text model and randomly sample the decoder structure of the first text model to obtain a second text model;

[0153] A second sampling subunit is configured to randomly sample the encoder structure of the first text model to obtain a target encoder structure, and use the decoder structure of the first text model as a target decoder structure, and combine the target encoder structure and the target decoder structure to obtain a second text model;

[0154] The third sampling subunit is configured to randomly sample the decoder structure of the first text model to obtain a target decoder structure, and use the encoder structure of the first text model as the target encoder structure, combine the target encoder structure and the target decoder structure to obtain a second text model.

[0155] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0156] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the model training method or text processing method provided by the present disclosure.

[0157] Figure 5 is a block diagram of a device for text processing according to an exemplary embodiment. Figure 5 , the device 500 may include one or more of the following components: a first processing component 502 , a first memory 504 , a first power component 506 , a multimedia component 508 , an audio component 510 , a first input / output interface 512 , a sensor component 514 , and a communication component 516 .

[0158] The first processing component 502 generally controls the overall operation of the device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The first processing component 502 may include one or more first processors 520 to execute instructions to complete all or part of the steps of the above-mentioned air conditioning control method. In addition, the first processing component 502 may include one or more modules to facilitate the interaction between the first processing component 502 and other components. For example, the first processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the first processing component 502.

[0159] The first memory 504 is configured to store various types of data to support operations on the device 500. Examples of such data include instructions, messages, etc. for any application or method operating on the device 500. The first memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0160] The first power source component 506 provides power to various components of the device 500. The first power source component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 500.

[0161] The multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and / or a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user.

[0162] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the device 500 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the first memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0163] The first input / output interface 512 provides an interface between the first processing component 502 and a peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0164] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the device 500. For example, the sensor assembly 514 can detect the open / closed state of the device 500, such as the display of the device 500. The sensor assembly 514 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications.

[0165] The communication component 516 is configured to facilitate wired or wireless communication between the device 500 and other devices. The device 500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0166] In an exemplary embodiment, the device 500 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned air conditioning control method.

[0167] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of the device 500 to complete the air conditioning control method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0168] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device. The computer program has a code portion for executing the above-mentioned air conditioning control method when executed by the programmable device.

[0169] Figure 6 600 is a block diagram of a device for text processing according to an exemplary embodiment. For example, the device 600 may be provided as a server. Figure 6, the apparatus 600 may include a second processing component 622, which further includes one or more second processors, and a memory resource represented by a second memory 632 for storing instructions executable by the second processing component 622, such as an application. The application stored in the second memory 632 may include one or more modules each corresponding to a set of instructions. In addition, the second processing component 622 is configured to execute the instructions to perform the above-mentioned text processing method.

[0170] The device 600 may also include a second power supply component 626 configured to perform power management of the device 600, a wired or wireless network interface 650 configured to connect the device 600 to a network, and a second input / output interface 658. The device 600 may operate based on an operating system stored in the memory 832, such as Windows Server™, Mac OSX™, Unix™, Linux™, FreeBSD™ or the like.

[0171] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0172] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A text processing method, characterized in that: The method comprises: Get the text to be processed; Inputting the text to be processed into the target text model to obtain a text processing result; The target text model is obtained in the following way: Get sample text annotated with labels; Determine a first text model to be trained, and sample the model structure of the first text model to obtain a second text model; According to the sample text, the first text model is used as a teacher model, and the second text model is used as a student model, and a model training process based on knowledge distillation is performed to obtain the target text model, wherein the target text model is the trained second text model.

2. The method according to claim 1, characterized in that The method of performing a model training process based on knowledge distillation based on the sample text, using the first text model as a teacher model and the second text model as a student model, to obtain the target text model includes: Inputting the sample text into the first text model and the second text model respectively, to obtain a first output result of the first text model and a second output result of the second text model; Determine a first loss function value according to the first output result and the label, determine a second loss function value according to the second output result and the label, and determine a third loss function value according to the first output result and the second output result; According to the first loss function value, the second loss function value and the third loss function value, the parameters of the first text model and the second text model are adjusted, and the second text model after the parameters are adjusted is determined as the target text model.

3. The method according to claim 2, characterized in that The method further comprises: Determining a model performance gap value between the first text model and the second text model according to the first loss function value and the second loss function value; The adjusting the parameters of the first text model according to the first loss function value, the second loss function value and the third loss function value comprises: When the model performance gap value is less than a preset model performance gap threshold, the parameters of the first text model are adjusted according to the first loss function value, the second loss function value and the third loss function value.

4. The method according to claim 3, characterized in that The method further comprises: When the model performance gap value is greater than a preset model performance gap threshold, the gradient of the first text model is set to a preset threshold to adjust the parameters of the first text model according to the first loss function value and the second loss function value.

5. The method according to claim 1, characterized in that When there are multiple second text models, the first text model is used as a teacher model and the second text model is used as a student model according to the sample text, and a model training process based on knowledge distillation is performed to obtain the target text model, including: Inputting the sample text into the first text model and the second text model respectively, to obtain a first output result of the first text model and a second output result of the second text model; Determine a first loss function value according to the first output result and the label, determine a second loss function value according to the second output result and the label, determine a third loss function value according to the first output result and the second output result, and determine a fourth loss function value according to the second output results of every two second text models; According to the first loss function value, the second loss function value and the third loss function value, the parameters of the first text model are adjusted, and according to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, the parameters of the second text model are adjusted, and the second text model after the adjusted parameters is determined as the target text model.

6. The method according to claim 5, characterized in that The adjusting the parameters of the second text model according to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value comprises: Determine the two second text models corresponding to the fourth loss function value as target text models; According to the first loss function value, the second loss function value, the third loss function value and the fourth loss function value, the parameters of the target text model are adjusted, and according to the first loss function value, the second loss function value and the third loss function value, the parameters of other second text models except the target text model are adjusted.

7. The method according to any one of claims 1 to 6, characterized in that: The step of sampling the model structure of the first text model to obtain a second text model includes: The model structure of the first text model is sampled by any of the following methods: Randomly sampling the encoder structure of the first text model and randomly sampling the decoder structure of the first text model to obtain a second text model; Randomly sampling the encoder structure of the first text model to obtain a target encoder structure, and using the decoder structure of the first text model as a target decoder structure, combining the target encoder structure and the target decoder structure to obtain a second text model; The decoder structure of the first text model is randomly sampled to obtain a target decoder structure, and the encoder structure of the first text model is used as a target encoder structure. The target encoder structure and the target decoder structure are combined to obtain a second text model.

8. A text processing device, characterized in that: include: A first acquisition module is configured to acquire the text to be processed; A processing module is configured to input the text to be processed into a target text model to obtain a text processing result; The text processing device also includes: A second acquisition module is configured to acquire sample text annotated with a label; A sampling module is configured to determine a first text model to be trained, and sample the model structure of the first text model to obtain a second text model; The training module is configured to perform a model training process based on knowledge distillation based on the sample text, using the first text model as a teacher model and the second text model as a student model to obtain the target text model, wherein the target text model is the trained second text model.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Knowledge distillation method based on joint training, processor and storage medium

    CN113486185A

  • Key point detection method and device of target image, electronic equipment and storage medium

    CN113723160A

  • Knowledge distillation-based text classification method and system

    CN114818902A

  • Generalized continuous classification method based on online contrast distillation network

    CN114972839A

  • Model training method and device, electronic equipment and storage medium

    CN115049074A

Cited By

  • Document image key information extraction method and device based on knowledge distillation

    CN119810856A