Model training method and device based on knowledge distillation

Abstract

The present invention discloses a model training method and device based on knowledge distillation, which relates to the field of knowledge distillation technology, and the main purpose is to improve the prediction accuracy of the student model. The main technical scheme of the present invention is: using a first data set to train a first model; based on data enhancement technology, the first data set is expanded to obtain a second data set; using the first model, the first data set and the second data set to train the second model, and determine the loss function of the second model. The present invention is used to train a high-precision student model.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge distillation, and in particular to a model training method and device based on knowledge distillation. Background Art

[0002] The concept of knowledge distillation was first proposed in the article "Distilling the Knowledge in a Neural Network". By introducing a teacher network to induce the training of the student network, knowledge transfer is achieved. Therefore, knowledge distillation is to transfer the knowledge of one network to another network. The two networks can be isomorphic or heterogeneous. The method is to first train a teacher model, and then use the output of this teacher model and the real label of the data to train the student model. Knowledge distillation can be used to transform the network from a large network into a small network while retaining performance close to that of the large network, thereby solving the problem of insufficient hardware for deploying the model at the edge segment.

[0003] However, in the process of knowledge distillation, since the structure of the student model is simpler, the accuracy of the student model can only be as close to the accuracy of the teacher model as possible. This means that in actual applications, the accuracy of the student model obtained based on knowledge distillation cannot meet application requirements. Summary of the invention

[0004] In view of the above problems, the present invention proposes a model training method and device based on knowledge distillation, the main purpose of which is to improve the prediction accuracy of the student model.

[0005] In order to achieve the above object, the present invention mainly provides the following technical solutions:

[0006] In a first aspect, the present invention provides a model training method based on knowledge distillation, comprising:

[0007] Training a first model using a first data set;

[0008] Expanding the first data set based on data enhancement technology to obtain a second data set;

[0009] A second model is trained using the first model, the first data set, and the second data set, and a loss function of the second model is determined.

[0010] Preferably, using the first model, the first data set, and the second data set to train a second model and determine a loss function of the second model includes:

[0011] Using the logical output of the first model to train the second model to obtain a first loss function;

[0012] Using the first data set and the second data set to train the second model, obtaining a second loss function;

[0013] The loss function of the second model is determined according to the first loss function and the second loss function.

[0014] Preferably, the second model is trained using the logical output of the first model to obtain a first loss function, including:

[0015] Input the same sample data into the first model and the second model respectively to obtain a first logic output and a second logic output;

[0016] The first loss function is determined based on a mean square error between the first logic output and the second logic output.

[0017] Preferably, the second model is trained using the first data set and the second data set to obtain a second loss function, including:

[0018] Using the second model to predict the data in the first data set and the second data set, and determine the predicted labeling information of the data;

[0019] The second loss function is determined according to the cross entropy between the predicted labeling information of the data and the labeling information carried by the data.

[0020] Preferably, determining the loss function of the second model according to the first loss function and the second loss function includes:

[0021] Obtaining the ratio of the first loss function to the second loss function;

[0022] The first loss function and the second loss function are weightedly summed according to the proportion coefficient to determine the loss function of the second model.

[0023] Preferably, before using the first data set to train the first model, the method further includes:

[0024] A first model is constructed, wherein the model structure of the first model contains the model structure of the second model.

[0025] Preferably, the second model adopts a TextCNN model, and constructing the first model includes:

[0026] The BERT model and the TextCNN model are combined to construct the first model, wherein the output of the BERT model is used as the input of the TextCNN model, the input of the first model is the input of the BERT model, and the output of the first model is the output of the TextCNN model.

[0027] Preferably, the data in the first data set are sentences carrying annotation information, and the first data set is expanded based on the data enhancement technology to obtain the second data set, including:

[0028] Processing the sentences in the first data set using the BERT model to generate extended sentences with similar meanings to the sentences;

[0029] The annotation information carried by the sentence is correspondingly annotated on the extended sentence to obtain the second data set.

[0030] Preferably, the method further comprises:

[0031] The second model is used to predict the data in the target data set.

[0032] In a second aspect, the present invention provides a model training device based on knowledge distillation, the device comprising:

[0033] A training unit, configured to train a first model using a first data set;

[0034] An expansion unit, configured to expand the first data set based on a data enhancement technology to obtain a second data set;

[0035] A determination unit is used to train a second model using the first model obtained by the training unit, the first data set and the second data set obtained by the expansion unit, and determine the loss function of the second model.

[0036] Preferably, the determining unit includes:

[0037] A first training module, used for training the second model using the logical output of the first model to obtain a first loss function;

[0038] A second training module, used for training the second model using the first data set and the second data set to obtain a second loss function;

[0039] A determination module is used to determine the loss function of the second model based on the first loss function obtained by the first training module and the second loss function obtained by the second training module.

[0040] Preferably, the first training module is specifically used for:

[0041] Input the same sample data into the first model and the second model respectively to obtain a first logic output and a second logic output;

[0042] The first loss function is determined based on a mean square error between the first logic output and the second logic output.

[0043] Preferably, the second training module is specifically used for:

[0044] Using the second model to predict the data in the first data set and the second data set, and determine the predicted labeling information of the data;

[0045] The second loss function is determined according to the cross entropy between the predicted labeling information of the data and the labeling information carried by the data.

[0046] Preferably, the determining module is specifically used for:

[0047] Obtaining the ratio of the first loss function to the second loss function;

[0048] The first loss function and the second loss function are weightedly summed according to the proportion coefficient to determine the loss function of the second model.

[0049] Preferably, the device further comprises:

[0050] The construction unit is used to construct the first model before the training unit trains the first model using the first data set, wherein the model structure of the first model contains the model structure of the second model.

[0051] Preferably, the second model adopts a TextCNN model, and the construction unit is specifically used to combine a BERT model with a TextCNN model to construct the first model, wherein the output of the BERT model serves as the input of the TextCNN model, the input of the first model is the input of the BERT model, and the output of the first model is the output of the TextCNN model.

[0052] Preferably, the expansion unit comprises:

[0053] A generation module, configured to process the sentences in the first data set using a BERT model to generate extended sentences with similar meanings to the sentences;

[0054] The annotation module is used to annotate the extended sentence obtained by the generation module with the annotation information carried by the sentence to obtain the second data set.

[0055] Preferably, the device further comprises:

[0056] A prediction unit is used to predict the data in the target data set by using the second model.

[0057] On the other hand, the present invention also provides a processor, which is used to run a program, wherein the program executes the model training method based on knowledge distillation of the above-mentioned first aspect when running.

[0058] On the other hand, the present invention also provides a storage medium, which is used to store a computer program, wherein when the computer program is running, it controls the device where the storage medium is located to execute the model training method based on knowledge distillation of the above-mentioned first aspect.

[0059] By means of the above technical scheme, the present invention provides a model training method and device based on knowledge distillation, wherein the first model is used as a teacher model and the second model is used as a student model. In the process of knowledge distillation, the first data set for training the first model is expanded, and the second model is trained in combination with the expanded second data set to determine the loss function of the second model, so that the second model can be trained with more data to achieve the purpose of improving the prediction accuracy of the model. Compared with the existing knowledge distillation method, the present invention uses the expanded second data set only for training the second model, and does not use the second data set for training the first model, so that when determining the loss function of the second function, the first model trained by the first data set is trained together with the second data set, and the difference between the two can avoid the overfitting problem caused by data approximation when the second model learns the first model. At the same time, the expanded second data set also provides a large number of training samples for the second model, and also makes the prediction accuracy of the second model more similar to the first model.

[0060] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0062] Figure 1 A flow chart of a model training method based on knowledge distillation proposed in an embodiment of the present invention is shown;

[0063] Figure 2 A flowchart of another model training method based on knowledge distillation proposed in an embodiment of the present invention is shown;

[0064] Figure 3 The model structure of the first model in the embodiment of the present invention is shown;

[0065] Figure 4A flowchart of training a second model based on knowledge distillation proposed in an embodiment of the present invention is shown;

[0066] Figure 5 A schematic diagram of the structure of a model training device based on knowledge distillation proposed in an embodiment of the present invention is shown;

[0067] Figure 6 A schematic diagram of the structure of another model training device based on knowledge distillation proposed in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0068] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0069] The embodiment of the present invention provides a model training method based on knowledge distillation. Compared with the existing knowledge distillation, this method can improve the training effect of the student model while avoiding overfitting during the knowledge distillation process, so that the accuracy of the student model is closer to or reaches the effect of the teacher model. The specific execution steps are as follows: Figure 1 As shown, including:

[0070] 101. Train a first model using a first data set.

[0071] This step uses the trained first model as the teacher model in the knowledge distillation process, and the first data set is a training sample set carrying labeled information.

[0072] The embodiment of the present invention does not specifically limit the structure of the first model, and the required model can be selected according to business requirements.

[0073] 102. Expand the first data set based on data enhancement technology to obtain a second data set.

[0074] In this step, the purpose of data enhancement is to obtain samples that are similar to the data samples in the first data set in order to obtain a sufficient number of data samples.

[0075] Specifically, this step can be based on modifying the data samples in the first data set to obtain similar data samples, or it can be based on obtaining data samples through other channels and then finding data samples similar to the data samples in the first data set from these data samples. This embodiment is not limited to this.

[0076] In this embodiment, the data samples in the second data set obtained after expansion do not participate in the training of the first model, but are only used for the training of the second model, so as to increase the difference between the training of the first model and the second model.

[0077] 103. Train the second model using the first model, the first data set, and the second data set, and determine a loss function of the second model.

[0078] The second model in this step refers to the student model in the knowledge distillation process. In the existing knowledge distillation process, the process of training the student model is also to use the hard labels in the first data set (i.e., the labels or annotation information carried by the data samples) to train the teacher model, and then use the soft labels obtained in the teacher model (i.e., the output of the softmax layer) combined with the hard labels in the first data set to jointly determine the loss function of the student model. The main feature that distinguishes the embodiment of the present invention from the existing knowledge distillation process is that the second model is trained using the second data set, which increases the sample base of the second model. At the same time, since the first model is not trained using the second data set, the distribution of the soft labels obtained will be significantly different, and this difference can prevent the second model from overfitting relative to the first model during the knowledge distillation process.

[0079] Based on the above Figure 1 It can be seen from the implementation method that the model training method based on knowledge distillation proposed in the embodiment of the present invention uses data enhancement technology to expand the first data set, and through the differentiated application of the first data set and the second data set, prevents the second model from overfitting and causing a decrease in accuracy during the knowledge distillation process. By increasing the number of samples used to train the second model, the similarity between the second model and the first model can be further improved, thereby improving the prediction accuracy of the second model.

[0080] Further, a preferred embodiment of the present invention is in the above Figure 1 On this basis, each step is optimized to ensure that the prediction accuracy of the second model can be closer to the prediction accuracy of the first model after knowledge distillation.

[0081] Specifically, the embodiment of the present invention takes the text classification model as an example to explain the proposed knowledge distillation process in detail, wherein the second model (student model) adopts the TextCNN model, and its specific steps are as follows: Figure 2 As shown, including:

[0082] 201. Build the first model.

[0083] In this embodiment, the model structure of the first model needs to be determined according to the model structure of the second model, that is, the model structure of the first model must contain the model structure of the second model.

[0084] The combination of the two models' structure needs to be determined according to the specific model structure. The purpose is that the two models are similar in structure, which can achieve better results in knowledge distillation.

[0085] Specifically in this embodiment, since the second model is a TextCNN model, the first model is a model built on the basis of the BERT model. The structure of the first model is as follows: Figure 3 As shown in the figure, E1, E2, ..., En are input word vectors, Trm is the transformer structure in the BERT model, and the above two parts represent the BERT model. The logical output of the last softmax layer of the BERT model is passed to TextCNN, which outputs the results T1, T2, ..., Tn. It can be seen that the first model in this embodiment is to connect the BERT model and the TextCNN model in series, and use the output of the BERT model as the input of the TextCNN model, so that the input of the first model is the input of the BERT model, and the output of the first model is the output of the TextCNN model.

[0086] 202. Train a first model using a first data set.

[0087] In this embodiment, the data samples in the first data set are sentences carrying annotation information, where common annotation information in text classification application scenarios includes sentiment identifiers, intent identifiers, etc., which are not specifically limited in this embodiment.

[0088] 203. Expand the first data set based on data enhancement technology to obtain a second data set.

[0089] Specifically, in this embodiment, the BERT model is used to process the sentences in the first data set to generate extended sentences with similar meanings to the sentences. The BERT model uses supervised training to achieve enhanced extension of input sentences, that is, Seq2Seq generation tasks are used to obtain extended sentences similar to the input sentences. The number of extended sentences generated for a sentence can be customized according to needs.

[0090] After the extended sentence is generated, the annotation information carried by the input sentence is labeled on the extended sentence, and the extended sentence with the annotation information is added to the second data set, so that a large number of new training samples with annotation information can be obtained.

[0091] 204. Train the second model using the logical output of the first model to obtain a first loss function.

[0092] Specifically, the same sample data is input into the first model and the second model respectively to obtain the first logic output and the second logic output, wherein the sample data is taken from the first data set.

[0093] Afterwards, a first loss function is determined based on a mean square error between the first logic output and the second logic output.

[0094] The first loss function can be expressed as:

[0095]

[0096] Among them, L DS represents the first loss function, z (T) 、z (S) Represents the logical output of the first model (teacher model) and the second model (student model).

[0097] 205. Train a second model using the first data set and the second data set to obtain a second loss function.

[0098] Specifically, the second model is used to predict the data in the first data set and the second data set to determine the predicted annotation information of the data, that is, the second model is trained using the data samples in the data set to obtain the prediction results of the second model.

[0099] Afterwards, the second loss function is determined based on the cross entropy between the predicted labeling information of the data and the labeling information carried by the data. That is, the cross entropy between the actual labeling information of the data sample and the predicted result predicted by the second model is calculated, and the second loss function is calculated using the cross entropy.

[0100] The second loss function can be expressed as:

[0101]

[0102] Among them, L CE represents the second loss function, t i represents the annotation information actually carried by the i-th data sample, y i It represents the prediction result of the i-th data sample after the second model, s represents the second model (student model), and N represents the number of data samples.

[0103] It should be noted that, in practical applications, the data samples in the second data set are mainly used to train the second model and determine the second loss function. In addition, there is no logical sequence relationship between this step and step 204.

[0104] 206. Determine a loss function of the second model according to the first loss function and the second loss function.

[0105] Specifically, the proportion coefficients of the first loss function and the second loss function are obtained. In this embodiment, since the loss function is composed of the first loss function and the second loss function, by determining one of the proportion coefficients (α), the other proportion coefficient (ie, 1-α) can be determined.

[0106] Afterwards, the first loss function and the second loss function are weightedly summed according to the proportion coefficient to determine the loss function of the second model.

[0107] The loss function of the second model can be specifically expressed as follows in combination with the above description:

[0108] L=α·L CE +(1-α)L DS

[0109] In summary, the model training process based on knowledge distillation proposed in the embodiment of the present invention can be achieved through Figure 4 express, Figure 4 A flowchart of training the second model based on knowledge distillation is shown. According to the training process, the loss function of the second model can be determined, and then the second model can be applied to predict the data in the target data set, which is the input data after the second model is deployed on the device side.

[0110] Furthermore, as a response to the above Figure 1 , 2 The implementation of the method embodiment shown in the figure, the embodiment of the present invention provides a model training device based on knowledge distillation, which is used to improve the prediction accuracy of the student model. The embodiment of the device corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will no longer repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. Specifically, Figure 5 As shown, the device comprises:

[0111] A training unit 31, configured to train a first model using a first data set;

[0112] An expansion unit 32, configured to expand the first data set based on a data enhancement technology to obtain a second data set;

[0113] The determination unit 33 is used to train a second model using the first model obtained by the training unit 31, the first data set and the second data set obtained by the expansion unit 32, and determine the loss function of the second model.

[0114] Further, such as Figure 6 As shown, the determining unit 33 includes:

[0115] A first training module 331 is used to train the second model using the logical output of the first model to obtain a first loss function;

[0116] A second training module 332, configured to train the second model using the first data set and the second data set to obtain a second loss function;

[0117] The determination module 333 is used to determine the loss function of the second model according to the first loss function obtained by the first training module 331 and the second loss function obtained by the second training module 332.

[0118] Furthermore, the first training module 331 is specifically used for:

[0119] Input the same sample data into the first model and the second model respectively to obtain a first logic output and a second logic output;

[0120] The first loss function is determined based on a mean square error between the first logic output and the second logic output.

[0121] Furthermore, the second training module 332 is specifically used for:

[0122] Using the second model to predict the data in the first data set and the second data set, and determine the predicted labeling information of the data;

[0123] The second loss function is determined according to the cross entropy between the predicted labeling information of the data and the labeling information carried by the data.

[0124] Furthermore, the determining module 333 is specifically used for:

[0125] Obtaining the ratio of the first loss function to the second loss function;

[0126] The first loss function and the second loss function are weightedly summed according to the proportion coefficient to determine the loss function of the second model.

[0127] Further, such as Figure 6 As shown, the device also includes:

[0128] The construction unit 34 is used to construct the first model before the training unit 31 trains the first model using the first data set, wherein the model structure of the first model contains the model structure of the second model.

[0129] Furthermore, the second model adopts the TextCNN model, and the construction unit 34 is specifically used to combine the BERT model and the TextCNN model to construct the first model, wherein the output of the BERT model is used as the input of the TextCNN model, the input of the first model is the input of the BERT model, and the output of the first model is the output of the TextCNN model.

[0130] Further, such as Figure 6 As shown, the expansion unit 32 includes:

[0131] A generating module 321, configured to process the sentences in the first data set using a BERT model to generate extended sentences with similar meanings to the sentences;

[0132] The marking module 322 is used to mark the extended sentence obtained by the generating module 321 with the marking information carried by the sentence to obtain the second data set.

[0133] Further, such as Figure 6 As shown, the device also includes:

[0134] The prediction unit 35 is used to predict the data in the target data set by using the second model.

[0135] Furthermore, an embodiment of the present invention further provides a processor, the processor is used to run a program, wherein the program executes the above Figure 1-2 The model training method based on knowledge distillation described in .

[0136] Furthermore, an embodiment of the present invention further provides a storage medium, wherein the storage medium is used to store a computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the above Figure 1-2 The model training method based on knowledge distillation described in .

[0137] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] It is understandable that the related features in the above methods and devices can be referenced to each other. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, but do not represent the advantages and disadvantages of the embodiments.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0141] In addition, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0143] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0146] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0147] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0148] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0149] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0150] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0151] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A model training method based on knowledge distillation, the method comprising: Constructing a first model, wherein the model structure of the first model contains the model structure of the second model, wherein constructing the first model comprises: combining a BERT model with a TextCNN model to construct the first model, wherein the output of the BERT model is used as the input of the TextCNN model, the input of the first model is the input of the BERT model, and the output of the first model is the output of the TextCNN model; Training a first model using a first data set, where data samples in the first data set are text sentences carrying annotation information; Expanding the first data set based on data enhancement technology to obtain a second data set; The second model is trained using the first model, the first data set and the second data set to determine the loss function of the second model, including: training the second model using the logical output of the first model to obtain a first loss function; training the second model using the first data set and the second data set to obtain a second loss function; and determining the loss function of the second model based on the first loss function and the second loss function.

2. The method according to claim 1, characterized in that The second model is trained using the logical output of the first model to obtain a first loss function, including: Input the same sample data into the first model and the second model respectively to obtain a first logic output and a second logic output; The first loss function is determined based on a mean square error between the first logic output and the second logic output.

3. The method according to claim 1, characterized in that The second model is trained using the first data set and the second data set to obtain a second loss function, including: Using the second model to predict the data in the first data set and the second data set, and determine the predicted labeling information of the data; The second loss function is determined according to the cross entropy between the predicted labeling information of the data and the labeling information carried by the data.

4. The method according to claim 1, characterized in that: Determining the loss function of the second model according to the first loss function and the second loss function includes: Obtaining the ratio of the first loss function to the second loss function; The first loss function and the second loss function are weightedly summed according to the proportion coefficient to determine the loss function of the second model.

5. A model training device based on knowledge distillation, the device comprising: A construction unit, configured to construct a first model, wherein the model structure of the first model contains the model structure of the second model, and the construction of the first model comprises: combining a BERT model with a TextCNN model to construct the first model, wherein the output of the BERT model is used as the input of the TextCNN model, the input of the first model is the input of the BERT model, and the output of the first model is the output of the TextCNN model; A training unit, configured to train a first model using a first data set, wherein data samples in the first data set are text sentences carrying annotation information; An expansion unit, configured to expand the first data set based on a data enhancement technology to obtain a second data set; A determination unit is used to train a second model using the first model obtained by the training unit, the first data set and the second data set obtained by the expansion unit, and determine the loss function of the second model, including: training the second model using the logical output of the first model to obtain a first loss function; training the second model using the first data set and the second data set to obtain a second loss function; determining the loss function of the second model according to the first loss function and the second loss function.

6. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes the model training method based on knowledge distillation described in any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium is used to store a computer program, wherein when the computer program is running, it controls the device where the storage medium is located to execute the model training method based on knowledge distillation according to any one of claims 1 to 4.

Images