Knowledge distillation-based text classification method, apparatus and device, and medium

Abstract

The invention discloses a text classification method and device based on knowledge distillation, equipment and a medium, and relates to the field of natural language processing, and the method comprises the steps: inputting a preset data set into a pre-training text classification model to obtain a first output result, and inputting the first output result into a preset convolutional neural network model to obtain a second output result; the pre-training text classification model is a model obtained after fine tuning of a pre-training model; determining a cross entropy of the first output result and the second output result, determining a target cross entropy loss according to the cross entropy, and determining a corresponding distillation loss; and training a preset convolutional neural network model by using a preset data set based on the target cross entropy loss and the distillation loss to obtain a target text classification model, and deploying the target text classification model to a general graphics processor so as to process a text classification task by using the general graphics processor. A lightweight large model is distilled through knowledge and is deployed to a general graphic processor, so that good calculation performance can be ensured while the size and complexity of the model are reduced.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a text classification method, device, equipment and medium based on knowledge distillation. Background Art

[0002] In recent years, with the introduction of the Transformer model, which can solve the problem of long-distance dependency by calculating the mutual influence between words and has excellent parallel computing capabilities, many researchers are committed to further improving the performance of the model and applying it to various machine learning tasks, but this also leads to more parameters and calculations in the model, limiting the application of the model on devices with limited computing resources. In addition, since machine learning tasks often involve a large number of parallel and vectorized operations, the traditional CPU (Central Processing Unit) has gradually been unable to meet the computing power requirements of machine learning tasks. General-purpose graphics processing units (GPGPU) are processors that use the characteristics of processor multi-core structure, multi-threading and high memory access bandwidth to handle high-performance computing tasks. They have more computing units and higher bandwidth to perform parallel and vectorized operations. At present, they have gradually replaced CPUs and become the mainstream high-performance computing processors for machine learning. However, although the traditional Transformer model performs well in many natural language processing tasks, it is difficult to deploy it directly on devices with limited resources due to its large number of parameters and computing requirements. Therefore, how to reduce the size and complexity of large models so that they can be deployed on devices such as GPGPU is an urgent problem to be solved in this field. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a text classification method, device, equipment and medium based on knowledge distillation, which can reduce the size and complexity of the model while ensuring good computing performance by lightweight large models through knowledge distillation and deploying them to general-purpose graphics processors. The specific scheme is as follows:

[0004] In a first aspect, the present application provides a text classification method based on knowledge distillation, comprising:

[0005] Inputting a preset data set into a pre-trained text classification model to obtain a corresponding first output result, and inputting the preset data set into a preset convolutional neural network model to obtain a corresponding second output result; the pre-trained text classification model is a model obtained by fine-tuning the pre-trained model based on a preset text classification task;

[0006] Determine a first cross entropy between the first output result and the second output result, determine a target cross entropy loss according to the first cross entropy, and determine a corresponding distillation loss according to the target cross entropy loss;

[0007] The preset convolutional neural network model is trained using the preset data set based on the target cross entropy loss and the distillation loss to obtain a target text classification model, and the target text classification model is deployed to a general-purpose graphics processor so as to use the general-purpose graphics processor to process the preset text classification task.

[0008] Optionally, before inputting the preset data set into the pre-trained text classification model to obtain the corresponding first output result, the method further includes:

[0009] The pre-trained model is obtained by pre-training an initial pre-trained language model based on a model pre-training task, and the pre-trained model is fine-tuned based on the preset text classification task to obtain the pre-trained text classification model; the model pre-training task includes a masked language model and next sentence prediction.

[0010] Optionally, inputting the preset data set into the pre-trained text classification model to obtain the corresponding first output result includes:

[0011] Inputting the preset data set into the pre-trained text classification model to obtain a first initial output result of the pre-trained text classification model, and generating a corresponding soft label according to the first initial output result using a preset normalization function at a first preset temperature, so as to obtain a corresponding first output result according to the soft label;

[0012] Correspondingly, the inputting the preset data set into the preset convolutional neural network model to obtain the corresponding second output result includes:

[0013] The preset data set is input into the preset convolutional neural network model to obtain a second initial output result of the preset convolutional neural network model, and the preset normalization function is used at the first preset temperature to generate the corresponding second output result according to the second initial output result.

[0014] Optionally, determining a first cross entropy between the first output result and the second output result, and determining a target cross entropy loss according to the first cross entropy includes:

[0015] Determine the first cross entropy of the first output result and the second output result, and use the first cross entropy as a first partial cross entropy loss;

[0016] Determine a second cross entropy between the second initial output result and the preset hard label using the preset normalization function at a second preset temperature, and use the second cross entropy as a second partial cross entropy loss;

[0017] The target cross entropy loss is determined according to the first partial cross entropy loss and the second partial cross entropy loss.

[0018] Optionally, determining the corresponding distillation loss according to the target cross entropy loss includes:

[0019] Based on the first preset weight corresponding to the first part of the cross entropy loss and the second preset weight corresponding to the second part of the cross entropy loss, the corresponding distillation loss is determined according to the target cross entropy loss; the sum of the first preset weight and the second preset weight is 1.

[0020] Optionally, deploying the target text classification model to a general-purpose graphics processor so as to use the general-purpose graphics processor to process the preset text classification task includes:

[0021] The target text classification model is deployed as a computing unit to the general-purpose graphics processor, so that after the general-purpose graphics processor receives the preset text classification task, it caches the text data of the preset text classification task to the global memory, and uses the instructions in the preset instruction set to forward the text data in the global memory to the computing unit for processing to complete the preset text classification task.

[0022] Optionally, the process of the computing unit processing the text data includes:

[0023] Convert the text data into a text data matrix, and use its own convolution kernel to extract the eigenvalues ​​of the text data matrix according to the model parameters of the target text classification model;

[0024] After the corresponding feature map is obtained according to the feature value, the feature map is selected and filtered to obtain a target feature vector, and the classification result obtained by classifying the target feature vector is returned to the global memory.

[0025] In a second aspect, the present application provides a text classification device based on knowledge distillation, comprising:

[0026] A result generation module is used to input a preset data set into a pre-trained text classification model to obtain a corresponding first output result, and input the preset data set into a preset convolutional neural network model to obtain a corresponding second output result; the pre-trained text classification model is a model obtained by fine-tuning the pre-trained model based on a preset text classification task;

[0027] a loss determination module, configured to determine a first cross entropy between the first output result and the second output result, determine a target cross entropy loss according to the first cross entropy, and determine a corresponding distillation loss according to the target cross entropy loss;

[0028] A model training module is used to train the preset convolutional neural network model based on the cross entropy loss and the distillation loss using the preset data set to obtain a target text classification model, and deploy the target text classification model to a general-purpose graphics processor so as to use the general-purpose graphics processor to process the preset text classification task.

[0029] In a third aspect, the present application provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned text classification method based on knowledge distillation.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned text classification method based on knowledge distillation.

[0031] In the present application, a preset data set can be input into a pre-trained text classification model after fine-tuning the pre-trained model based on a preset text classification task to obtain a corresponding first output result, and the preset data set can be input into a preset convolutional neural network model to obtain a corresponding second output result, and then the first cross entropy between the first output result and the second output result is determined, and the target cross entropy loss is determined based on the first cross entropy, and the corresponding distillation loss is determined based on the target cross entropy loss. Thereafter, the preset convolutional neural network model is trained using the preset data set based on the target cross entropy loss and the distillation loss to obtain a target text classification model, and the target text classification model is deployed to a general-purpose graphics processor so that the preset text classification task can be processed using the general-purpose graphics processor. In this way, the present application can be based on the knowledge distillation method, based on the pre-trained large model and the small model based on the convolutional neural network to perform distillation to distill the large model into a natural language processing model based on the convolutional neural network, so that the small model has the same generalization ability as the large model, and the large model is lightweight. The lightweight small model is then deployed to a general-purpose graphics processor with limited resources. The general-purpose graphics processor uses the lightweight model to process the corresponding text classification task, thereby greatly reducing the size and complexity of the model through the knowledge distillation method. The obtained small model is easier to implement in hardware and deployed to the computing core of a general-purpose graphics processor with limited resources. While reducing the size and complexity of the model by lightweighting the large model, the model can also ensure good computing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0033] Figure 1 A flow chart of a text classification method based on knowledge distillation provided for this application;

[0034] Figure 2 A schematic diagram of a knowledge distillation algorithm provided for this application;

[0035] Figure 3 A schematic diagram of a general graphics processor architecture provided for this application;

[0036] Figure 4 A specific flow chart of a text classification method based on knowledge distillation provided for this application;

[0037] Figure 5 A schematic diagram of a Transformer computing unit provided for this application;

[0038] Figure 6 A schematic diagram of the structure of a text classification device based on knowledge distillation provided in this application;

[0039] Figure 7 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] As model performance improves, traditional CPUs have gradually been unable to meet the computing power requirements of machine learning tasks, and general-purpose graphics processors need to gradually replace CPUs as computing processors for machine learning. However, due to the huge number of parameters and computing requirements, traditional Transformer models are difficult to deploy directly on devices with limited resources. The present application can distill the large model into a natural language processing model based on a convolutional neural network based on the knowledge distillation method, so that the small model has the same generalization ability as the large model, and then deploy the lightweight small model to a general-purpose graphics processor with limited resources. This can reduce the model size and complexity by lightweighting the large model while ensuring the computing performance of the model.

[0042] See also Figure 1 As shown, the embodiment of the present invention discloses a text classification method based on knowledge distillation, including:

[0043] Step S11, inputting a preset data set into a pre-trained text classification model to obtain a corresponding first output result, and inputting the preset data set into a preset convolutional neural network model to obtain a corresponding second output result; the pre-trained text classification model is a model obtained by fine-tuning the pre-trained model based on a preset text classification task.

[0044] It should be pointed out that in this embodiment, the Transformer large model can be distilled to other natural language processing models by adopting two-stage distillation based on the knowledge distillation algorithm, so that the small model has the same generalization ability as the large model, and the large model is lightweight. Then, the lightweight small model BERT-CNN is instantiated as a Transformer computing unit and deployed to the GPGPU with limited resources, which can greatly reduce the number of model parameters. Therefore, before inputting the preset data set into the pre-trained text classification model to obtain the corresponding first output result, this embodiment can first pre-train the initial pre-trained language model based on the model pre-training task to obtain a pre-trained model, and fine-tune the pre-trained model based on the preset text classification task to obtain a pre-trained text classification model; wherein the above-mentioned model pre-training tasks include masked language model (MLM, Masked Language Model) and next sentence prediction (NSP, Next Sentence Prediction). That is to say, the large model in this embodiment uses a pre-trained Transformer-based BERT model, and a two-stage training method can be adopted to implement two pre-training tasks, namely, masked language model and next sentence prediction, in the pre-training stage, and then fine-tune the model parameters to adapt to different downstream tasks. In this embodiment, you can choose to fine-tune the text task to obtain a complete BERT model suitable for text classification.

[0045] Correspondingly, the small model in this embodiment is based on the CNN (Convolutional Neural Networks) convolutional neural network to implement the text classification task. The model is divided into four layers: the first layer is the embedding layer, which is used to map words into word vectors when text is input, that is, to convert the text into a corresponding text matrix, that is, a two-dimensional array; the second layer is the convolution layer. In this embodiment, since the text convolution only needs to be performed in one direction of the text sequence, different convolution kernels are used to use one-dimensional convolution to extract the feature values ​​of the word vector to obtain different feature maps; the third layer is the pooling layer. In the convolution layer, due to the different convolution kernels, feature maps of different dimensions will be generated, so each different feature map obtained by the convolution layer can be uniformly pooled; the fourth layer is the fully connected layer, which is responsible for classifying the feature vectors obtained in the previous layer.

[0046] Next, the preset data set can be input into the pre-trained text classification model to obtain the corresponding first output result, and the preset data set can be input into the preset convolutional neural network model to obtain the corresponding second output result. Specifically, the preset data set can be input into the pre-trained text classification model to obtain the first initial output result of the pre-trained text classification model, and the corresponding soft label can be generated according to the first initial output result using the preset normalization function at the first preset temperature, so as to obtain the corresponding first output result according to the soft label, and then the preset data set can be input into the preset convolutional neural network model to obtain the second initial output result of the preset convolutional neural network model, and the corresponding second output result can be generated according to the second initial output result using the preset normalization function at the first preset temperature. That is to say, after obtaining the pre-trained large model, the data set with labels in the preset data set can be input into the BERT model, and the output B of the BERT model can be converted into the BERT model. out As "knowledge", it is distilled, that is, a normalized function is used at temperature T to generate soft labels to obtain the first output result of the large model; at the same time, the same label data set is input into the small model based on CNN and its output S out A normalization operation is performed at the same temperature T to obtain a second output result of the small model.

[0047] Step S12: determine a first cross entropy between the first output result and the second output result, determine a target cross entropy loss according to the first cross entropy, and determine a corresponding distillation loss according to the target cross entropy loss.

[0048] In this embodiment, the first cross entropy between the first output result and the second output result can be determined, and the first cross entropy is used as the first partial cross entropy loss, and then the second cross entropy between the second initial output result and the preset hard label is determined using a preset normalization function at a second preset temperature, and the second cross entropy is used as the second partial cross entropy loss, and then the target cross entropy loss is determined according to the first partial cross entropy loss and the second partial cross entropy loss. After the cross entropy loss is determined, the corresponding distillation loss is determined according to the target cross entropy loss based on the first preset weight corresponding to the first partial cross entropy loss and the second preset weight corresponding to the second partial cross entropy loss; wherein the sum of the first preset weight and the second preset weight is 1.

[0049] That is to say, Figure 2 As shown, in this embodiment, the cross entropy of the calculation results of the BERT model and CNN can be recorded as part of the distillation loss as L CE-Soft It should be noted that, since the large model has more training data than the small model during the training process, it may still make mistakes. Therefore, in order to prevent the small model from learning these mistakes, the hard label, that is, the probability distribution of the real data, can be appropriately added in this embodiment, and this part of the distillation loss is recorded as the distillation loss of the hard label L CE-Hard , and with L CE-Soft The calculation process is similar, the output S of the small model out The result of normalization operation under the condition of temperature T=1 is obtained by cross entropy with the hard label (real data probability distribution) to obtain the other part of the distillation loss, which is recorded as L CE-Hard , from which the total distillation loss L can be obtained KD The formula for calculating distillation loss is as follows:

[0050] L KD =αL CE-Soft +(1-α)L CE-Hard ;

[0051] Among them, α is a preset weight that can be adjusted according to actual conditions to balance the impact of the two parts of the cross entropy loss on the prediction results.

[0052] In this way, in this embodiment, the pre-trained BERT model can be fine-tuned as a large model, and the knowledge distillation method can be used to distill it to the small CNN model, that is, the learning results of the large model are used as the training target of the small model. Through this form of distillation, the small model not only learns the correct classification results (by the cross entropy loss L CE The guidance is used to measure the difference between the output of the small model and the soft label output by the large model), and the confidence of the large model for different categories is also learned (by the distillation loss L KDGuide), so that the model can be lightweight while maintaining high performance, thereby greatly reducing the number of model parameters and improving the model running speed.

[0053] Step S13: Based on the target cross entropy loss and the distillation loss, the preset convolutional neural network model is trained using the preset data set to obtain a target text classification model, and the target text classification model is deployed to a general-purpose graphics processor so as to use the general-purpose graphics processor to process the preset text classification task.

[0054] In this embodiment, the preset convolutional neural network model can be trained based on the target cross entropy loss and distillation loss using a preset data set to obtain a target text classification model, and the target text classification model can be deployed to a general-purpose graphics processor so that the general-purpose graphics processor can be used to process the preset text classification task. That is to say, after the above knowledge distillation process, the small model BERT-CNN obtained has the same generalization ability as the large model, and can then be deployed to the GPGPU as a Transformer computing unit. And a specific architecture of the above general-purpose graphics processor is as follows Figure 3 shown.

[0055] In this embodiment, the preset data set can be input into a pre-trained text classification model after fine-tuning the pre-trained model based on a preset text classification task to obtain a corresponding first output result, and the preset data set can be input into a preset convolutional neural network model to obtain a corresponding second output result, and then the first cross entropy between the first output result and the second output result is determined, and the target cross entropy loss is determined based on the first cross entropy, and the corresponding distillation loss is determined based on the target cross entropy loss. Thereafter, the preset convolutional neural network model is trained using the preset data set based on the target cross entropy loss and the distillation loss to obtain a target text classification model, and the target text classification model is deployed to a general-purpose graphics processor so that the general-purpose graphics processor can be used to process the preset text classification task. In this way, based on the knowledge distillation method, the pre-trained large model and the small model based on the convolutional neural network can be distilled to the natural language processing model based on the convolutional neural network, so that the small model has the same generalization ability as the large model, and the large model is lightweight. Then, the lightweight small model is deployed to a general-purpose graphics processor with limited resources. The lightweight model is used by the general-purpose graphics processor to process the corresponding text classification task, thereby greatly reducing the size and complexity of the model through the knowledge distillation method. The obtained small model is easier to implement in hardware and deployed to the computing core of a general-purpose graphics processor with limited resources. While reducing the size and complexity of the model by lightweighting the large model, the model can also ensure good computing performance.

[0056] Based on the previous embodiment, it can be seen that the present application can distill the large model into a natural language processing model based on a convolutional neural network based on the knowledge distillation method. Next, this embodiment will describe in detail the process of processing the text classification task based on the lightweight model. Figure 4 As shown, the embodiment of the present application discloses a specific text classification method based on knowledge distillation, including:

[0057] Step S21: Based on the target cross entropy loss and distillation loss, a preset convolutional neural network model is trained using a preset data set to obtain a target text classification model.

[0058] Step S22: deploying the target text classification model to a general-purpose graphics processor so as to utilize the general-purpose graphics processor to process a preset text classification task.

[0059] In this embodiment, the target text classification model can be deployed as a computing unit to a general-purpose graphics processor, so that after the general-purpose graphics processor receives the preset text classification task, the text data of the preset text classification task is cached in the global memory, and the text data in the global memory is forwarded to the computing unit for processing using the instructions in the preset instruction set to complete the preset text classification task. And in the process of processing text data by the computing unit, the text data can be converted into a text data matrix, and the eigenvalues ​​of the text data matrix can be extracted according to the model parameters of the target text classification model using its own convolution kernel, and then after obtaining the corresponding feature map according to the eigenvalue, the feature map is selected and filtered to obtain the target feature vector, and the classification result obtained by classifying the target feature vector is returned to the global memory.

[0060] It should be pointed out that if Figure 5As shown, the computing unit in this embodiment is composed of a shaping module, a convolution module, a pooling module, and a fully connected module. When GPGPU processes a text classification task, the text data can be first cached to the global memory, and the computing task can be assigned to the computing core. Then, according to the task type, the customized text classification instruction TFVV based on the RISC-V (RISC, Reduced Instruction Set Computer) instruction set is issued to the instruction cache I-Cache. Then, the GPGPU execution pipeline reads the TFVV instruction in the I-Cache and reads the model parameters from the register file according to the parsed instruction information. At the same time, data is read from the data cache D-Cache based on the RISC-V native instruction LW. If there is a miss, the text data is read from the global memory and forwarded to the Transformer computing unit. After that, the computing unit can be activated, in which the shaping module is responsible for the format conversion of the text data. When the input is text data, it is converted into a text data matrix, that is, a two-dimensional array, and then the vector data is forwarded to the convolution module via the AXI-ST (AXI, Advanced Extensible Interface) data bus. The convolution module uses the convolution kernel to extract feature values ​​of the two-dimensional data according to the model parameters. Since the text convolution only needs to be convolved in one direction of the text sequence, one-dimensional convolution is used in this embodiment to extract feature values ​​to obtain different feature maps. After that, the feature map data output by the convolution module is selected and filtered through the pooling module. Finally, the feature vector can be classified and output through the fully connected module, and the classification label operation result is returned to the global memory.

[0061] For a more specific processing procedure of the above step S21, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.

[0062] Through the above technical solution, in this embodiment, the small model is allowed to learn the "knowledge" of the large model, so that the small model has the same generalization ability as the large model. After the large model is lightweight, the lightweight small model BERT-CNN can be instantiated as a Transformer computing unit. The computing unit constructs a shaping module, a convolution module, a pooling module, and a fully connected module according to the training parameters and number of layers of the BERT-CNN model, and processes the corresponding text classification tasks. In this way, the size and complexity of the model are greatly reduced by the knowledge distillation method in this embodiment. Compared with the traditional Transformer model, although it performs well in many natural language processing tasks, it is difficult to directly deploy it on a resource-limited device due to its huge number of parameters and computing requirements. Under the premise of ensuring good performance of the model, the obtained small model is easier to implement in hardware, and the lightweight Transformer model is deployed as a computing unit in a GPGPU computing core with limited resources.

[0063] See also Figure 6 As shown, the embodiment of the present application also discloses a text classification device based on knowledge distillation, including:

[0064] A result generation module 11 is used to input a preset data set into a pre-trained text classification model to obtain a corresponding first output result, and input the preset data set into a preset convolutional neural network model to obtain a corresponding second output result; the pre-trained text classification model is a model obtained by fine-tuning the pre-trained model based on a preset text classification task;

[0065] A loss determination module 12, configured to determine a first cross entropy between the first output result and the second output result, determine a target cross entropy loss according to the first cross entropy, and determine a corresponding distillation loss according to the target cross entropy loss;

[0066] The model training module 13 is used to train the preset convolutional neural network model based on the cross entropy loss and the distillation loss using the preset data set to obtain a target text classification model, and deploy the target text classification model to a general-purpose graphics processor so as to use the general-purpose graphics processor to process the preset text classification task.

[0067] In this embodiment, a preset data set can be input into a pre-trained text classification model after fine-tuning the pre-trained model based on a preset text classification task to obtain a corresponding first output result, and the preset data set can be input into a preset convolutional neural network model to obtain a corresponding second output result, and then a first cross entropy between the first output result and the second output result is determined, and a target cross entropy loss is determined based on the first cross entropy, and a corresponding distillation loss is determined based on the target cross entropy loss, and then the preset convolutional neural network model is trained using the preset data set based on the target cross entropy loss and the distillation loss to obtain a target text classification model, and the target text classification model is deployed to a general-purpose graphics processor so that the general-purpose graphics processor can be used to process the preset text classification task. Through the above technical scheme, based on the knowledge distillation method, the pre-trained large model and the small model based on the convolutional neural network can be used for distillation to distill the large model into a natural language processing model based on the convolutional neural network. Then, the lightweight small model can be deployed to a general-purpose graphics processor with limited resources. The lightweight model is used by the general-purpose graphics processor to process the corresponding text classification task. In this way, the small model obtained in this embodiment is easier to implement in hardware and deployed to the computing core of a general-purpose graphics processor with limited resources. While reducing the size and complexity of the model through the knowledge distillation lightweight large model, the model can also ensure good computing performance.

[0068] In some specific embodiments, the text classification device based on knowledge distillation further includes:

[0069] A model pre-training module is used to pre-train the initial pre-trained language model based on the model pre-training task to obtain the pre-trained model, and fine-tune the pre-trained model based on the preset text classification task to obtain the pre-trained text classification model; the model pre-training task includes a masked language model and next sentence prediction.

[0070] In some specific embodiments, the result generation module 11 specifically includes:

[0071] A first result generating unit, configured to input the preset data set into the pre-trained text classification model to obtain a first initial output result of the pre-trained text classification model, and generate a corresponding soft label according to the first initial output result using a preset normalization function at a first preset temperature, so as to obtain a corresponding first output result according to the soft label;

[0072] A second result generating unit is used to input the preset data set into the preset convolutional neural network model to obtain a second initial output result of the preset convolutional neural network model, and generate a corresponding second output result according to the second initial output result using the preset normalization function at the first preset temperature.

[0073] In some specific embodiments, the loss determination module 12 specifically includes:

[0074] A first loss determining unit, configured to determine the first cross entropy between the first output result and the second output result, and use the first cross entropy as a first partial cross entropy loss;

[0075] A second loss determination unit, configured to determine a second cross entropy between the second initial output result and a preset hard label using the preset normalization function at a second preset temperature, and use the second cross entropy as a second partial cross entropy loss;

[0076] The third loss determination unit is used to determine the target cross entropy loss according to the first partial cross entropy loss and the second partial cross entropy loss.

[0077] In some specific embodiments, the loss determination module 12 specifically includes:

[0078] A fourth loss determination unit is used to determine the corresponding distillation loss according to the target cross entropy loss based on the first preset weight corresponding to the first part of the cross entropy loss and the second preset weight corresponding to the second part of the cross entropy loss; the sum of the first preset weight and the second preset weight is 1.

[0079] In some specific embodiments, the model training module 13 specifically includes:

[0080] A model deployment unit is used to deploy the target text classification model as a computing unit to the general-purpose graphics processor, so that after the general-purpose graphics processor receives the preset text classification task, it caches the text data of the preset text classification task to the global memory, and uses the instructions in the preset instruction set to forward the text data in the global memory to the computing unit for processing to complete the preset text classification task.

[0081] In some specific embodiments, the model training module 13 specifically includes:

[0082] A data conversion unit, used for converting the text data into a text data matrix, and extracting eigenvalues ​​of the text data matrix according to model parameters of the target text classification model using its own convolution kernel;

[0083] The vector classification unit is used to obtain a corresponding feature map according to the feature value, select and filter the feature map to obtain a target feature vector, and return the classification result obtained by classifying the target feature vector to the global memory.

[0084] Furthermore, the present application also discloses an electronic device. Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0085] Figure 7 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the text classification method based on knowledge distillation disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0086] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0087] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0088] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the text classification method based on knowledge distillation performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0089] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the text classification method based on knowledge distillation disclosed above is implemented. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0090] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0091] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0092] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0093] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0094] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A text classification method based on knowledge distillation, characterized in that: include: Inputting a preset data set into a pre-trained text classification model to obtain a corresponding first output result, and inputting the preset data set into a preset convolutional neural network model to obtain a corresponding second output result; The pre-trained text classification model is a model obtained by fine-tuning the pre-trained model based on a preset text classification task; Determine a first cross entropy between the first output result and the second output result, determine a target cross entropy loss according to the first cross entropy, and determine a corresponding distillation loss according to the target cross entropy loss; The preset convolutional neural network model is trained using the preset data set based on the target cross entropy loss and the distillation loss to obtain a target text classification model, and the target text classification model is deployed to a general-purpose graphics processor so as to use the general-purpose graphics processor to process the preset text classification task.

2. The text classification method based on knowledge distillation according to claim 1 is characterized in that: Before inputting the preset data set into the pre-trained text classification model to obtain the corresponding first output result, the method further includes: The pre-trained model is obtained by pre-training an initial pre-trained language model based on a model pre-training task, and the pre-trained model is fine-tuned based on the preset text classification task to obtain the pre-trained text classification model; the model pre-training task includes a masked language model and next sentence prediction.

3. The text classification method based on knowledge distillation according to claim 1 is characterized in that: The step of inputting the preset data set into the pre-trained text classification model to obtain a corresponding first output result includes: Inputting the preset data set into the pre-trained text classification model to obtain a first initial output result of the pre-trained text classification model, and generating a corresponding soft label according to the first initial output result using a preset normalization function at a first preset temperature, so as to obtain a corresponding first output result according to the soft label; Correspondingly, the inputting the preset data set into the preset convolutional neural network model to obtain the corresponding second output result includes: The preset data set is input into the preset convolutional neural network model to obtain a second initial output result of the preset convolutional neural network model, and the preset normalization function is used at the first preset temperature to generate the corresponding second output result according to the second initial output result.

4. The text classification method based on knowledge distillation according to claim 3 is characterized in that: The determining a first cross entropy between the first output result and the second output result, and determining a target cross entropy loss according to the first cross entropy, includes: Determine the first cross entropy of the first output result and the second output result, and use the first cross entropy as a first partial cross entropy loss; Determine a second cross entropy between the second initial output result and the preset hard label using the preset normalization function at a second preset temperature, and use the second cross entropy as a second partial cross entropy loss; The target cross entropy loss is determined according to the first partial cross entropy loss and the second partial cross entropy loss.

5. The text classification method based on knowledge distillation according to claim 4 is characterized in that: Determining the corresponding distillation loss according to the target cross entropy loss includes: Based on the first preset weight corresponding to the first part of the cross entropy loss and the second preset weight corresponding to the second part of the cross entropy loss, the corresponding distillation loss is determined according to the target cross entropy loss; the sum of the first preset weight and the second preset weight is 1.

6. The text classification method based on knowledge distillation according to any one of claims 1 to 5, characterized in that: The step of deploying the target text classification model to a general-purpose graphics processor so as to use the general-purpose graphics processor to process the preset text classification task includes: The target text classification model is deployed as a computing unit to the general-purpose graphics processor, so that after the general-purpose graphics processor receives the preset text classification task, it caches the text data of the preset text classification task to the global memory, and uses the instructions in the preset instruction set to forward the text data in the global memory to the computing unit for processing to complete the preset text classification task.

7. The text classification method based on knowledge distillation according to claim 6, characterized in that: The process of the computing unit processing the text data includes: Convert the text data into a text data matrix, and use its own convolution kernel to extract the eigenvalues ​​of the text data matrix according to the model parameters of the target text classification model; After the corresponding feature map is obtained according to the feature value, the feature map is selected and filtered to obtain a target feature vector, and the classification result obtained by classifying the target feature vector is returned to the global memory.

8. A text classification device based on knowledge distillation, characterized in that: include: A result generation module, used for inputting a preset data set into a pre-trained text classification model to obtain a corresponding first output result, and inputting the preset data set into a preset convolutional neural network model to obtain a corresponding second output result; The pre-trained text classification model is a model obtained by fine-tuning the pre-trained model based on a preset text classification task; a loss determination module, configured to determine a first cross entropy between the first output result and the second output result, determine a target cross entropy loss according to the first cross entropy, and determine a corresponding distillation loss according to the target cross entropy loss; A model training module is used to train the preset convolutional neural network model based on the cross entropy loss and the distillation loss using the preset data set to obtain a target text classification model, and deploy the target text classification model to a general-purpose graphics processor so as to use the general-purpose graphics processor to process the preset text classification task.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the text classification method based on knowledge distillation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the text classification method based on knowledge distillation as described in any one of claims 1 to 7.

Images