A Model Compression Method and System Based on Meta-Learning
The pre-trained language model is compressed through the knowledge distillation method based on meta-learning, which solves the problem of insufficient interaction between the teacher model and the student model, improves the accuracy of text emotion classification and the deployment applicability of the model, and is suitable for embedded and mobile devices.
Patent Information
- Application Number
- CN202311119652.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-31
- Publication Date
- 2025-07-29
- Estimated Expiration
- 2043-08-31
AI Technical Summary
Existing model compression algorithms such as knowledge distillation algorithms cannot effectively adjust the interaction between teacher models and student models, resulting in poor applicability and accuracy of compressed text sentiment classification models, and the high parameter volume and high computational cost of large-scale language models limit their application in resource-constrained scenarios.
The knowledge distillation method based on meta-learning is adopted to compress the pre-trained language model Bert_base by constructing emotional classification data sets, fine-tuning, pruning and meta-learning distillation stages, and the meta-learning mechanism is used to achieve synchronous updates between teacher models and student models, improving the generalization ability of students models.
In the context of resource-constrained text sentiment analysis, more accurate sentiment classification results are achieved, and the model parameter quantity and calculation cost are reduced, making the compressed model suitable for embedded and mobile devices.
Smart Images
Figure CN117151173B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of deep learning model compression, and particularly relates to a model compression method and system based on meta-learning. Background Art
[0002] The statements in this part merely provide background technical information related to the present invention and do not necessarily constitute prior art.
[0003] Sentiment analysis, also known as opinion mining, is used to determine the sentiment tendency of text corpora. Determining the sentiment tendency has important practical value in many fields. Currently, with the continuous increase in the parameter scale of pre-trained language models, ultra-large-scale language models such as ChatGPT, GPT-3, T5, and GLM have emerged, with the number of parameters reaching millions or even billions. Large-scale language models have shown remarkable performance in text sentiment classification tasks, bringing high classification accuracy. However, due to the huge number of parameters and high computational cost of large-scale language models, the model deployment cost is getting higher and higher. Taking the GPT-175B model as an example, it still requires 320GB of storage space in the half-precision (FP16) format. In scenarios with high real-time requirements and limited resources for text sentiment analysis, directly using large-scale language models will cause application limitations.
[0004] Conducting model compression algorithms on large-scale language models can effectively reduce parameter redundancy, thereby reducing storage occupancy, communication bandwidth, and computational complexity, and contributing to the application deployment of deep learning. However, existing model compression algorithms all have certain problems. Taking the knowledge distillation algorithm as an example, existing knowledge distillation algorithms usually fix the teacher model parameters, and the student model can only passively receive knowledge from the teacher model, but the teacher model cannot interact with the student model and make adjustments according to the learning situation of the student model. At the same time, with the derivation of large-scale models, the scale gap between the teacher model and the student model is huge, resulting in poor applicability of the text sentiment classification model obtained after compression, and ultimately the accuracy of the determined sentiment classification results is not good. Summary of the Invention
[0005] To overcome the above-mentioned deficiencies of the prior art, the present invention provides a model compression method and system based on meta-learning, which can obtain more accurate sentiment classification results in scenarios of text sentiment analysis with high real-time requirements and limited resources, while reducing the number of model parameters and being more conducive to the deployment and application of the model.
[0006] To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:
[0007] The first aspect of the present invention provides a model compression method based on meta-learning, and the specific steps are as follows:
[0008] Construct a sentiment classification dataset, divide the training, test, and validation sample sets, preprocess the dataset, and adjust the input format;
[0009] Use the preprocessed data samples as input to obtain the pre-trained language model Bert_base, and use the knowledge distillation compression method based on meta-learning to achieve model compression. In the model compression inference deployment stage, output the text sentiment classification results; the knowledge distillation compression method based on meta-learning includes fine-tuning, pruning, and meta-learning distillation stages.
[0010] As a further technical solution, the preprocessing of the dataset is specifically as follows: use the SST2 sentiment classification dataset in the GLUE dataset, divide the training, test, and validation sample sets, and preprocess the data, including using a tokenizer to convert the text into word segments and annotating sentiment classification labels, and adapting to the input format.
[0011] As a further technical solution, the fine-tuning stage is specifically as follows: train the pre-trained language model based on the preprocessed sentiment classification dataset to obtain the first teacher model.
[0012] As a further technical solution, the pruning stage is specifically as follows: prune the first teacher model using a scaling factor to obtain the first student model.
[0013] As a further technical solution, the meta-learning distillation stage is specifically as follows: based on the obtained first teacher model and first student model, train the second student model based on the preprocessed sentiment classification dataset and the distillation training method based on meta-learning.
[0014] As a further technical solution, the distillation training method based on meta-learning includes:
[0015] Obtain the preprocessed SST2 training and test datasets, and preset parameters such as the learning rate, batch size, and number of iterations;
[0016] First, copy the first student model S to S’, input the SST2 training data, use the Tokenizer to convert the input data format. The input data first passes through the Embedding layer, and then sequentially passes through the 12 encoding layers of S’, including the MHA layer and the FFN layer; this process performs backpropagation on S’.
[0017] Divide an independent test set Q in the test dataset, test S’, and calculate the loss function; at the same time, the first teacher model uses the distillation loss result of S’ as a feedback signal and passes the loss result to the first teacher model; then discard S’, and update S with the updated loss parameters. After iterative training, the second student model is obtained.
[0018] As a further technical solution, during the model compression inference and deployment stage, the second student model is deployed on the terminal to achieve text sentiment classification prediction.
[0019] The second aspect of the present invention provides a model compression system based on meta-learning, including:
[0020] A data acquisition module, configured to: construct a sentiment classification data set, divide the training, test, and validation sample sets, and preprocess the data set to adjust the input format;
[0021] A model compression inference module, configured to: use the preprocessed data samples as input to obtain the pre-trained language model Bert_base, and implement model compression using the knowledge distillation compression method based on meta-learning. During the model compression inference and deployment stage, output the text sentiment classification result; the knowledge distillation compression method based on meta-learning includes fine-tuning, pruning, and meta-learning distillation stages.
[0022] The third aspect of the present invention provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, it implements the steps in a model compression method based on meta-learning as described in the first aspect of the present invention.
[0023] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in a model compression method based on meta-learning as described in the first aspect of the present invention.
[0024] The above one or more technical solutions have the following beneficial effects:
[0025] The present invention constructs a preprocessed sentiment classification data set as input, uses the knowledge distillation compression method based on meta-learning to compress the pre-trained language model Bert_base, and outputs the sentiment classification result during the model inference and deployment stage. The above method can obtain more accurate sentiment classification results in text sentiment analysis scenarios with high real-time requirements and limited resources.
[0026] In the present invention, the pruning compression method is used to reduce the number of model parameters and the volume. Through the knowledge distillation method based on meta-learning, the model accuracy is improved under the same number of parameters, so that the compressed model can be applied to downstream text classification task scenarios.
[0027] The present invention can be effectively applied to large model text classification task scenarios, uses model compression means to compress large models, is more convenient to be applied to embedded and mobile devices, has good robustness, and avoids the problem that it is difficult to apply and deploy large models with a large number of parameters.
[0028] Advantages of additional aspects of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation to the present invention.
[0030] Figure 1 It is a flowchart of the first embodiment of the present invention;
[0031] Figure 2 It is a schematic diagram of model pruning of the first embodiment of the present invention;
[0032] Figure 3 It is a schematic diagram of the distillation loss of the first embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0033] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.
[0034] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention.
[0035] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.
[0036] Term Explanation:
[0037] Knowledge distillation technology: A large model with a larger parameter scale but better accuracy effect is used as the teacher model, and a model with a smaller parameter scale and a more compact and lightweight model structure is used as the student model. The research aims to transfer the generalization ability of the teacher model to the student model, so that the student model with a smaller number of parameters and a smaller model volume can also achieve an accuracy effect similar to that of the teacher model.
[0038] Pruning technology: Removing redundant parts in the model to make the model more concise, with the focus on reducing the computational amount and the number of parameters of the model.
[0039] Quantization technology: It is a method to improve the inference speed of the model by reducing the numerical precision. It can store and calculate floating-point numbers represented by FP32 in forms with less memory space occupancy such as INT8 and INT4, achieving the effects of reducing the memory bandwidth requirement, reducing the number of model parameters, and reducing power consumption.
[0040] The First Embodiment
[0041] This embodiment discloses a model compression method based on meta-learning. Figure 1 The schematic diagram of the model compression method based on meta-learning provided by the present invention is shown, as Figure 1 shown, the embodiments of the present invention include the following steps:
[0042] Construct a sentiment classification dataset, divide the training, test, and validation sample sets, preprocess the data, and adjust the input format.
[0043] Use the preprocessed data samples as input to obtain the pre-trained language model Bert_base to achieve large model compression. The model compression process includes fine-tuning, pruning, and meta-learning distillation operations.
[0044] In the model inference and deployment stage, input the data sample information, and output the sentiment classification result through the compressed model.
[0045] As one or more embodiments, use the SST2 sentiment classification dataset in the GLUE dataset, divide the training, test, and validation sample sets, and preprocess the data, including operations such as using a tokenizer to convert text into word segments and annotating sentiment classification labels, to adapt to the input format.
[0046] As one or more embodiments, achieve large model compression by constructing a model compression method based on meta-learning. The compression process includes operations such as fine-tuning, pruning, and meta-learning distillation.
[0047] Further, the fine-tuning process includes inputting the preprocessed SST2 dataset into the pre-trained language model Bert_base. The model includes 12 encoding layers, and the output is a 768-dimensional vector. The encoding layer consists of multi-headed self-attention (MHA) and a feedforward neural network (FFN). Set fine-tuning hyperparameters such as the learning rate and batch size. During training, input the batch data of SST2 into the Bert_base model, calculate the loss and backpropagate to optimize the model parameters. After multiple iterations, stop the iteration when the loss function converges to obtain the fine-tuned model, which is the first teacher model.
[0048] Further, the pruning process is specifically to perform a pruning operation on the first teacher model. The operation of pruning the teacher model before distillation is as Figure 2 shown. After pruning, the first student model is obtained. This process uses the Network Rewiring method to prune the MHA and FFN of Bert_base.
[0049] First, copy the teacher model, input the preprocessed SST2 training dataset into the teacher model, calculate the loss and backpropagate the teacher model for training. Prune the model according to the importance. After multiple iterations, the pruned model is the first student model.
[0050] Specifically, the importance calculation is to calculate the change in the training loss after removing the parameters. The greater the change, the more important the parameter. Let the output of a head be h. According to the first-order Taylor expansion, its importance I h can be equivalent to:
[0051]
[0052] where R is the remainder and L is the loss. In the same way, for the neurons in the FFN layer of the middle layer, focus on pruning the connection weights, and its importance can be equivalent to where W is the weight value.
[0053] Sort the neurons of the head ([H1, H2,... H n ) and FFN ([d1, d2,... d ff ) of Bert_base MHA according to the importance, set the scaling factor γ. For the number of attention heads N H , and the number of neurons d ff in the middle layer, after pruning, the number of attention heads is reduced to and the number of neurons in the FFN layer is reduced to .
[0054] After implementing this method, the model width is reduced by γ times, and the pruned model is the first student model.
[0055] Prune the teacher model before the distillation operation to obtain the first student model, making the model sparse, reducing the number of model parameters and volume, which is beneficial to accelerating the model inference speed.
[0056] Furthermore, the meta-learning distillation process is specifically as follows: use the knowledge distillation compression method based on meta-learning to achieve compression; use the meta-learning mechanism to distill the teacher model and the student model. The meta-learning method realizes the synchronous update of the teacher model and the student model. Different from the previous operation where the weight parameters of the teacher model are fixed, in this method, the teacher model can be adjusted according to the feedback effect of the student model, better transferring knowledge to the student model and improving the generalization ability of the student model.
[0057] Obtain the preprocessed SST2 training and test datasets, and preset parameters such as the learning rate, the number of batches, and the number of iterations.
[0058] In each batch of training, first, the first student model S is copied as S'. The SST2 training data is input, and the Tokenizer is used to convert the input data format. The input data first passes through the Embedding layer and then sequentially passes through the 12 encoding layers of S', including the MHA layer and the FFN layer. This process performs backpropagation on S'.
[0059] In the validation stage, an independent test set Q is divided from the test dataset, S' is tested, and the loss function is calculated. At the same time, the first teacher model uses the distillation loss result of S' as a feedback signal and passes the loss result to the first teacher model. Subsequently, S' is discarded, and S is updated with the updated loss parameters to complete the entire meta-learning distillation process, as shown in Figure 3 the following. Iterate multiple batches of data in the same way, iteratively optimize the first student model S, and finally obtain the second student model.
[0060] The loss function mainly includes the distillation loss of the Embedding layer, the distillation loss of the Hidden layer, and the logit distillation loss for the results.
[0061] Among them, the distillation loss L embd of the Embedding layer can be expressed as:
[0062] L embd = MSE(E S W e , E T )
[0063] Among them, E S is the output of the Embedding layer of the first student model, and E T is the output of the Embedding layer of the first teacher model. Since the dimensions of the embeddings between the two are different and the loss cannot be directly calculated, W e is introduced as a mapping matrix to make the dimension of the first student model the same as that of the first teacher model after mapping. MSE (Mean Squared Error) is the mean squared error loss function, and its formula is:
[0064]
[0065] where n is the number of samples, x i is the true value, and y i is the predicted value.
[0066] The distillation loss L hidden of the Hidden layer can be expressed as:
[0067] L hidden = MSE(H S W h,H T )
[0068] Similar to the distillation of the Embedding layer, in the Hidden layer, H S is the output of the hidden layer of the first student model, and H T is the output of the hidden layer of the first teacher model. W h is the mapping matrix, which makes the dimensions of the hidden layer of the first student model and the first teacher model consistent.
[0069] The Logit distillation loss L logit can be expressed as:
[0070] L logit = -softmax(z T )·log_softmax(z s )
[0071] where z T is the predicted probability of the first teacher model, and z s is the predicted probability of the first student model. T is the temperature of model distillation. The formula for softmax is:
[0072]
[0073] where x i is the output value of the i-th node, and j is the number of output nodes, that is, the number of categories. Using softmax to bound the output values of multi-classification in [0,1] can represent the output probabilities of different categories. Log_softmax is log(softmax(x)), which is used to take the logarithm of softmax.
[0074] The final distillation loss consists of the distillation of the Embedding layer, the Hidden layer, and the logit distillation. M is the number of layers of the teacher network, and m is the number of layers of the student network.
[0075]
[0076] In this embodiment, the distillation iteration epoch is set to 50, the distillation loss is calculated, the first student model is optimized iteratively, and the final second student model is obtained and saved in the form of a weight file for prediction and inference to complete the sentiment classification task.
[0077] This embodiment constructs a preprocessed sentiment classification dataset as input, uses a knowledge distillation compression method based on meta-learning to compress the pre-trained language model Bert_base, and outputs the sentiment classification results at the model inference and deployment stage. The above method can obtain more accurate sentiment classification results in text sentiment analysis scenarios with high real-time requirements and limited resources.
[0078] Embodiment 2
[0079] The purpose of this embodiment is to provide a model compression system based on meta-learning, including:
[0080] A data acquisition module, configured to: construct a sentiment classification dataset, divide the training, test, and validation sample sets, preprocess the dataset, and adjust the input format;
[0081] A model compression and inference module, configured to: take the preprocessed data samples as input, obtain the pre-trained language model Bert_base, use a knowledge distillation compression method based on meta-learning to achieve model compression, and output the text sentiment classification results at the model compression, inference, and deployment stage; the knowledge distillation compression method based on meta-learning includes a fine-tuning, pruning, and meta-learning distillation stage.
[0082] Embodiment 3
[0083] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above method are implemented.
[0084] Embodiment 4
[0085] The purpose of this embodiment is to provide a computer-readable storage medium.
[0086] A computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the above method are executed.
[0087] Although the specific implementation manners of the present invention are described above in conjunction with the accompanying drawings, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that based on the technical solutions of the present invention, various modifications or deformations that can be made by those skilled in the art without creative efforts are still within the protection scope of the present invention.
Claims
1. A model compression method based on meta-learning, characterized in that It includes the following steps: Construct a sentiment classification dataset, divide the training, test, and validation sample sets, preprocess the dataset, and adjust the input format; Use the preprocessed data samples as input to obtain the pre-trained language model Bert_base, and use the knowledge distillation compression method based on meta-learning to compress the model. In the model compression inference deployment stage, output the text sentiment classification result; the knowledge distillation compression method based on meta-learning includes fine-tuning, pruning, and meta-learning distillation stages; Among them, the fine-tuning stage is to train the pre-trained language model based on the preprocessed sentiment classification dataset to obtain the first teacher model; the pruning stage is to prune the first teacher model using a scaling factor to obtain the first student model; the meta-learning distillation stage is to train the second student model based on the obtained first teacher model and first student model, the preprocessed sentiment classification dataset, and the distillation training method based on meta-learning; the distillation training method based on meta-learning includes: Obtain the preprocessed SST2 training and test datasets, and preset parameters such as learning rate, number of batches, and number of iterations; First, copy the first student model S to S', input the SST2 training data, use the Tokenizer to convert the input data format. The input data first passes through the Embedding layer, and then sequentially passes through the 12 encoding layers of S', including the MHA layer and the FFN layer. This process performs backpropagation on S'; Divide an independent test set Q in the test dataset, test S', and calculate the loss function; at the same time, the first teacher model uses the distillation loss result of S' as a feedback signal and passes the loss result to the first teacher model; then discard S', and update S with the updated loss parameters. After iterative training, obtain the second student model.
2. The model compression method based on meta-learning according to claim 1, wherein The specific preprocessing of the dataset is to use the SST2 sentiment classification dataset in the GLUE dataset, divide the training, test, and validation sample sets, and preprocess the data, including using a tokenizer to convert the text into word segments and annotating sentiment classification labels, to adapt to the input format.
3. The model compression method based on meta - learning according to claim 1, characterized in that, In the model compression inference deployment stage, deploy the second student model to the terminal to achieve text sentiment classification prediction.
4. A model compression system based on meta - learning, characterized in that, It includes: A data acquisition module configured to: construct a sentiment classification dataset, divide the training, test, and validation sample sets, preprocess the dataset, and adjust the input format; A model compression inference module configured to: use the preprocessed data samples as input to obtain the pre-trained language model Bert_base, use the knowledge distillation compression method based on meta-learning to compress the model, and in the model compression inference deployment stage, output the text sentiment classification result; the knowledge distillation compression method based on meta-learning includes fine-tuning, pruning, and meta-learning distillation stages; Among them, the fine-tuning stage is to train a pre-trained language model based on the pre-processed sentiment classification dataset to obtain a first teacher model; the pruning stage is to prune the first teacher model using a scaling factor to obtain a first student model; the meta-learning distillation stage is to train a second student model based on the obtained first teacher model and first student model, the pre-processed sentiment classification dataset, and a distillation training method based on meta-learning; the distillation training method based on meta-learning includes: Obtain the pre-processed SST2 training and test datasets, and preset the parameters of the learning rate, number of batches, and number of iterations. First, copy the first student model S to S', input the SST2 training data, use the Tokenizer to convert the format of the input data, the input data first passes through the Embedding layer, and then sequentially passes through the 12 encoding layers of S', including the MHA layer and the FFN layer, and backpropagation is performed on S' in this process. Divide an independent test set Q in the test dataset, test S', and calculate the loss function; at the same time, the first teacher model uses the distillation loss result of S' as a feedback signal and passes the loss result to the first teacher model; then discard S', update S with the updated loss parameters, and obtain the second student model after iterative training.
5. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in a model compression method based on meta-learning according to any one of claims 1-3.
6. An electronic device, comprising a memory, a processor, and a program stored on the memory and executable on the processor, characterized in that When the processor executes the program, it implements the steps in a model compression method based on meta-learning according to any one of claims 1-3.