Model lightweight method based on knowledge distillation

Abstract

The invention discloses a model lightweight method based on knowledge distillation, which is characterized in that soft labels generated by a plurality of complex models are used as training data to train a small model, and the small model can simulate behaviors of a large model. Experiments show that by using the knowledge distillation method, the calculation overhead can be remarkably reduced while the similar performance is maintained, and the small model can exceed the single large model under some conditions; according to the method, knowledge of the large model is transmitted to the small model, the knowledge distillation method can enable the small model to achieve similar performance with the large model, meanwhile, calculation resource consumption is greatly reduced, and the small model has higher efficiency in the aspects of processing speed and memory consumption.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning knowledge distillation, and in particular relates to a model lightweight method based on knowledge distillation. Background Art

[0002] Knowledge distillation is a model compression technique in deep learning that aims to reduce the computational cost and storage requirements of a model while ensuring its performance by transferring the knowledge of a complex model, usually called a "teacher model", to a simpler model, called a "student model".

[0003] With the rapid development of deep learning, especially in fields such as computer vision, natural language processing, and speech recognition, neural network models have become increasingly complex, often requiring large amounts of training data and powerful computing resources to train for good performance. However, the deployment of these complex models in real-world applications is often limited, especially on resource-constrained devices.

[0004] Ensemble learning improves prediction accuracy by combining multiple models, but it also brings computational and storage burdens. Therefore, how to effectively transfer the knowledge of a large model to a streamlined small model has become a key research topic.

[0005] The basic idea of ​​knowledge distillation is to train the student model through the "soft targets" generated by the teacher model, so that it learns the knowledge of the teacher model, especially the output probability distribution of the teacher model, rather than just the hard labels. The distillation process uses the hyperparameter "temperature" to adjust the smoothness of the soft targets, helping the student model to extract more valuable information from the teacher model, thereby improving the generalization ability of the student model. Application scenarios of knowledge distillation include model compression and acceleration, transfer learning, ensemble learning, etc., and are especially suitable for devices that require fast response and resource constraints. However, some information may be lost during the distillation process, resulting in a decrease in the performance of the student model. At the same time, the distillation process itself may require a lot of computing resources. Therefore, how to avoid information loss while ensuring compression efficiency, and how to effectively transfer the core knowledge of complex models, remain important challenges in current research.

[0006] Existing techniques include boosting, which improves overall model performance by gradually correcting the errors of base learners. Typically, each time a new base learner is trained, more attention is paid to samples that were misclassified by the previous model. The specific steps are: first, a weak learner is trained and its error on the training set is calculated. The weights of the samples are then adjusted based on the error samples. Next, the next learner is trained, and the previous error is further optimized. Finally, the predictions of all base learners are weighted together to obtain the final result. However, this method has the following drawbacks: In the boosting method, new base learners are trained on samples with errors from the previous model. This means that if certain data points are noise or outliers, they are given higher weights, resulting in them being focused on and corrected by subsequent base learners. Because the errors in these noise points may not be universal or representative, the model may overfit to these noise points, resulting in a decrease in the overall model's generalization ability. Furthermore, while the basic idea of ​​the boosting method is to improve model performance by gradually correcting errors, this process can also lead to overfitting.

[0007] Existing techniques also include bagging, which generates multiple training subsets by resampling the original training data with replacement. A base learner is then trained on each subset, and the final prediction result is obtained by voting or averaging the predictions of the base learners. The specific steps include: first, randomly selecting multiple subsets from the original training data, then using these subsets to train multiple models, and finally aggregating the predictions of these models, usually using a voting mechanism or averaging. The disadvantage of bagging is that although it can reduce overfitting, if the base learners themselves are not very different, the performance improvement of the model may be limited, and the computational and storage overhead is high.

[0008] In summary, the existing technology has the following technical problems that need to be solved urgently: 1. Computational overhead of ensemble learning: Traditional ensemble learning methods improve prediction accuracy by combining multiple models, but this usually requires a lot of computing resources and storage space, especially when faced with large-scale data and complex models.

[0009] 2. Challenges in large-scale model deployment: Although large neural networks can achieve good performance during the training phase, they are often not suitable for direct deployment on resource-constrained devices, especially in practical applications that require low latency and high efficiency.

[0010] 3. Different requirements for training data and inference phases: In many tasks, the training phase needs to process a large amount of redundant data and extract data structures, while the deployment phase faces strict requirements on latency and computing resources.

[0011] Therefore, it is necessary to propose a model lightweight method based on knowledge distillation to solve the above problems. Summary of the Invention

[0012] The technical problem to be solved by the present invention is to provide a model lightweight method based on knowledge distillation. By using the soft labels generated by multiple complex models as training data, a smaller model is trained. This small model can simulate the behavior of the large model, effectively solving the limitations of ensemble learning in large-scale data and actual deployment, and can significantly reduce computing and storage requirements without sacrificing performance.

[0013] In order to solve the above technical problems, the technical solution adopted by the present invention is: The model lightweight method based on knowledge distillation includes the following steps: S1: Train multiple large models: Use the same dataset and training objectives to train multiple large neural network models as teacher models; S2: Generate soft labels: Obtain the prediction results of the teacher model for the input data. The soft labels are the probability distribution predicted by the model, including the relative confidence information between categories. S3: Train the small model: Using the soft labels generated by multiple teacher models as training data, train the student model to fit the output probability distribution of the teacher model, so that the student model can simulate the behavior of multiple teacher models; S4: Optimize the distillation loss function; S5: Control the smoothness of the soft label through the temperature parameter T; S6: Training and evaluation.

[0014] Preferably, the soft label is a probability distribution vector output by the teacher model, whose element values ​​represent the probability that the input data belongs to the corresponding category.

[0015] Preferably, the teacher model is an integrated model composed of multiple base models, and its soft label is generated by weighted averaging the outputs of each base model.

[0016] Preferably, the distillation loss function in step S4 includes two loss functions, the first term is the difference between the student model output and the true label, and the second term is the difference between the student model output and the teacher model soft label.

[0017] Preferably, in step S5, the smoothness of the soft label is controlled by the temperature parameter T to satisfy: When T>1, the output probability distribution is smoothed to strengthen the learning of inter-category relationships, and when T<1, the output probability distribution is sharpened to expand the differences between categories.

[0018] Preferably, in step S6, a weighted sum is performed on the two losses in S4, and the student model is optimized by minimizing the total loss to obtain a lightweight model with lower computational complexity than a traditional single neural network and performance close to that of multiple teacher models.

[0019] Preferably, the distillation loss function of S4 is expressed as: ; in: is the weight coefficient, $y$ is the true label, is the output of the student model, is the soft label of the teacher model, is the cross entropy loss function, is the KL divergence loss function.

[0020] Preferably, the temperature parameter T acts on the output layer of the teacher model, and the soft label is calculated by the following formula: ; in For category The original output value of .

[0021] Preferably, the weighted sum satisfies ,and .

[0022] Preferably, the data set is the MNIST handwritten digit data set.

[0023] The beneficial effects of the present invention are as follows: By transferring knowledge from the large model to the small model, knowledge distillation enables the small model to achieve similar performance to the large model while significantly reducing computing resource consumption. This results in a smaller model with higher efficiency in both processing speed and memory consumption.

[0024] 2. Directly integrating multiple large models for prediction can be very time-consuming and consumes a lot of computing resources. Knowledge distillation greatly improves prediction efficiency by compressing the integrated model into a single small model, making it suitable for deployment in real-world applications.

[0025] 3. This method is not limited to specific models or tasks and can be applied to almost any machine learning algorithm, especially neural network training. Whether it is classification problems, regression problems, or other types of learning tasks, the knowledge distillation method has good adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of a flow chart of the present invention; Figure 2 Schematic diagram of the knowledge distillation structure framework in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] Example 1: like Figure 1 As shown in the figure, the model lightweight method based on knowledge distillation includes the following steps: S1: Train multiple large models: Use the same dataset and training objectives to train multiple large neural network models as teacher models; S2: Generate soft labels: Obtain the prediction results of the teacher model for the input data. The soft labels are the probability distribution predicted by the model, including the relative confidence information between categories. S3: Train the small model: Using the soft labels generated by multiple teacher models as training data, train the student model to fit the output probability distribution of the teacher model, so that the student model can simulate the behavior of multiple teacher models; S4: Optimize the distillation loss function; S5: Control the smoothness of the soft label through the temperature parameter T; S6: Training and evaluation.

[0028] Preferably, the soft label is a probability distribution vector output by the teacher model, whose element values ​​represent the probability that the input data belongs to the corresponding category.

[0029] Preferably, the teacher model is an integrated model composed of multiple base models, and its soft label is generated by weighted averaging the outputs of each base model.

[0030] Preferably, the distillation loss function in step S4 includes two loss functions, the first term is the difference between the student model output and the true label, and the second term is the difference between the student model output and the teacher model soft label.

[0031] Preferably, in step S5, the smoothness of the soft label is controlled by the temperature parameter T to satisfy: When T>1, the output probability distribution is smoothed to strengthen the learning of inter-category relationships, and when T<1, the output probability distribution is sharpened to expand the differences between categories.

[0032] Preferably, in step S6, a weighted sum is performed on the two losses in S4, and the student model is optimized by minimizing the total loss to obtain a lightweight model with lower computational complexity than a traditional single neural network and performance close to that of multiple teacher models.

[0033] Preferably, the distillation loss function of S4 is expressed as: ; in: is the weight coefficient, $y$ is the true label, is the output of the student model, is the soft label of the teacher model, is the cross entropy loss function, is the KL divergence loss function.

[0034] Preferably, the temperature parameter T acts on the output layer of the teacher model, and the soft label is calculated by the following formula: ; in For category The original output value of .

[0035] Preferably, the weighted sum satisfies ,and .

[0036] Preferably, the data set is the MNIST handwritten digit data set.

[0037] Example 2: like Figure 2 As shown in Figure 2, building a knowledge distillation structure includes: The first step in knowledge distillation is to train a complex teacher model. This teacher model is usually a deep neural network, which can be a very large model that can extract complex features from a large amount of data. In the present invention, the teacher model is trained with a large amount of training data, and conventional training methods such as maximizing the log-likelihood function are used to adjust the parameters of the model so that it can accurately predict the category of the sample. After the training is completed, the teacher model will output the probability distribution of each sample belonging to different categories. These probability distributions are called "soft targets". These soft targets contain the relationship and relative probabilities between categories. Even for misclassified samples, the teacher model will still give some probabilities about the wrong category, providing richer category information.

[0038] To perform knowledge distillation, the output of the teacher model needs to be "temperatured." Temperature is a hyperparameter that controls the smoothness of the output probabilities. Typically, the temperature is set to 1, but during distillation, it's often set higher. A high temperature smoothes the output of the teacher model, minimizing the differences in probabilities between categories, allowing the student model to learn more information.

[0039] During distillation, a student model is trained to mimic the output of the teacher model. The student model is typically a smaller network, trained to produce outputs that are as close as possible to the temperature-adjusted probability distribution of the teacher model's soft targets. The student model is typically trained using a cross-entropy loss function, but unlike traditional training methods, the student model no longer relies on hard labels (the true class labels) but instead relies on soft targets.

[0040] The goal of the student model is to learn the knowledge contained in the teacher model by minimizing the difference between the soft targets and the teacher model. In this process, the student model is trained by using the soft targets generated by the teacher model as labels. This training method enables the student model to learn the generalization ability of the teacher model, rather than simply memorizing the labels in the training set.

[0041] Unlike hard targets in traditional training methods, soft targets provide richer information. In the output of the teacher model, soft targets contain information about the similarity between categories. For example, suppose an image belongs to the category "cat", but the teacher model may give a very small probability of the "dog" category. Even if this probability is small, it still provides useful information about the similarity between "dog" and "cat". In contrast, hard targets only provide a category label and do not contain the relative probabilities between categories. By using soft targets, the student model not only learns the correct category labels, but also learns the similarities between incorrect categories, which helps improve the generalization ability of the student model. By imitating the soft targets of the teacher model, the student model can achieve better performance on different training data.

[0042] When the temperature is too high, the output probability of the teacher model becomes too smooth, which may lose some fine-grained classification information. To optimize this process, the authors proposed adjusting the soft target by selecting an appropriate temperature. A high temperature helps the student model better learn the relationship between categories, but a temperature that is too high may cause excessive smoothing of information, making the student model unable to fully capture details. Therefore, choosing an appropriate temperature is very important. In some cases, the teacher model may be an ensemble model rather than a single deep neural network. The ensemble model improves performance by combining the prediction results of multiple models, but this will increase the computational cost. To avoid this, the knowledge distillation method allows the knowledge of multiple teacher models to be merged into a student model. Specifically, the teacher model can be a combination of multiple models, and a "soft target" is formed by taking a weighted average of the output of each model to train the student model.

[0043] Example 3: In this example, multiple experiments were conducted to verify the effectiveness of knowledge distillation, especially its performance on the MNIST dataset. The experimental process and results are shown in Tables 1 and 2 below: Table 1: Experimental dataset division;

[0044] Table 2: Comparison with existing technologies

[0045] As shown in Table 2 above, experiments show that although the student model is much smaller than the teacher model, through the distillation process, the student model is able to learn the generalization capabilities of the teacher model, ultimately achieving performance close to or even exceeding that of the teacher model on the test set. In speech recognition tasks in particular, distillation technology successfully transfers the knowledge of a large ensemble model to a small model, significantly reducing computational overhead while maintaining high accuracy.

Claims

1. A model lightweight method based on knowledge distillation, characterized by: The following steps are involved: S1: Train multiple large models: Use the same dataset and training objectives to train multiple large neural network models as teacher models; S2: Generate soft labels: Obtain the prediction results of the teacher model for the input data. The soft labels are the probability distribution predicted by the model, including the relative confidence information between categories. S3: Train the small model: Using the soft labels generated by multiple teacher models as training data, train the student model to fit the output probability distribution of the teacher model, so that the student model can simulate the behavior of multiple teacher models; S4: Optimize the distillation loss function; S5: Control the smoothness of the soft label through the temperature parameter T; S6: Training and evaluation.

2. The model lightweight method based on knowledge distillation according to claim 1 is characterized in that: The soft label is a probability distribution vector output by the teacher model, and its element value represents the probability that the input data belongs to the corresponding category.

3. The model lightweight method based on knowledge distillation according to claim 1 is characterized in that: The teacher model is an integrated model composed of multiple base models, and its soft label is generated by weighted averaging the outputs of each base model.

4. The model lightweight method based on knowledge distillation according to claim 1 is characterized in that: The distillation loss function in step S4 contains two loss functions. The first one is the difference between the student model output and the true label, and the second one is the difference between the student model output and the teacher model soft label.

5. The model lightweight method based on knowledge distillation according to claim 1 is characterized in that: In step S5, the smoothness of the soft label is controlled by the temperature parameter T to meet the following conditions: When T>1, the output probability distribution is smoothed to strengthen the learning of inter-category relationships, and when T<1, the output probability distribution is sharpened to expand the differences between categories.

6. The model lightweight method based on knowledge distillation according to claim 1, characterized in that: In step S6, the two losses in S4 are weightedly summed, and the student model is optimized by minimizing the total loss to obtain a lightweight model with lower computational complexity than a traditional single neural network and performance close to that of multiple teacher models.

7. The model lightweight method based on knowledge distillation according to claim 4 is characterized in that: The distillation loss function of S4 is expressed as: ； in: is the weight coefficient, $y$ is the true label, is the output of the student model, is the soft label of the teacher model, is the cross entropy loss function, is the KL divergence loss function.

8. The model lightweight method based on knowledge distillation according to claim 5, characterized in that: The temperature parameter T acts on the output layer of the teacher model, and the soft label is calculated by the following formula: ； in For category The original output value of .

9. The model lightweight method based on knowledge distillation according to claim 7, characterized in that: The weighted sum satisfies ,and .

10. The model lightweight method based on knowledge distillation according to claim 1, characterized in that: The dataset is the MNIST handwritten digit dataset.

Images