Neural network model knowledge distillation method and system based on automatic gradient mixing
By introducing an automatic gradient mixing method in knowledge distillation, the weight coefficients α and β are automatically calculated using the angle θ of the task gradient vector and the distillation gradient vector and the total gradient vector, which solves the problem of high time complexity of adjusting the weight coefficient hyperparameters in the prior art, and achieves a more efficient training effect.
Patent Information
- Application Number
- CN202210544252.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-18
- Publication Date
- 2025-04-08
- Estimated Expiration
- 2042-05-18
AI Technical Summary
When the prior art adjusts the weight coefficient hyperparameters in knowledge distillation, the time complexity is high and the relevant prior knowledge of knowledge distillation cannot be effectively utilized, resulting in the search process being time-consuming and labor-intensive and the optimality of the results cannot be guaranteed.
A neural network model knowledge distillation method based on automatic gradient mixing is proposed. The knowledge of the teacher network is distilled to the student network through the loss function, and the task gradient vector and the distillation gradient vector are backpropagated to obtain the task gradient vector and the included angle θ are calculated using these gradient vectors, and the total gradient vector is constrained according to the task gradient vector, candidate deflection coefficients are obtained, and candidate deflection coefficients with the best effect are selected as the final deflection coefficient λ, and the weight coefficients α and β are calculated.
With the same computing resources, the effect of running about 30 trials in the hyperparameter optimization method in the same operation time is achieved, which greatly saves manpower and material resources for parameter adjustment and effectively improves the training effect.
Smart Images
Figure CN115019085B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical fields of computer vision, knowledge distillation, image classification and hyperparameter optimization, and in particular to an automatic gradient mixing method and system based on knowledge distillation. Background Art
[0002] Knowledge distillation is an effective model compression method that transfers knowledge from a large teacher network to a small student network to improve the performance of the small network. In knowledge distillation, the loss function L is calculated. kd In addition to considering the task loss function l of the non-distilled item, task In addition, we also need to consider the distillation loss function L distill , the formula is as follows, where α and β are weight coefficients:
[0003] L kd =αL task +βL distill ,
[0004] In actual experiments, when switching datasets, distillation methods, or teacher-student network combinations, it is often necessary to adjust the two weight coefficients. A grid search method is used to find suitable hyperparameters, that is, a method similar to traversal search to process the parameter search space discretized by gridding.
[0005] Use grid search to configure the hyperparameters of the loss function. However, this type of method is a completely traversal search and does not utilize the relevant prior knowledge of knowledge distillation. Even if the continuous parameter space is discretized, the search space is still very large. For example, in the knowledge distillation framework RepDistiller, the weight of the distillation loss function in the same data set ranges from 0.02 (Probabilistic Knowledge Transfer) to 30,000 (Correlation Congruence for Knowledge Distillation), so the search process is time-consuming and labor-intensive and cannot guarantee that the result is an optimal situation. At present, there has been no research on the configuration of weight coefficients for knowledge distillation, but adjusting the weight of the loss function has some similarities with multi-task learning and hyperparameter optimization methods. The following is a brief introduction to these two methods.
[0006] Multi-task learning generally uses the knowledge contained in all tasks or some other tasks to jointly learn multiple tasks to improve the learning of the model for each task. The multi-loss optimization in multi-task learning is similar to the optimization problem of distillation loss, both of which are to optimize the coefficients before the loss function so that the entire network can be trained to produce a good result. However, this type of method also has some problems. The multi-task learning method will optimize all tasks or all loss functions equally, which is somewhat in conflict with the goal of knowledge distillation, because in knowledge distillation, the task loss L task is needed to dominate, and the distillation loss L distill It is used to assist the model to better learn the task objectives. Under equal weight optimization, it is easy to over-balance the distillation loss and damage the task objectives.
[0007] Hyperparameter optimization is a universal method that can be used in all application scenarios of hyperparameter adjustment in theory. There are methods such as Bayesian modeling and heuristic search to adjust hyperparameters. In theory, the performance of these methods can be better than grid search. However, in the parameter adjustment of knowledge distillation, such methods do not use some unique prior knowledge related to knowledge distillation, so there is still a large computational complexity. Summary of the invention
[0008] The purpose of the present invention is to overcome the problem of high time complexity in adjusting weight coefficient hyperparameters in the above-mentioned prior art, and propose a method that can automatically mix gradients. Specifically, the present invention proposes a neural network model knowledge distillation method based on automatic gradient mixing, which includes:
[0009] Step 1: Distill the knowledge of the teacher network to the student network through the loss function to obtain the task loss L task and distillation loss L distill ;
[0010] Step 2: Loss L for this task task and the distillation loss L distill Perform back propagation to obtain the task loss L task and the distillation loss L distill The corresponding task gradient vector G task and the distilled gradient vector G distill , as well as the task gradient vector and the distillation gradient vector G distill The angle θ between them;
[0011] Step 3: According to the task gradient vector G task Constrain the total gradient vector G kd , and obtain multiple candidate deflection coefficients, and use each candidate deflection coefficient to adjust the total gradient vector G kdDeflection is performed, and the candidate deflection coefficient with the best effect is selected as the final deflection coefficient λ;
[0012] Step 4: Based on the final deflection coefficient λ, the weight coefficients α and β are obtained by the following formula:
[0013]
[0014] Add the weight coefficients α and β to the loss function L kd =αL task +βL distill The final loss function is obtained, and the knowledge of the teacher network is distilled to the student network with the final loss function.
[0015] The neural network model knowledge distillation method based on automatic gradient mixing, in which the task gradient vector G is calculated by the following formula task and the distilled gradient vector G distill The angle θ between them:
[0016]
[0017] The neural network model knowledge distillation method based on automatic gradient mixing, wherein the network scale of the teacher network is larger than the network scale of the student network.
[0018] The neural network model knowledge distillation method based on automatic gradient mixing, wherein the teacher network and the student network are both image classification neural network models;
[0019] This step 3 includes selecting the candidate deflection coefficient with the highest classification accuracy of the student network image as the final deflection coefficient λ;
[0020] The step 4 includes classifying the images using the student network after knowledge distillation, and obtaining the category label of each image as the image classification result.
[0021] The present invention also proposes a neural network model knowledge distillation system based on automatic gradient mixing, which includes:
[0022] The initial module is used to distill the knowledge of the teacher network to the student network through the loss function to obtain the task loss L task and distillation loss L distill ;
[0023] The propagation module is used to calculate the loss L for this task. task and the distillation loss L distill Perform back propagation to obtain the task loss L task and the distillation loss L distill The corresponding task gradient vector G task and the distilled gradient vector Gdistill , as well as the task gradient vector and the distillation gradient vector G distill The angle θ between them;
[0024] The deflection module is used to transform the gradient vector G according to the task task Constrain the total gradient vector G kd , and obtain multiple candidate deflection coefficients, and use each candidate deflection coefficient to adjust the total gradient vector G kd Deflection is performed, and the candidate deflection coefficient with the best effect is selected as the final deflection coefficient λ;
[0025] The distillation module is used to obtain the weight coefficients α and β based on the final deflection coefficient λ by the following formula:
[0026]
[0027] Add the weight coefficients α and β to the loss function L kd =αL task +βL distill The final loss function is obtained, and the knowledge of the teacher network is distilled to the student network with the final loss function.
[0028] The neural network model knowledge distillation system based on automatic gradient mixing, in which the task gradient vector G is calculated by the following formula task and the distilled gradient vector G distill The angle θ between them:
[0029]
[0030] The neural network model knowledge distillation system based on automatic gradient mixing, wherein the network scale of the teacher network is larger than the network scale of the student network.
[0031] The neural network model knowledge distillation system based on automatic gradient mixing, wherein the teacher network and the student network are both image classification neural network models;
[0032] The deflection module is used to select the candidate deflection coefficient with the highest classification accuracy of student network pictures as the final deflection coefficient λ;
[0033] The distillation module is used to classify images using the student network after knowledge distillation, and obtain the category label of each image as the image classification result.
[0034] The present invention also proposes a storage medium for storing a program for executing any one of the neural network model knowledge distillation methods based on automatic gradient mixing.
[0035] The present invention also proposes a client for use in any of the neural network model knowledge distillation systems based on automatic gradient mixing.
[0036] It can be seen from the above scheme that the advantages of the present invention are:
[0037] The automatic gradient hybrid method can achieve the effect of running about 30 trials of the HPO method (because manual grid search is inconvenient to set up, the better performing HPO method is used for simulation) in only 1.6 training trials under the same computing resources, thus greatly saving manpower and material resources for parameter adjustment. The effect diagram is shown in the figure Figure 3 As shown in the figure, the left figure is the simulated annealing method, the middle figure is the TPE method, and the right figure is the Hyperban method. Each point represents a trial; the dotted line represents the optimal accuracy adjusted by HPO as the number of trials increases; and the regular triangle mark is the effect of the automatic gradient mixing method (i.e., 72.48% accuracy in 1.6 tirals).
[0038] Figure 4 The technical effect table shows the effect comparison between 13 different teacher-student combinations and the manual adjustment method on the CIFAR-100 dataset on 10 different distillation methods. We can find that the automatic gradient mixing method we proposed can be widely applied to a variety of distillation methods and effectively improve their training effects. Figure 3 , Figure 4 ,The method we proposed can achieve better training results using a shorter training time. Compared with manual adjustment, it is a universal, simple and efficient method. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 Schematic diagram of gradient vector synthesis;
[0040] Figure 2 A schematic diagram of program code corresponding to the training process of the present invention;
[0041] Figure 3 This is a comparison chart of the effects of three types of HPO algorithms and automatic gradient hybrid methods;
[0042] Figure 4 A comparative table chart is provided to show the technical effects. DETAILED DESCRIPTION
[0043] In order to achieve the above technical effects, the present invention mainly includes the following key technical points:
[0044] Key point 1: constrain the modulus of the total gradient vector and use a set of hyperparameters from non-distillation training. This ensures the stability of the entire training process and prevents gradient explosion or vanishing. Since the hyperparameters from non-distillation training are inherited, there is no need to train a separate set of content.
[0045] Key point 2: Search for the deflection coefficient λ in the early rounds to find the appropriate deflection value under the network configuration; experiments have shown that after constraining the total gradient modulus, compared with the original direct adjustment of the weight coefficient, the experimental results of the early rounds and the final experimental results in the automatic gradient hybrid method have an order-preserving property, so the search in the first 5% of the experimental runs is faster than searching in the entire round.
[0046] Key point 3, a conversion method for gradient vector bias and weight coefficient deflection is proposed:
[0047]
[0048] Ability to convert the weight coefficient configuration problem into a gradient vector mixing problem.
[0049] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.
[0050] The loss function parameters are difficult to adjust because all existing technologies ignore some prior knowledge brought by knowledge distillation and adopt a traversal search method, which results in a large search space and makes the entire hyperparameter adjustment process have a large time complexity.
[0051] In solving the above problems, the present invention considers the significance of knowledge distillation for the entire model training. For each neural network model, it is composed of multiple parameters. Assuming that the number of parameters of a model is n, then in this case, imagine an n-dimensional Euclidean space. The state of the model at each moment can correspond to a point in this parameter space. Each time the model is iterated in the forward-backward propagation process, the position of this point actually changes, and the content of this point displacement is the total gradient vector G obtained by the loss function in the back propagation. kd :
[0052] G kd =αG task +βG distill , (1)
[0053] Vector G kd The synthesis is composed of the task gradient vector G task and the distilled gradient vector G distill Formula (1) is composed of L kd During back propagation, the derivative of the trainable parameters is obtained.
[0054] In the training of deep neural networks, the step size of each training iteration is mainly used to control the stability of the training, while the direction of the gradient vector controls the development of the neural network training. Therefore, the present invention can use the first prior knowledge of knowledge distillation here - before the distillation training, there will be a non-distilled student network training. In common data sets and network structures, there will be many successful training processes. The present invention utilizes the above-mentioned non-distilled training process, that is, using all the same hyperparameters, such as training rounds, training step size, batch size, etc. In this case, the present invention adds a constraint on the total gradient vector modulus:
[0055] ||G kd ||=||G task ||, (2)
[0056] Under this condition, the training stability of the model can be ensured, and at the same time, a great constraint is imposed on the search space of the hyperparameters. In the prior art, determining the mixed gradient vector requires searching for two values, namely, its direction and modulus. The present invention introduces the prior knowledge of knowledge distillation through the above formula (2), so that the length can be determined and the search is only performed in the direction, thus successfully reducing the search space. The final gradient vector G kd The length has also been determined, and only its direction needs to be determined to completely determine the update vector.
[0057] The process of determining the direction can be understood as rotating the original non-distilled update vector toward the distilled update vector, such as Figure 1 As shown, the angle between the original two vectors can be calculated using the cosine theorem:
[0058]
[0059] And we can assume that the new resultant vector G kd The original non-distilled task is rotated by an angle of λθ, where λ∈[0,1]. When λ is 0, it degenerates to the situation of non-distilled training, and when λ is 1, it becomes the direction of the vector generated by the distillation loss. In addition, this mode can also help understand the role of the knowledge distillation mode, that is, using the auxiliary distillation loss to make the gradient update vector generated by the original task loss complete a deflection, so that the entire training can correct some knowledge blind spots generated by the original training using only real data. But now the question is, how large should the correction be, in other words, what value of this λ should be.
[0060] Through experiments, we found that the values of λ that achieve the best results are not exactly the same under different datasets, distillation methods, and network combinations. This is obviously reasonable because λ provides a relative angle, while the absolute angle θ and the new vector G distillIt will change according to different combinations, so we need to find the value of λ under each combination. If we directly use certain methods to search, we can get good results, but this goes against our idea of automatically setting coefficients. In order to solve this problem, we verified that after the module length constraint mentioned above, when only λ is changed, the effects of the early rounds and the final rounds are rank-preserving. Therefore, we can perform a small-range search in the early rounds to find a relatively suitable value of λ. For example, the training conducted on the CIFAR-100 dataset requires 240 rounds to complete the training. After the module length constraint is imposed, we can find the appropriate λ through only a small-range search in the first 5 rounds. Figure 2 The algorithm flow shown is the training flow of the present invention.
[0061] Follow this training process to complete the training. In each iteration, we can directly derive two weight coefficients α and β from the direction and modulus:
[0062]
[0063] This can save a lot of time and speed up the entire training process.
[0064] The present invention is temporarily applied to image classification tasks in the field of deep learning. In theory, it can be applied to any distillation method with multiple loss functions, including detection in image tasks or some natural language processing tasks.
[0065] Technical solution: First, for the entire training process, the following two stages are used to complete it. First, a fixed λ parameter group is set to search (for example, with a step size of 0.2, λ is set to 0.1 to 0.9 in the early rounds), that is, the automatic gradient mixing method is used to experiment on the early rounds to find the deflection coefficient λ corresponding to the optimal effect (image classification effect, language recognition effect). The results of the early rounds and the final results are rank-preserving, so the λ setting that is optimal for the early results is likely to also have a better final result, so the automatic gradient mixing training is performed again using the λ corresponding to the optimal effect. The entire experimental process is as follows: Figure 2 shown.
[0066] In the specific implementation of automatic gradient mixing, first for task L task And the distillation loss function L distill Perform back propagation, that is, derive the student network parameters to obtain two gradient update vectors G task and G distill , and then use the law of cosines to calculate the angle between the two: After completing this step, we can use the formula obtained above: Then add these two weight coefficients to the loss function, that is, L kd =αL task +βL distill , and use this loss function to do back propagation to update the student network parameters.
[0067] The algorithm can be run and stored in deep learning hardware such as GPU or TPU. The experiment is mainly completed in the RepDistiller framework under pytorch. It can also be carried out in various deep learning frameworks.
[0068] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.
[0069] The present invention also proposes a neural network model knowledge distillation system based on automatic gradient mixing, which includes:
[0070] The initial module is used to distill the knowledge of the teacher network to the student network through the loss function to obtain the task loss L task and distillation loss L distill ;
[0071] The propagation module is used to calculate the loss L for this task. task and the distillation loss L distill Perform back propagation to obtain the task loss L task and the distillation loss L distill The corresponding task gradient vector G task and the distilled gradient vector G distill , as well as the task gradient vector and the distillation gradient vector G distill The angle θ between them;
[0072] The deflection module is used to transform the gradient vector G according to the task task Constrain the total gradient vector G kd , and obtain multiple candidate deflection coefficients, and use each candidate deflection coefficient to adjust the total gradient vector G kd Deflection is performed, and the candidate deflection coefficient with the best effect is selected as the final deflection coefficient λ;
[0073] The distillation module is used to obtain the weight coefficients α and β based on the final deflection coefficient λ by the following formula:
[0074]
[0075] Add the weight coefficients α and β to the loss function Lkd =αL task +βL distill The final loss function is obtained, and the knowledge of the teacher network is distilled to the student network with the final loss function.
[0076] The neural network model knowledge distillation system based on automatic gradient mixing, in which the task gradient vector G is calculated by the following formula task and the distilled gradient vector G distill The angle θ between them:
[0077]
[0078] The neural network model knowledge distillation system based on automatic gradient mixing, wherein the network scale of the teacher network is larger than the network scale of the student network.
[0079] The neural network model knowledge distillation system based on automatic gradient mixing, wherein the teacher network and the student network are both image classification neural network models;
[0080] The deflection module is used to select the candidate deflection coefficient with the highest classification accuracy of student network pictures as the final deflection coefficient λ;
[0081] The distillation module is used to classify images using the student network after knowledge distillation, and obtain the category label of each image as the image classification result.
[0082] The present invention also proposes a storage medium for storing a program for executing any one of the neural network model knowledge distillation methods based on automatic gradient mixing.
[0083] The present invention also proposes a client for use in any of the neural network model knowledge distillation systems based on automatic gradient mixing.
Claims
1. A neural network model knowledge distillation method based on automatic gradient mixing, characterized in that: include: Step 1: Distill the knowledge of the teacher network to the student network through the loss function to obtain the task loss L task and distillation loss L distill ; Step 2: Loss L for this task task and the distillation loss L distill Perform back propagation to obtain the task loss L task and the distillation loss L distill The corresponding task gradient vector G task and the distilled gradient vector G distill , and the task gradient vector and distillation gradient vector G are calculated by the following formula distill The angle θ between them; Step 3: According to the task gradient vector G task Constrain the total gradient vector G kd , and obtain multiple candidate deflection coefficients, and use each candidate deflection coefficient to adjust the total gradient vector G kd Deflection is performed, and the candidate deflection coefficient with the best effect is selected as the final deflection coefficient λ; Step 4: Based on the final deflection coefficient λ, the weight coefficients α and β are obtained by the following formula: Add the weight coefficients α and β to the loss function L kd =αL task +βL distill In the process, the final loss function is obtained, and the knowledge of the teacher network is distilled into the student network by the final loss function; The teacher network and the student network are both image classification neural network models; This step 3 includes selecting the candidate deflection coefficient with the highest classification accuracy of the student network image as the final deflection coefficient λ; The step 4 includes classifying the images using the student network after knowledge distillation, and obtaining the category label of each image as the image classification result.
2. The neural network model knowledge distillation method based on automatic gradient mixing according to claim 1, characterized in that: The network size of the teacher network is larger than the network size of the student network.
3. A neural network model knowledge distillation system based on automatic gradient mixing, characterized in that: include: The initial module is used to distill the knowledge of the teacher network to the student network through the loss function to obtain the task loss L task and distillation loss L distill ; The propagation module is used to calculate the loss L for this task. task and the distillation loss L distill Perform back propagation to obtain the task loss L task and the distillation loss L distill The corresponding task gradient vector G task and the distilled gradient vector G distill , and the task gradient vector and distillation gradient vector G are calculated by the following formula distill The angle θ between them; The deflection module is used to transform the gradient vector G according to the task task Constrain the total gradient vector G kd , and obtain multiple candidate deflection coefficients, and use each candidate deflection coefficient to adjust the total gradient vector G kd Deflection is performed, and the candidate deflection coefficient with the best effect is selected as the final deflection coefficient λ; The distillation module is used to obtain the weight coefficients α and β based on the final deflection coefficient λ by the following formula: Add the weight coefficients α and β to the loss function L kd =αL task +βL distill In the process, the final loss function is obtained, and the knowledge of the teacher network is distilled into the student network by the final loss function; The teacher network and the student network are both image classification neural network models; The deflection module is used to select the candidate deflection coefficient with the highest classification accuracy of student network pictures as the final deflection coefficient λ; The distillation module is used to classify images using the student network after knowledge distillation, and obtain the category label of each image as the image classification result.
4. The neural network model knowledge distillation system based on automatic gradient mixing according to claim 3, characterized in that: The network size of the teacher network is larger than the network size of the student network.
5. A storage medium for storing a program for executing the neural network model knowledge distillation method based on automatic gradient mixing as described in claim 1 or 2.
6. A client for use in the neural network model knowledge distillation system based on automatic gradient mixing as described in claim 3 or 4.