Self-distillation of multi-output architectures based on stage training and attention fusion
By adopting a multi-export architecture of stage training and attention fusion in knowledge distillation, the problem of low knowledge transfer efficiency between teachers and students' models is solved, and more efficient knowledge transfer and performance improvement of student models is achieved.
Patent Information
- Application Number
- CN202411172305.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-26
AI Technical Summary
In the existing knowledge distillation method, the knowledge transfer efficiency between the teacher model and the student model is inefficient, especially when the model gap is large, it is difficult for the student model to effectively obtain help from the teacher model.
Using a multi-exit architecture self-distillation method based on stage training and attention fusion, the teacher model is divided into multiple outlet branches, and the student model is trained through the total loss function and attention fusion algorithm to gradually reduce the ability gap between teachers and students. , and integrate attention information from the intermediate model.
It improves the efficiency of knowledge transfer, enables the student model to obtain valuable information from the teacher model more effectively, reduces the difficulty of transferring knowledge by the teacher model, and significantly improves accuracy in image classification tasks.
Smart Images

Figure CN118690831B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of knowledge distillation, and in particular to a multi-export architecture self-distillation method based on stage training and attention fusion. Background Art
[0002] Artificial intelligence is the most revolutionary and disruptive technology field in the world today. However, limited by weak computing power and data acquisition capabilities, early artificial intelligence research made relatively limited contributions to changing the production and lifestyle of human society. With the arrival of more powerful GPU computing power and the increased feasibility of large-scale data collection, neural networks began to exert their true power. Since 2012, deep learning methods have gradually dominated the accuracy benchmarks of various computer vision and natural language processing related tasks, and have been widely used in image classification, target monitoring, machine translation and other fields, making the development of the global artificial intelligence industry reach a cyclical peak. However, this surge in algorithm performance is usually at the expense of increasing model complexity. On the one hand, high complexity leads to a long overall response time of the model, which is difficult to accept in application scenarios with high timeliness requirements such as intelligent driving and intelligent medical care. On the other hand, these trained models carry a large number of redundant parameters, which will far exceed the limited computing resource capacity of edge devices. In order to solve the problem of model parameter redundancy, researchers have done a lot of work on the lightweighting of the network. Existing solutions can be roughly divided into three categories: parameter pruning, parameter quantization and knowledge distillation. The core idea of parameter pruning is to remove meaningless parameters by developing different evaluation strategies. However, many methods still rely on specific hardware or implementation, so there are certain limitations in implementation and it is difficult to be widely implemented. Parameter quantization can increase the computation speed by reducing the number of bits. However, quantizing weights usually makes it more difficult for neural networks to converge. In contrast, knowledge extraction can directly improve the accuracy of small networks through multi-supervisory objectives, which makes its application very flexible and therefore has broad optimization prospects.
[0003] Knowledge distillation is an important method in the field of neural network compression. The core idea is to let a weak but compact student model improve its performance by learning the soft objectives of an advanced but large teacher model. Knowledge transfer is not restricted by the model structure and therefore has high flexibility. Existing knowledge distillation has the following disadvantages: the knowledge transfer efficiency between different models is low, and it is difficult for the student model to effectively get help from the teacher model, especially when the gap between the teacher model and the student model is large.
[0004] Therefore, it is necessary to provide a multi-export architecture self-distillation method based on stage training and attention fusion to improve the efficiency of knowledge transfer and improve the performance of the knowledge self-distillation framework. Summary of the invention
[0005] The present invention provides a multi-export architecture self-distillation method based on stage training and attention fusion, including: dividing a teacher model into multiple export branches according to depth, wherein, among the multiple export branches, the branch with the deepest depth is the teacher model, the export branch with the shallowest depth is the student model, and the remaining export branches are intermediate models, and the teacher model and the student model are both used for image classification; establishing a total loss function; based on the multiple export branches and the attention fusion algorithm, training the student model, based on the total loss function, calculating the total loss, and based on the total loss, optimizing the student model until the student model meets preset conditions.
[0006] Furthermore, dividing the teacher model into multiple branches according to the depth includes: dividing the teacher model into multiple exit branches according to the number of convolutional layers.
[0007] Furthermore, based on the multiple exit branches and the attention fusion algorithm, the student model is trained, including: dividing the training cycle into multiple stages; in the multiple stages, based on the multiple exit branches and the attention fusion algorithm, the student model is cascaded trained.
[0008] Furthermore, in the multiple stages, the student model is cascade trained based on the multiple exit branches, including: in the multiple stages, the student model is cascade trained based on the multiple exit branches and the attention fusion algorithm, and each of the intermediate models is cascade trained.
[0009] Furthermore, the total loss function is: , where L is the total loss, is the cross entropy loss of the i-th export branch, is a function that takes values of 0 or 1, is the distillation loss between the i-th outlet branch and the j-th outlet branch, is the feature map loss of the i-th export branch, M is the total number of export branches, and are weights, and are index variables used for summation. For stage.
[0010] Furthermore, the cross entropy loss of the i-th export branch is calculated based on the following formula: , where N is the total number of samples and G is the total number of image categories. For the i The samples correspond to j The true labels of the image categories, Predict the i-th exit branch iThe samples belong to j The probability of the image category, k is the index variable used for summation, and g is the index variable used for summation.
[0011] Furthermore, the distillation loss of the i-th outlet branch and the j-th outlet branch is calculated based on the following formula: , where k is the sample input to the i-th outlet branch and the j-th outlet branch, T is the temperature parameter, For the j-th outlet branch, given input k and temperature parameters T Next, the predicted probability of the g-th image category is, For the i-th outlet branch, given input k and temperature parameters T Next, the predicted probability for the g-th image category.
[0012] Furthermore, the feature map loss of the i-th outlet branch is calculated based on the following formula: ,in, is a hyperparameter, is the feature map output by the teacher model, is the feature map output by the i-th export branch, It means calculating the k-order Euclidean distance between the feature map output by the teacher model and the feature map output by the i-th output branch.
[0013] Furthermore, based on the multiple exit branches and the attention fusion algorithm, the student model is cascade trained, including: for each of the exit branches, calculating the attention information of the exit branch through the attention module; integrating the attention information of each of the exit branches into the student branch, and cascading training the student model.
[0014] Furthermore, the attention information of each of the exit branches is integrated based on the following formula: ,in, is the fused attention information, is the attention information of the i-th exit branch, M is the total number of exit branches, is the weighted feature map output by the spatial attention module, is the weighted feature map output by the channel attention module, F is the input feature map, is element-wise multiplication.
[0015] Compared with the prior art, the multi-export architecture self-distillation method based on stage training and attention fusion provided by the present invention has at least the following beneficial effects:
[0016] Under the framework of multi-exit architecture, a progressive student training method is proposed, which can effectively reduce the ability gap between teachers and students and achieve the purpose of improving the efficiency of knowledge transfer. Under the framework of multi-exit architecture, an attention integration method is proposed, which can make the student model pay attention to the attention information of the teacher model at different levels of abstraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:
[0018] Figure 1 is a flowchart of a multi-export architecture self-distillation method based on stage training and attention fusion according to some embodiments of this specification;
[0019] Figure 2 It is a flowchart of training a student model according to some embodiments of this specification. DETAILED DESCRIPTION
[0020] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.
[0021] Knowledge distillation can be understood as a special type of transfer learning, the difference being that transfer learning usually refers to transferring the parameters of a training model to a new model, emphasizing the impact of one type of learning on another. Knowledge distillation refers to transferring knowledge from a large model (i.e., the teacher model) to a small model (i.e., the student model), emphasizing the impact of one type of understanding on another type of understanding under the same learning task. In this mode, in addition to simply learning which category a sample belongs to from label information, the student model can also learn the degree of similarity between samples and different categories from the transferred knowledge (such as predicted probability).
[0022] To formulate this process, given an input sample , and a label vector , the student model and the teacher model will output logical vectors respectively and , as shown below, after the softmax function processed by temperature T, the teacher model and the student model will give their respective predicted probabilities of sample X:
[0023]
[0024] in, are the respective predicted probabilities of sample X predicted by the teacher model, The respective predicted probabilities of sample X predicted by the student model.
[0025] Although the student model is a model with a small number of parameters, it still relies on a larger pre-trained teacher model, which incurs additional training costs. Therefore, it is necessary to provide a multi-output architecture self-distillation method based on stage training and attention fusion to solve this problem.
[0026] Figure 1 is a flowchart of a multi-export architecture self-distillation method based on stage training and attention fusion according to some embodiments of this specification, such as Figure 1 As shown, the multi-output architecture self-distillation method based on stage training and attention fusion can include the following steps.
[0027] Step 110, dividing the teacher model into multiple export branches according to the depth.
[0028] Among them, the teacher model and the student model are used for image classification. Among the multiple export branches, the deepest branch is the teacher model, the shallowest export branch is the student model, and the remaining export branches are intermediate models. The performance of these export branches tends to improve with the increase of depth.
[0029] For example, the teacher model is divided into four export branches according to the depth, namely export branch 1, export branch 2, export branch 3 and export branch 4, among which the depths of export branch 1, export branch 2, export branch 3 and export branch 4 gradually increase, export branch 1 is the student model, export branch 4 is the teacher model, and the rest are intermediate models.
[0030] In some embodiments, the teacher model can be divided into multiple exit branches according to the number of convolutional layers.
[0031] Step 120, establishing a total loss function.
[0032] In some embodiments, the total loss function is:
[0033]
[0034] Where L is the total loss, is the cross entropy loss of the i-th export branch, is a function with a value of 0 or 1, which activates the corresponding transfer path according to the stage. For example, in the first stage, exit branch 1 learns from exit branch 2, then The value of is 1, while the other stages The value of is 0. is the distillation loss between the i-th outlet branch and the j-th outlet branch, is the feature map loss of the i-th export branch, M is the total number of export branches, and are weights, and are index variables used for summation. For stage.
[0035] The cross entropy loss of the i-th exit branch is calculated based on the following formula:
[0036]
[0037] Among them, N is the total number of samples, G is the total number of image categories, For the i The samples correspond to j The true labels of the image categories, Predict the i-th exit branch i The samples belong to j The probability of the image category.
[0038] The distillation loss of the i-th outlet branch and the j-th outlet branch is calculated based on the following formula:
[0039]
[0040] Where k is the sample input to the i-th outlet branch and the j-th outlet branch, T is the temperature parameter, For the j-th exit branch at a given input k and temperature parameters T Next, the predicted probability of the g-th image category is, For the i-th exit branch at a given input k and temperature parameters T Below, for the predicted probability of the g-th image category, k is the index variable used for summation, and g is the index variable used for summation.
[0041] The feature map loss of the i-th outlet branch is calculated based on the following formula:
[0042]
[0043] in, is a hyperparameter, is the feature map output by the teacher model, is the feature map output by the i-th export branch, It means calculating the k-order Euclidean distance between the feature map output by the teacher model and the feature map output by the i-th output branch.
[0044] Step 130, based on multiple export branches and the attention fusion algorithm, train the student model, calculate the total loss based on the total loss function, and optimize the student model based on the total loss until the student model meets the preset conditions.
[0045] If the teacher and the student are trained simultaneously, this reduces the additional cost of pre-training the teacher model separately, however, this training method ignores the importance of the intermediate branches. When the ability gap between the teacher model and the student model is too large, the student cannot get help from the knowledge distilled from the teacher.
[0046] Based on the fact that the performance of each branch increases with the depth, in some embodiments, the student model is trained based on multiple exit branches and an attention fusion algorithm, including:
[0047] Divide the training cycle into multiple phases;
[0048] In multiple stages, the student model is cascaded trained based on multiple export branches and attention fusion algorithm. Figure 2 is a flow chart of training a student model according to some embodiments of this specification, such as Figure 2 As shown, for example, the entire training cycle is divided into no more than M-1 different stages according to the number of cycles. The earliest exit branch (i.e., the student model) no longer learns from the deepest exit branch (i.e., the teacher model), but only learns one exit branch at any given stage. For example, in the first stage, exit branch 1 learns from exit branch 2, in the second stage, exit branch 1 learns from exit branch 3, in the third stage, exit branch 1 learns from exit branch 4, and so on. The advantage of this step-by-step training method is that it reduces the ability gap between the teacher and the student. Therefore, the difficulty of obtaining valuable information from the knowledge transferred from the teacher model is reduced.
[0049] In some embodiments, the student model is cascaded trained based on multiple exit branches in multiple stages, including:
[0050] In multiple stages, based on multiple exit branches and the attention fusion algorithm, the student model is cascaded trained, and each intermediate model is cascaded trained. Specifically, the intermediate model can sequentially learn deeper exit branches in multiple stages of the training cycle.
[0051] In some embodiments, based on multiple exit branches and an attention fusion algorithm, a student model is cascaded trained, including:
[0052] For each exit branch, the attention information of the exit branch is calculated through the attention module;
[0053] The attention information of each exit branch is integrated into the student branch, and the student model is trained in cascade.
[0054] Since the attention mechanism refers to the human visual mechanism and has excellent visualization capabilities, it is often used to analyze the interpretability of deep models. Unlike existing methods that construct loss functions through attention maps and narrow the attention map gap, in the multi-export architecture self-distillation method based on stage training and attention fusion, the attention information of each export branch can be considered valid for the entire structure.
[0055] Preferably, the attention module may be a convolutional block attention module (CBAM).
[0056] In some embodiments, the attention information of each exit branch is integrated based on the following formula:
[0057]
[0058] in, is the fused attention information, is the attention information of the i-th exit branch, M is the total number of exit branches, is the weighted feature map output by the spatial attention module, is the weighted feature map output by the channel attention module, F is the input feature map, is element-wise multiplication, is the global average pooling, is the global maximum pooling, The two-dimensional feature map generated by applying average pooling of the channel dimension, The two-dimensional feature map generated by applying the maximum pooling of the channel dimension, is a multi-layer perceptron, is the Sigmoid activation function.
[0059] Different from the existing methods that treat attention as a feature and incorporate its distance into the loss function, the multi-export architecture self-distillation method based on stage training and attention fusion believes that the attention information in all export branches is valuable, and there is no right or wrong, because the features they extract belong to different levels of abstraction. Therefore, the attention fusion method is adopted to integrate all deep-level attention information into the student model, which enables the student model to focus on valuable information from micro to macro levels.
[0060] The preset condition may be that the ability gap between the teacher model and the student model is less than an ability gap threshold, etc.
[0061] To verify the effectiveness of the proposal, the proposed multi-exit architecture self-distillation method based on stage training and attention fusion (hereinafter referred to as "this method") is evaluated on two datasets. Experimental results show that this method is competitive in terms of accuracy. We first introduce the details of the dataset and experimental settings, and then discuss and analyze the experimental results.
[0062] The experimental dataset contains 50,000 training images and 10,000 test images, divided into 10 categories. Each category includes 6,000 images, and the size of each color image is 32×32. The samples are divided into 100 subclasses, and each subclass belongs to one of the 10 categories. Each category contains 600 images, including 500 training images and 100 test images, and the size of each color image is 32×32.
[0063] In this experiment, three models ResNet-18, ResNet-50, and ResNet-101 were selected for experiments on two datasets. First, the weight parameters were trained and adjusted by a stochastic gradient descent optimizer with momentum. The learning rate was initially set to 1e-1 and then decayed to 1e-4 using the MultiStepLR strategy. Then, all models were trained for 200 cycles with a batch size of 128. Finally, all experiments were implemented using pytorch using three GeForce GTX 1080Ti with 11GB of memory.
[0064] The core idea of stage training is to reduce the ability gap between the teacher model and the student model and improve the performance of knowledge distillation by letting the first output (student model) learn different levels of outputs at different times. However, in specific implementations, there are usually different strategies for the specific learning order.
[0065] For example, for a network with four exit branches, the performance ranking of exits 1-4 is usually exit branch 1 ≤ exit branch 2 ≤ exit branch 3 ≤ exit branch 4. A sequential learning strategy for the student model (exit branch 1) is to learn slightly stronger exit branches 2 and 3 in the early and middle stages, respectively. The teacher model (exit branch 4) is the learning target in the late stage. In contrast, the reverse learning strategy is to learn the teacher (exit branch 4) in the early stage, and then learn the weaker exit branches 3 and 2 in the middle and late stages. When the ability gap of the intermediate exits is not large, the intermediate learning process can be simplified. Table 1 shows the results of different learning strategies for ResNet-18 under the dataset. The results show that the forward sequential learning strategy, in which the student model (exit branch 1) learns the second, third, and fourth exit branches in the early, middle, and late stages, respectively, enables the student model to achieve the best accuracy of 68.15%.
[0066] Table 1
[0067] Learning sequence Export branch 1 (%) Export branch 2(%) Export branch 3(%) Export branch 4(%) 4 67.56 74.06 77.82 78.28 2-4 68.08 72.60 74.12 74.33 3-4 67.70 72.36 74.06 74.07 2-3-4 68.15 72.56 73.41 73.83 4-3-2 67.59 72.29 73.88 73.66
[0068] In order to further verify the effectiveness of the attention integration idea proposed in this specification, attention integration and forward sequential learning strategies are combined based on the above experimental results, and the results are compared with existing related methods. The experimental results are shown in Table 2 and Table 3 respectively.
[0069]
[0070]
[0071] As shown in Table 2, when the model is ResNet-18, the first exit branch (student model) obtains the best accuracy. When the model is ResNet-50 and ResNet-101, although this method is not the best, it is close to the performance of other methods on the entire model using only the first branch. In Table 3, it can be noted that on three different models, this method is the best among all substructure methods and is only lower than the non-substructure method experiment conducted on the ResNet-18 model.
[0072] The above experimental results show that this method can make the student model that inherits only one-fourth of the original teacher model achieve good performance, and even exceed some results obtained by other methods on the teacher model.
[0073] In summary, in order to improve the learning efficiency of the student model, this method introduces a stage training method to guide the student to learn different branches at different training stages. In addition, in order to more reasonably utilize the increasingly abstract nature of the features extracted by the convolutional neural network as the network depth increases, this method integrates the attention information from small to large to the student. Experimental results show that this method can achieve better results on two different datasets.
[0074] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.
Claims
1. Multi-exit architecture self-distillation method based on stage training and attention fusion, It is characterized in that include: Dividing the teacher model into multiple export branches according to the depth, wherein, among the multiple export branches, the branch with the deepest depth is the teacher model, the export branch with the shallowest depth is the student model, and the remaining export branches are intermediate models, and both the teacher model and the student model are used for image classification; Establish the total loss function; Based on the multiple export branches and the attention fusion algorithm, the student model is trained, based on the total loss function, the total loss is calculated, and based on the total loss, the student model is optimized until the student model meets the preset conditions; Wherein, based on the multiple export branches and the attention fusion algorithm, training the student model includes: Divide the training cycle into multiple phases; In the multiple stages, for each of the exit branches, the attention information of the exit branch is calculated by the attention module, the attention information of each of the exit branches is integrated into the student branch, the student model is cascaded trained, and each of the intermediate models is cascaded trained, and the intermediate models sequentially learn deeper exit branches in multiple stages of the training cycle; The total loss function is: Among them, L is the total loss, L CE (i) is the cross entropy loss of the i-th export branch, θ i,j,t is a function with a value of 0 or 1, which is used to activate the corresponding transfer path according to the stage. KD (i,j) is the distillation loss between the i-th outlet branch and the j-th outlet branch, L F (i,M) is the feature map loss of the i-th outlet branch, M is the total number of outlet branches, α 1 , α 2 and α 3 are weights, i and j are index variables used for summation, and t is the stage; The cross entropy loss of the i-th exit branch is calculated based on the following formula: Among them, N is the total number of samples, G is the total number of image categories, and y kg is the true label of the k-th sample corresponding to the g-th image category, p kg Predict the probability that the kth sample belongs to the gth image category for the i-th export branch, where k is the index variable used for summation and g is the index variable used for summation; The distillation loss of the i-th outlet branch and the j-th outlet branch is calculated based on the following formula: Where k is the sample input to the i-th outlet branch and the j-th outlet branch, T is the temperature parameter, and p kg (k,T) is the predicted probability of the j-th export branch for the g-th image category under the given input k and temperature parameter T, q kg (k,T) is the predicted probability of the g-th image category of the i-th export branch under the given input k and temperature parameter T; The feature map loss of the i-th outlet branch is calculated based on the following formula: Among them, β is a hyperparameter, F T is the feature map output by the teacher model, F S i is the feature map output by the i-th export branch, It means calculating the k-order Euclidean distance between the feature map output by the teacher model and the feature map output by the i-th output branch.
2. The multi-exit architecture self-distillation method based on stage training and attention fusion according to claim 1, It is characterized in that The dividing the teacher model into multiple branches according to the depth includes: dividing the teacher model into multiple export branches according to the number of convolutional layers.
3. The multi-exit architecture self-distillation method based on stage training and attention fusion according to claim 1, It is characterized in that After the softmax function processed by temperature T, the teacher model and the student model give their respective prediction probabilities.
4. The multi-exit architecture self-distillation method based on stage training and attention fusion according to claim 3, It is characterized in that The teacher model output is: Among them, the logical vector output by the teacher model is p i (x,T) is the predicted probability output by the teacher model, T is the temperature, and N is the total number of classes.
5. The multi-exit architecture self-distillation method based on stage training and attention fusion according to claim 4, It is characterized in that The output of the student model is: Among them, the logical vector output by the student model is q i (x,T) is the predicted probability output by the student model.
6. The multi-exit architecture self-distillation method based on stage training and attention fusion according to claim 1, It is characterized in that The preset condition is that the ability gap between the teacher model and the student model is less than the ability gap threshold.
7. The multi-exit architecture self-distillation method based on stage training and attention fusion according to claim 1, It is characterized in that The attention module is a convolutional block attention module.
8. The multi-exit architecture self-distillation method based on stage training and attention fusion according to claim 1, It is characterized in that The attention information of each of the exit branches is integrated based on the following formula: Among them, F′ out is the fused attention information, F outi is the attention information of the i-th exit branch, M is the total number of exit branches, is the weighted feature map output by the spatial attention module, M c (F) is the weighted feature map output by the channel attention module, F is the input feature map, is element-wise multiplication.
9. The multi-exit architecture self-distillation method based on stage training and attention fusion according to claim 8, It is characterized in that The weighted feature map output by the channel attention module is obtained based on the following formula: in, is the global average pooling, is the global maximum pooling, MLP is the multi-layer perceptron, and σ is the Sigmoid activation function.
10. The multi-exit architecture self-distillation method based on stage training and attention fusion according to claim 8, It is characterized in that The spatial attention module generates a weighted feature map based on the following formula: Among them, M s (F) is the feature weighting function, The two-dimensional feature map generated by applying average pooling of the channel dimension, The two-dimensional feature map generated by applying the maximum pooling of the channel dimension, σ is the Sigmoid activation function.
