Knowledge distillation method based on heterogeneous branch fusion
By adopting the method of fusion of heterogeneous branches in knowledge distillation, the problem of difficulty learning and multi -branch architecture of traditional knowledge distillations, and the effect of improving model classification accuracy.
Patent Information
- Application Number
- CN202411445094.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-16
- Publication Date
- 2025-01-24
AI Technical Summary
In the traditional knowledge distillation method, it is difficult for students to fully learn the knowledge provided by teachers, and the teachers' fixed knowledge cannot fully improve students' generalization ability, and multi-branch structures lead to homogenization problems.
Using the knowledge distillation method based on heterogeneous branches, through the pre -training weight of the teacher model, the student model of the heterogeneous branch is constructed, and the characteristic fusion module is used to gradually integrate the branch characteristics. Come to train the student model.
It effectively overcomes the problem of difficulty in learning when teachers have single knowledge and a large gap in ability between teachers and students, alleviates the homogenization problem caused by multi-branch architecture, and improves the classification accuracy of the model without increasing additional costs.
Smart Images
Figure CN119358635A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a knowledge distillation method based on heterogeneous branch fusion. Background Art
[0002] Deep neural networks have demonstrated powerful performance in many tasks in the field of computer vision, such as image classification, object detection, and semantic segmentation. However, these high-performance models are often accompanied by large parameters and computational costs, which poses a considerable challenge in many application scenarios that pursue lightweight models and efficient deployment. In order to balance the number of parameters and performance of the model, knowledge distillation technology came into being. As an efficient knowledge transfer strategy, knowledge distillation can extract implicit knowledge from large and complex models (teacher models) and transfer it to more compact lightweight models (student models), effectively improving the performance of these lightweight models in actual tasks.
[0003] Existing knowledge distillation methods can be roughly divided into offline knowledge distillation and online knowledge distillation. Offline knowledge distillation adopts a two-stage training method: first, a large teacher model is pre-trained, and then the extracted knowledge is transferred to a smaller student model to help the student learn the complex knowledge in the teacher model. Online knowledge distillation adopts a single-stage training method, which directly optimizes the target model by continuously updating knowledge during the training process, so that the student model can make full use of the rich information from multiple outputs.
[0004] In traditional knowledge distillation methods, it is difficult for students to fully learn the knowledge provided by teachers because there is a large ability gap between the converged teacher model and the student model trained from scratch. In addition, the teacher's fixed knowledge cannot fully improve the generalization ability of students, and the real-time information of the training process can be used as a source of knowledge. The introduction of online knowledge distillation strategies provides new ideas for solving these problems. Summary of the invention
[0005] The purpose of the present invention is to provide a knowledge distillation method based on heterogeneous branch fusion to overcome the limitations of traditional knowledge distillation, such as the single teacher knowledge and the difficulty in learning when there is a large gap in the ability between teachers and students, and to alleviate the homogenization problem caused by the multi-branch architecture.
[0006] To achieve the above object, the present invention provides the following technical solution: a knowledge distillation method based on heterogeneous branch fusion, comprising at least the following steps:
[0007] Step 1: Get the original data set;
[0008] Step 2: Divide the original data set into training set and test set, and perform preprocessing;
[0009] Step 3: Use the training set to pre-train the teacher model and save the trained teacher model;
[0010] Step 4: Load the pre-trained weights, build the corresponding student model, and train the student model;
[0011] Step 5: In the inference phase, only the target branch is kept as the baseline model.
[0012] Furthermore, the step 4 at least includes the following steps:
[0013] Step 4-1: Freeze the pre-trained weights of the teacher network model;
[0014] Step 4-2: Construct a student model based on heterogeneous branches and use a progressive feature fusion module to gradually fuse branch features;
[0015] Step 4-3: Train the student model using the co-learning objective and teacher guidance as the distillation loss;
[0016] Further, the step 4-2 at least includes the following steps:
[0017] The student model is a multi-branch architecture, where each branch shares parameters at low-level layers and has independent parameters at high-level layers, acting as multiple students trained in parallel;
[0018] Name the first branch Branch-1 and set it as the target branch, which together with the lower-level layers constitutes the baseline model;
[0019] Use B to represent the total number of branches, b represents the branch number b∈(1,2,...,B), and represent the feature generated by the bth branch as F b , the fusion feature generated by the progressive fusion module is represented as F b m ;
[0020] On the basis of the target branch, auxiliary blocks with the number of channels (b-1)·C are gradually added to construct the heterogeneous branches in the student model, where C is the number of channels in the high-level layer;
[0021] Assume that each progressive fusion feature module has two input entries and one output exit;
[0022] The feature map of the first branch, i.e. the target branch, is used as the fusion feature of the first input, i.e. F1 m =F1;
[0023] In the progressive feature fusion module, the feature F of the b-th branch is first b And the fusion features of the previous layer Perform initial fusion;
[0024] Then, the initial fused features are sent to two paths for processing:
[0025] One path contains two convolutional layers with 1x1 kernels to capture local information;
[0026] The other path captures global information through an additional global pooling layer;
[0027] The feature information captured by the two paths is activated by the Sigmoid function to obtain the weight score, and is respectively compared with the feature F b , Multiply them together and use the parameter w to adjust the importance of the branch, and finally get the fusion features at this level
[0028] Furthermore, the expression using the progressive feature fusion module is:
[0029]
[0030] Where D(·) represents the fusion function.
[0031] Furthermore, the step 4-3 is specifically as follows:
[0032] By adding a softmax function with a hyperparameter temperature τ to soften the logits, a knowledge distillation loss function is constructed;
[0033] The softened outputs of the teacher and the student are and The distillation loss L is calculated by Kullback-Leibler divergence kl ;
[0034] Combine the cross entropy loss function of the true label and the student prediction, as well as the balance parameter, to construct a complete knowledge distillation loss function L kd ;
[0035] According to the knowledge distillation loss function, two losses, collaborative learning objective and teacher guidance, are constructed. The weight of knowledge distillation in the training process is controlled by the ramp-up function, and the total loss is finally obtained to train the student model.
[0036] The collaborative learning objectives and teacher guidance losses are specifically:
[0037] The collaborative learning goal consists of two parts:
[0038] (1) Fusion features Output and get FL;
[0039] (2) Integrate the branch logits to get EL, and then soften FL and EL to get and and calculate distillation losses;
[0040] Teacher guidance is achieved by calculating the distillation loss of the teacher model and each branch. The softened outputs of the teacher and the student are and
[0041] Furthermore, the collaborative learning objective loss function and the teacher guidance loss function are expressed as:
[0042]
[0043]
[0044] Where L cl represents the collaborative learning objective loss, L tg Denotes the teacher guidance loss. τ 2 Represents the weight parameter, which is used to balance the cross entropy loss and distillation loss;
[0045] The complete loss function is expressed as:
[0046]
[0047] Where L total Represents the complete loss function, L ce represents the cross entropy loss function. w(i) represents the ramp-up function, which is used to adjust the distillation loss weights at different epochs.
[0048] Furthermore, the step 5 at least includes the following steps:
[0049] In the student inference phase, the pre-trained teacher model and the branches with added auxiliary blocks are removed, and only the target branch is retained as the original model architecture, with no additional cost added.
[0050] Compared with the prior art, the present invention has the following beneficial effects:
[0051] The present invention can overcome the limitations of teachers' single knowledge and difficulty in learning when there is a large gap in the abilities of teachers and students, alleviate the homogenization problem caused by the multi-branch architecture, and effectively improve the classification accuracy of the model without increasing additional costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0053] Figure 1 is a flowchart of an embodiment of the present invention;
[0054] Figure 2 It is a framework diagram of the overall knowledge distillation method of an embodiment of the present invention;
[0055] Figure 3 A diagram of a progressive feature fusion module according to an embodiment of the present invention;
[0056] Figure 4 is a graph of inter-branch diversity analysis according to an embodiment of the present invention;
[0057] Figure 5 A bar chart showing the impact of different loss components on performance of an embodiment of the invention. DETAILED DESCRIPTION
[0058] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.
[0059] See also Figure 1 , a knowledge distillation method based on heterogeneous branch fusion, comprising at least the following steps:
[0060] Step 1: Get the original data set;
[0061] In this embodiment, the original data set obtained is the CIFAR-100 data set, which contains 60,000 RGB images of 32×32 size in 100 categories.
[0062] Step 2: Divide the original data set into training set and test set, and perform preprocessing;
[0063] In this embodiment, the training set and test set are 50k and 10k images respectively, which are processed and normalized by channel mean and standard deviation, and standard data enhancement operations are adopted.
[0064] Step 3: Use the training set to pre-train the teacher model and save the trained teacher model;
[0065] In this embodiment, the teacher models include resnet56, resnet110, resnet32×4, WRN-40-2, and ResNet50. The experimental parameters for training the teacher model are set as follows: BatchSize is 64, Epoch is 240. The initial learning rate is set to 0.05, and the gradient learning rate method is used. The learning rate is divided by 10 when Epoch is 150, 180, and 210. The optimizer uses the machine gradient descent method, the weight decay is 5e-4, and the momentum is 0.9.
[0066] Step 4: Freeze the pre-trained weights of the teacher network model, build a student model based on heterogeneous branches, use the progressive feature fusion module to gradually fuse branch features, and use the collaborative learning objective and teacher guidance as the distillation loss to train the student model;
[0067] In this embodiment, the student models include resnet, WRN, ShuffleNet and MobileNet. In the experimental parameters for training the student model, the initial learning rate of ShuffleNet and MobileNet is 0.02, the initial learning rate used by other models is 0.05, and the other parameter settings are the same as those for training the teacher model.
[0068] Step 4 includes:
[0069] Step 4-1: Freeze the pre-trained weights of the teacher network model;
[0070] Step 4-2: Construct a student model based on heterogeneous branches and use a progressive feature fusion module to gradually fuse branch features;
[0071] Step 4-3: Train the student model using the co-learning objective and teacher guidance as the distillation loss;
[0072] Specifically, during the training of the student model, the pre-trained weights of the teacher model obtained in step 3 are frozen to ensure that the weights of the teacher model do not change during the training process.
[0073] like Figure 2 As shown in the figure, the student model is a multi-branch architecture, which constructs heterogeneous branches by adding auxiliary blocks on the basis of the target branch, and uses a progressive feature fusion module to gradually fuse the features of each branch.
[0074] Each branch shares parameters in the low-level layers and has independent parameters in the high-level layers, acting as multiple students trained in parallel. The first branch is named Branch-1 and set as the target branch, which together with the low-level layers constitutes the baseline model.
[0075] Use B to represent the total number of branches, b represents the branch number b∈(1,2,...,B), and represent the feature generated by the bth branch as F b , the fusion feature generated by the progressive fusion module is expressed as
[0076] The same construction between branches makes each student's ability roughly the same, which easily leads to homogenization problems and affects performance during training. Therefore, the designed heterogeneous branches construct students with different abilities by adding additional auxiliary blocks on the basis of the target branch to capture differentiated feature maps, providing a basis for the diversity of collaborative learning.
[0077] For example Figure 3 As shown in the progressive feature fusion module, the feature F of the b-th branch is first b And the fusion features of the previous layer Perform initial fusion.
[0078] The initial fused features are sent to two paths for processing: one path contains two convolutional layers with 1x1 convolution kernels to capture local information; the other path captures global information through an additional global pooling layer.
[0079] The feature information captured by the two paths is activated by the Sigmoid function to obtain the weight score, and is respectively compared with the feature F b , Multiply them together and use the parameter w to adjust the importance of the branch, and finally get the fusion features at this level
[0080] The expression of the progressive feature fusion module is:
[0081]
[0082] Therefore, the designed heterogeneous branches not only increase the network depth, but also capture the feature information of different receptive fields by doubling the number of channels, which provides a rich feature basis for the fusion of global and local information. Gradually fusing in a progressive manner can effectively improve the fusion quality compared to fusing the features of B branches at the same time.
[0083] And again Figure 2 The loss calculation part shown uses the collaborative learning objective and teacher guidance as the distillation loss to train the student model, specifically:
[0084] The logits are softened by adding a softmax function with a hyperparameter temperature τ, and a knowledge distillation loss function is constructed; the softened outputs of the teacher and the student are and The distillation loss L is calculated by Kullback-Leibler divergence kl ;
[0085] Combine the cross entropy loss function of the true label and the student prediction, as well as the balance parameter, to construct a complete knowledge distillation loss function L kd ;
[0086] According to the knowledge distillation loss function, two losses, collaborative learning objective and teacher guidance, are constructed. The weight of knowledge distillation in the training process is controlled by the ramp-up function, and the total loss is finally obtained to train the student model.
[0087] The collaborative learning objectives and teacher guidance losses are specifically:
[0088] By adding a softmax function with a hyperparameter temperature τ to soften the logits, a knowledge distillation loss function is constructed;
[0089] The softened outputs of the teacher and the student are and The distillation loss L is calculated by Kullback-Leibler divergence kl ;
[0090] Combine the cross entropy loss function of the true label and the student prediction, as well as the balance parameter, to construct a complete knowledge distillation loss function L kd ;
[0091] According to the knowledge distillation loss function, two losses, collaborative learning objective and teacher guidance, are constructed. The weight of knowledge distillation in the training process is controlled by the ramp-up function, and the total loss is finally obtained to train the student model.
[0092] The collaborative learning objectives and teacher guidance losses are specifically:
[0093] The collaborative learning objective consists of two parts: (1) integrating the features Output to get FL; (2) Integrate the branch logits to get EL. Then, soften FL and EL to get and And calculate the distillation loss.
[0094] Teacher guidance is achieved by calculating the distillation loss of the teacher model and each branch. The softened outputs of the teacher and the student are and
[0095] The collaborative learning objective and teacher guidance loss function are expressed as:
[0096]
[0097]
[0098] Where L cl represents the collaborative learning objective loss, L tg Denotes the teacher guidance loss. τ 2 Represents the weight parameter used to balance the cross entropy loss and distillation loss.
[0099] The complete loss function is expressed as:
[0100]
[0101] Where L total Represents the complete loss function, L ce represents the cross entropy loss function. w(i) represents the ramp-up function, which is used to adjust the distillation loss weights at different training stages.
[0102] Therefore, the teacher's guidance provides students with reliable optimization directions in the early stages of training and delivers diverse and easy-to-learn knowledge through collaborative learning goals, which can provide a diverse source of knowledge and thus overcome the limitations of the teacher's single knowledge and learning difficulties.
[0103] Step 5: In the inference phase, only the target branch is retained as the baseline model.
[0104] The pre-trained teacher model and the branches with added auxiliary blocks are removed, and only the target branch is retained as the original model architecture without any additional cost increase.
[0105] To verify the effectiveness of the proposed knowledge distillation method, the experimental results are shown in Table 1.
[0106] Table 1 Experimental results of comparing knowledge distillation methods on CIFAR-100
[0107]
[0108]
[0109] The experimental results in Table 1 show that the performance of the heterogeneous branch fusion knowledge distillation method (HBFKD) of the present invention is at least 3% higher than the baseline student trained with cross entropy under various model architectures. Compared with the SOTA method, HBKFD achieves the best performance when the teacher and student use the same architecture. Using different teacher-student architectures, the combination of ResNet32x4 and ShuffleNetV2 achieves comparable performance to ReviewKD.
[0110] Next, we compare four online knowledge distillation methods and measure diversity using the Euclidean distance between each pair of branches to illustrate their effectiveness in alleviating the homogenization problem.
[0111] Table 2 Experimental results of comparing online knowledge distillation methods on CIFAR-100
[0112]
[0113] HBFKD (OKD) is used to represent the version after removing the teacher model. Compared with OKDDip, it is improved by 2.36% when using ResNet8x4 and 1.29% when using WRN16-2.
[0114] like Figure 5 As shown in the figure, the diversity comparison results of the four methods show that after several rounds of optimization, the diversity increases rapidly, HBFKD can remain stable, while OKDDip, ONE and CL show a downward trend until the learning rate decreases and then rises again. Throughout the training phase, the diversity of HBFKD is significantly higher than that of the other three methods, which shows that the heterogeneous branch designed in this paper can effectively alleviate the homogenization problem.
[0115] Next, the influence of different loss components on the distillation effect is analyzed. Figure 5 As shown, the total loss L of the collaborative learning objective cl By L fl and L el Composition, using L cl The best performance was achieved during training. cl Training is done using L fl Improved by 0.52% compared to L el The improvement is 0.86%, and it is better than that of only using teacher guidance. tg The improvement is 1.18%. This shows that the two proposed losses can effectively improve the generalization ability of the student model.
[0116] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
Claims
1. A knowledge distillation method based on heterogeneous branch fusion, characterized in that: At least the following steps are included: Step 1: Get the original data set; Step 2: Divide the original data set into training set and test set, and perform preprocessing; Step 3: Use the training set to pre-train the teacher model and save the trained teacher model; Step 4: Load the pre-trained weights, build the corresponding student model, and train the student model; Step 5: In the inference phase, only the target branch is kept as the baseline model.
2. The knowledge distillation method based on heterogeneous branch fusion according to claim 1, characterized in that: The step 4 at least comprises the following steps: Step 4-1: Freeze the pre-trained weights of the teacher network model; Step 4-2: Construct a student model based on heterogeneous branches and use a progressive feature fusion module to gradually fuse branch features; Step 4-3: Train the student model using the co-learning objective and teacher guidance as the distillation loss.
3. The knowledge distillation method based on heterogeneous branch fusion according to claim 2 is characterized in that: The step 4-2 at least includes the following steps: The student model is a multi-branch architecture, where each branch shares parameters at low-level layers and has independent parameters at high-level layers, acting as multiple students trained in parallel; Name the first branch Branch-1 and set it as the target branch, which together with the lower-level layers constitutes the baseline model; Use B to represent the total number of branches, b represents the branch number b∈(1,2,...,B), and denote the feature generated by the bth branch as F b , the fusion feature generated by the progressive fusion module is expressed as On the basis of the target branch, auxiliary blocks with the number of channels (b-1)·C are gradually added to construct the heterogeneous branches in the student model, where C is the number of channels in the high-level layer; Assume that each progressive fusion feature module has two input entries and one output exit; The feature map of the first branch, i.e. the target branch, is used as the fusion feature of the first input, i.e. F1 m =F1; In the progressive feature fusion module, the feature F of the b-th branch is first b And the fusion features of the previous layer Perform initial fusion; Then, the initial fused features are sent to two paths for processing: One path contains two convolutional layers with 1x1 kernels to capture local information; The other path captures global information through an additional global pooling layer; The feature information captured by the two paths is activated by the Sigmoid function to obtain the weight score, and is respectively compared with the feature F b , Multiply them together and use the parameter w to adjust the importance of the branch, and finally get the fusion features at this level 4. The knowledge distillation method based on heterogeneous branch fusion according to claim 3 is characterized in that: The expression using the progressive feature fusion module is: Where D(·) represents the fusion function.
5. The knowledge distillation method based on heterogeneous branch fusion according to claim 2, characterized in that: The step 4-3 is specifically as follows: By adding a softmax function with a hyperparameter temperature τ to soften the logits, a knowledge distillation loss function is constructed; The softened outputs of the teacher and the student are and The distillation loss L is calculated by Kullback-Leibler divergence kl ; Combine the cross entropy loss function of the true label and the student prediction, as well as the balance parameter, to construct a complete knowledge distillation loss function L kd ; According to the knowledge distillation loss function, two losses, collaborative learning objective and teacher guidance, are constructed. The weight of knowledge distillation in the training process is controlled by the ramp-up function, and the total loss is finally obtained to train the student model. The collaborative learning objectives and teacher guidance losses are specifically: The collaborative learning goal consists of two parts: (1) Fusion features Output and get FL; (2) Integrate the branch logits to get EL, and then soften FL and EL to get and and calculate distillation losses; Teacher guidance is achieved by calculating the distillation loss of the teacher model and each branch. The softened outputs of the teacher and the student are and 6. The knowledge distillation method based on heterogeneous branch fusion according to claim 5, characterized in that: The collaborative learning objective loss function and the teacher guidance loss function are expressed as: Where L cl represents the collaborative learning objective loss, L tg Denotes the teacher guidance loss. τ 2 Represents the weight parameter, which is used to balance the cross entropy loss and distillation loss; The complete loss function is expressed as: Where L total Represents the complete loss function, L ce represents the cross entropy loss function. w(i) represents the ramp-up function, which is used to adjust the distillation loss weights at different epochs.
7. The knowledge distillation method based on heterogeneous branch fusion according to claim 6, characterized in that: The step 5 at least comprises the following steps: In the student inference phase, the pre-trained teacher model and the branches with added auxiliary blocks are removed, and only the target branch is retained as the original model architecture, with no additional cost added.
Citation Information
Cited By
Educational resource sequence recommendation method and device based on three-view structural distillation, equipment and storage medium
CN121834065A