Self-distillation method for making up teacher-student gap through residual learning

Abstract

The invention discloses a self-distillation method for making up the teacher-student gap through residual learning, and relates to the technical field of computers. The method at least comprises the following steps: S1, obtaining an original data set, and dividing the original data set into a training set and a test set; s2, preprocessing the original data set; s3, dividing the network model into a plurality of outlets, introducing a cross-layer feature fusion module, and designing a residual learning strategy for multi-outlet self-distillation; s4, constructing a complete knowledge distillation loss function, improving a feature loss function based on a mean square error, and proposing multi-scale loss; and S5, inputting training set image data into the modified network model, storing the weight of the trained model, and then loading the model for reasoning to obtain a prediction result. Through the design of the method, the problems that students are difficult to learn and the knowledge of teachers is single when the teacher-student gap is large in the self-distillation process can be solved, and the classification precision of the model is effectively improved on the premise of not increasing extra cost during reasoning.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a self-distillation method that uses residual learning to bridge the gap between teachers and students. Background Technology

[0002] In computer vision, deep neural networks (DNNs) have been widely applied to tasks such as image classification, object detection, and semantic segmentation, demonstrating their superior performance. However, the excellent performance of these network models comes with a large number of parameters and high computational costs, posing a deployment challenge when resources are limited. To address the balance between model parameter count and performance, knowledge distillation has been proposed. This method extracts useful knowledge from large models to significantly improve the performance of lightweight models. However, training the teacher model requires substantial resources. Online knowledge distillation proposes training a group of student models simultaneously, transferring group knowledge to each student model in this dynamic process. Essentially, online knowledge distillation leverages the student group to integrate a powerful virtual teacher, enabling the student group to efficiently learn the necessary knowledge. To further reduce training costs, self-distillation divides the network into multiple stages, allowing shallower stages to learn from deeper stages, thereby improving model performance.

[0003] However, existing self-distillation methods neglect the learning gap between students and teachers at different depths. When the focus is on providing students with stronger teachers, the student-teacher learning gap is amplified, especially for early exits. Existing research indicates that an inappropriate learning gap, or a mismatch in learning abilities between teachers and students, can negatively impact knowledge transfer between them.

[0004] Therefore, a new solution is needed to address the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a self-distillation method that compensates for the gap between teachers and students through residual learning, so as to overcome the problems of students having difficulty learning and teachers having limited knowledge when the gap between teachers and students is large in self-distillation.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a self-distillation method for bridging the gap between teachers and students through residual learning, comprising at least the following steps:

[0007] S1: Obtain the original dataset and divide it into training and test sets;

[0008] S2: Preprocess the original dataset;

[0009] S3: Divide the network model into multiple exits, introduce a cross-layer feature fusion module, and design a residual learning strategy for multi-exit self-distillation;

[0010] S4: Construct a complete knowledge distillation loss function, improve the feature loss function based on mean squared error, and propose a multi-scale loss.

[0011] S5: Input the training set image data into the modified network model, save the model weights after training, and then load the model for inference to obtain the prediction results.

[0012] Furthermore, S3 includes at least the following steps:

[0013] S3.1: Divide the network model into multiple stages, each stage being an independent output that can produce prediction results;

[0014] S3.2: Use the cross-layer feature fusion module to perform cross-layer feature fusion on the features of each stage, and fuse deep features and shallow features stage by stage;

[0015] S3.3: Treat the fusion feature as the difference between teachers and students, and design residual learning strategies to enable students to learn the differences between teacher and student features.

[0016] Furthermore, S3.1 includes at least the following steps:

[0017] First, the target neural network is divided into multiple stages (usually 4 stages) based on its depth and original structure;

[0018] Then, a bottleneck layer and a fully connected layer are set after each shallow stage. These two layers are only used during training and are discarded during inference. The bottleneck layer is added to reduce the influence between each shallow classifier and to unify the scale of shallow features.

[0019] Divide the network into The first stage, for the first ( The original characteristics of each stage The scale and deepest features will be obtained by aligning and transforming multiple convolutional layers. Shallow features of the same scale are used for subsequent operations;

[0020] Predictive results can be generated at each stage;

[0021] During the training phase, all shallow stages are considered students by extracting knowledge from deeper stages, while the deepest stage is considered the teacher.

[0022] For multi-class classification tasks, given a set of values... A dataset with 1 class and 1 training sample and corresponding tags Then the first The first export in The output probability of the class is calculated as follows:

[0023]

[0024] in, For the first The logits of each branch; For the corresponding number The exponential form of the predicted scores for a class amplifies the relative differences in higher scores; To sum the predicted scores for all categories, it is ensured that the sum of the probabilities for all categories is 1;

[0025] By minimizing the prediction vector and the ground-truth label Train using the cross entropy loss between them:

[0026]

[0027] in, It is the one-hot encoded form of the actual label y; It is the logarithm of the predicted probability, which quantifies the difference between the model's prediction and the actual label.

[0028] Furthermore, S3.2 includes at least the following steps:

[0029] The first Features of the layer Features of the previous level Connecting along the channel yields preliminary fusion characteristics;

[0030] Then, the fused features are used with a channel selection mechanism. To restore the number of channels and ensure subsequent feature fusion, It consists of cascaded pooling layers, convolutional layers, and a sigmoid activation function, used to obtain important channel information;

[0031] In addition, a spatial selection mechanism is used for the two original input features. To emphasize important spatial areas;

[0032] First, perform one-dimensional convolution on the two features respectively, add them together, and then activate them using the Sigmoid function to obtain important spatial information;

[0033] By converging the channel information with the spatial system, the final result is obtained. Layer fusion features ;

[0034] The deepest feature map The fusion features, as the initial input, are ;

[0035] The feature expression for the subsequent cross-layer fusion output is:

[0036]

[0037] in, It is a fusion function from the feature fusion module.

[0038] Furthermore, S3.3 includes at least the following steps:

[0039] The fusion feature contains information at multiple levels, and student characteristics are incorporated into it, enabling the fusion feature to represent the differences between teachers and students and to provide a foundation for learning with disabilities.

[0040] Then, for teachers who use the fused features as the shallow features of the corresponding stage, the shallow student features no longer directly learn the difficult-to-understand deep features, but instead learn the residual parts of the fused features, that is, the differences between them. The residual learning of a layer can be represented as:

[0041]

[0042] Among them, MSL is a multi-scale loss used to transfer information between teacher and student features. Throughout the training process, shallow features at all stages will learn the corresponding fused features.

[0043] Furthermore, the knowledge distillation loss function is based on a self-distillation framework, in which all exits should be guided by ground-truth labels. The knowledge distillation loss function is used to transfer information between teachers and students and includes feature distillation and logits distillation.

[0044] Furthermore, while mean squared error can effectively transfer information between feature maps at the same level, it struggles to transfer complex information that converges across multiple stages. Based on the feature pyramid, dividing features into multiple scales allows feature maps to contain richer semantic information, facilitating information transfer. Therefore, a multi-scale loss function is proposed to divide the transfer of information between teachers and students across different scales, thereby better extracting information from different levels.

[0045] The feature distillation is an information extraction method that involves pooling the scale of the original features by halving it each time, calculating the mean squared error loss for each pooling, and finally summing them up to obtain distance information at multiple scales.

[0046] The feature loss during network training is represented as: .

[0047] Furthermore, the logits distillation is performed by training using the softened teacher output as the student's supervision signal, and the ensemble of all outputs is taken as the teacher, represented as:

[0048]

[0049] in, It is the output probability of the teacher model for the b-th sample belonging to the k-th class;

[0050] Then, the outputs of teachers and students are softened, and the probability formulas for the softened outputs of teachers and students are as follows:

[0051]

[0052] in, and These represent the softened outputs of the teacher and student, respectively. For temperature parameters; and These are teacher and student model pairs. logits;

[0053] Then minimize and The Kullback-Leibler divergence between them, and its loss calculation formula is:

[0054]

[0055] Finally, the complete distillation loss function is expressed as:

[0056]

[0057] Where total represents the complete loss function; hyperparameters and Weights used to balance the losses of each part; It is the standard cross-entropy loss, which measures the difference between the student model and the true label; It is the distillation loss, which measures the difference in output distribution between the student model and the teacher model; These are potential additional loss items.

[0058] Furthermore, S5 includes at least the following steps:

[0059] The training set is input into the modified self-distillation network for feature extraction, feature fusion, residual learning, and after training, the weight model is saved. Then, the weight model is loaded and input into the test set for accuracy testing.

[0060] Compared with the prior art, the beneficial effects of the present invention are:

[0061] This invention, through its method design, overcomes the problems of students struggling to learn and teachers having limited knowledge when there is a significant gap between teachers and students in self-distillation. Furthermore, it effectively improves the classification accuracy of the model without increasing additional costs during reasoning. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of the present invention;

[0064] Figure 2 This is a schematic diagram of the overall self-distillation method of the present invention;

[0065] Figure 3 This is a diagram of the cross-layer feature fusion module of the present invention;

[0066] Figure 4 (a) is a schematic diagram of the characteristic distillation method before the improvement of the present invention. Figure 4 (b) is a schematic diagram of the improved characteristic distillation method of the present invention;

[0067] Figure 5 A schematic diagram of the multi-scale loss of the invention;

[0068] Figure 6 This is a heatmap for the visualization analysis of the fusion features of the present invention. Detailed Implementation

[0069] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0070] Please see Figure 1 A self-distillation method for bridging the gap between teachers and students through residual learning includes at least the following steps:

[0071] Step 1: Obtain the original dataset and divide it into training and test sets;

[0072] In this embodiment, the original dataset obtained is the CIFAR-100 dataset, which contains 60,000 RGB images of size 32 × 32 in 100 categories. The training set and test set are divided into 50k and 10k images, respectively.

[0073] Step 2: Preprocess the original dataset;

[0074] In this embodiment, all images in the original dataset are processed and normalized using channel mean and standard deviation, and standard data augmentation operations are employed.

[0075] Step 3: Divide the network model into multiple exits, introduce a cross-layer feature fusion module for feature fusion, and design a residual learning strategy for multi-exit self-distillation;

[0076] Step 3 includes:

[0077] Step 3-1: Divide the network model into multiple stages, with each stage being an independent exit point;

[0078] Step 3-2: Perform cross-layer feature fusion using the cross-layer feature fusion module for the features from each stage;

[0079] Step 3-3: Treat the fusion feature as the difference between teachers and students, and design residual learning strategies to enable students to learn the differences between teacher and student features;

[0080] Specifically, such as Figure 2 As shown, let's take dividing the network into four stages as an example. First, the target neural network is divided into four stages based on its depth and original structure. Then, a bottleneck layer and a fully connected layer are set after each shallow stage. These two layers are only used during training and can be discarded during inference. The bottleneck layer is added to mitigate the influence between each shallow classifier and to unify the scale of shallow features. For the first stage... ( The original characteristics of each stage Multiple convolutional layers will be used to transform it to obtain the scale and deepest features. Shallow features of the same scale are used for subsequent operations. During the training phase, all shallow stages are treated as students by extracting knowledge from deeper stages, while the deepest stage is treated as the teacher. Combining a multi-exit structure with self-distillation, by utilizing the results of the network at different training stages, predictions from some stages can be used to guide predictions from other stages, improving model performance and enabling adaptive inference.

[0081] For multi-class classification tasks, given a set of values... A dataset with 1 class and 1 training sample and corresponding tags Then the first The first export in The output probability of the class is calculated as follows:

[0082]

[0083] in, For the first The logits of each branch; For the corresponding number The exponential form of the predicted scores for a class amplifies the relative differences in higher scores; To sum the predicted scores for all categories, it is ensured that the sum of the probabilities for all categories is 1;

[0084] By minimizing the prediction vector and the ground-truth label Train using the cross entropy loss between them:

[0085]

[0086] in, It is the one-hot encoded form of the actual label y; It is the logarithm of the predicted probability, which quantifies the difference between the model's prediction and the actual label.

[0087] For example Figure 3 As shown, in the cross-layer feature fusion module, the first step is to... Features of the layer Features of the previous layer Connecting along the channel number yields preliminary fused features. Then, a channel selection mechanism is applied to the fused features. This is used to restore the number of channels and ensure subsequent feature fusion. It consists of cascaded pooling layers, convolutional layers, and a sigmoid activation function to capture important channel information. Furthermore, a spatial selection mechanism is used on the two original input features. To emphasize important spatial regions, two features are convolved separately using one-dimensional methods, summed, and then activated using the Sigmoid function to obtain crucial spatial information. Finally, the channel information is converged with the spatial system to obtain the final result. Layer fusion features The deepest feature map The fusion features, as the initial input, are The feature expression for the subsequent cross-layer fusion output is:

[0088]

[0089] The designed cross-layer feature fusion module extracts important spatial and channel information, preserving crucial information during feature fusion and achieving efficient fusion between features at different levels. This provides richer fused features for subsequent residual learning, laying a solid foundation.

[0090] The fusion feature incorporates information from multiple stages. More importantly, student characteristic information is integrated, enabling the fusion feature to represent the differences between teachers and students, which provides a foundation for residual learning. Then, when teachers use the fusion feature as the shallow features of the corresponding stage, the shallow student features no longer directly learn the difficult-to-understand deep features, but instead learn the residual part of the fusion feature, that is, the differences between them. For example... Figure 4 As shown in (a) and (b), the residual learning of the i-th layer can be expressed as:

[0091] MSL (Multi-Scale Loss) is used to transfer information between teacher and student features. Throughout the training process, shallow features at all stages will learn the corresponding fused features.

[0092] By allowing all shallow features to directly mimic deep features, ignoring the skill gap between teachers and students, and attempting to have students directly learn from the teacher to reach the teacher's level, it is generally difficult. A significant gap between teachers and students makes learning challenging for students. Residual learning, on the other hand, allows students to learn the differences between their features and deep features. By learning these residuals, students can gradually address these differences, continuously narrowing the gap with deep features and improving the efficiency of the distillation process.

[0093] Step 4: Construct a complete knowledge distillation loss function, improve the feature loss function based on mean squared error, and propose a multi-scale loss.

[0094] Within the self-distillation framework, all exits should be guided by ground-truth labels. Furthermore, distillation loss is used to transfer information between faculty and students; distillation loss consists of two parts: characteristic distillation and logits distillation.

[0095] (1) Feature Distillation: Mean squared error can effectively transfer information between feature maps at the same level, but it is difficult to transfer composite-level information that converges information from multiple stages. Based on the feature pyramid, dividing features into multiple scales can make feature maps contain richer semantic information, which is beneficial for information transfer. Therefore, multi-scale loss is proposed. For example... Figure 5 As shown, the information transmission between teachers and students is divided into different scales to better extract information at different levels. The specific information extraction method involves pooling the original features at a scale that is halved each time, calculating the mean squared error loss after each pooling, and finally summing the results to obtain distance information at multiple scales. The feature loss during network training is expressed as:

[0096] (2) Logits distillation: To train students using the softened teacher output as the supervisory signal, the ensemble of all outputs is taken as the teacher, and is expressed as:

[0097]

[0098] in, It is the output probability of the teacher model for the b-th sample belonging to the k-th class;

[0099] Then, the outputs of teachers and students are softened, and the probability formulas for the softened outputs of teachers and students are as follows:

[0100]

[0101] in, and These represent the softened outputs of the teacher and student, respectively. For temperature parameters; and These are teacher and student model pairs. logits;

[0102] Then minimize and The Kullback-Leibler divergence between them, and its loss calculation formula is:

[0103]

[0104] Finally, the complete distillation loss function is expressed as:

[0105]

[0106] Where total represents the complete loss function; hyperparameters and weights used to balance the loss of each part; It is the standard cross-entropy loss, which measures the difference between the student model and the true label; It is the distillation loss, which measures the difference in output distribution between the student model and the teacher model; These are potential additional loss items.

[0107] Step 5: Input the training set image data into the modified network model, save the model weights after training, and then load the model weights for inference to obtain the prediction results.

[0108] The specific steps are as follows: input the training set into the modified self-distillation network, perform feature extraction, feature fusion, residual learning, save the weight model after training, and then load the weight model into the test set for accuracy testing.

[0109] To verify the effectiveness of the extraction method proposed in this invention, specific experiments were conducted, and the experimental results are shown in Tables 1 and 2.

[0110] Table 1. Top-1 test accuracy on CIFAR-100

[0111] Table 2 Comparison of learning gaps of different methods on CIFAR-100

[0112] The experimental results in Table 1 show that the self-distillation method (RLSD) of this invention, which compensates for the gap between teachers and students through residual learning, exhibits the best performance on various model architectures, with a more significant improvement at shallow exits. For example, on the ResNeXt-50 model, the deep exit (exit 4) improved by 0.68% compared to the state-of-the-art (SOTA), while the shallow exits improved by 1.17%, 1.01%, and 0.75% respectively (exits 1, 2, and 3).

[0113] Learning gaps are often difficult to measure accurately, but they should exhibit relatively stable characteristics, meaning the overall gap between teachers and students should decrease over time. Furthermore, the knowledge provided by teachers should be more advanced than the knowledge already possessed by students. This paper uses top-1 test accuracy to represent knowledge advantage and considers the difference in test accuracy between teachers and students as the learning gap. A positive difference indicates that the teacher's knowledge is superior to the student's, and a smaller difference indicates a smaller, more stable learning gap. The experimental results in Table 2 show that the self-distillation method of bridging the teacher-student gap through residual learning in this invention has the most stable learning gap. This indicates that residual learning effectively bridges the learning gap between teachers and students and makes it more stable, thus better improving student performance.

[0114] like Figure 6 As shown, to further illustrate that the fused features can gather rich information from multiple levels and effectively bridge the gap between teachers and students, providing a good foundation for residual learning, the effectiveness of the cross-layer feature fusion module is demonstrated from the perspective of feature maps. This represents the fused features from deep fusion to layer 1, which were used to guide the features of layer 1 during training. Rows 1-2 show the cases where each output and both are correctly classified. It can be seen that for simple images, each layer makes the correct prediction, and deeper layers focus more on local regions. Row 3 shows a more complex case where, when shallow features misclassify, both deep features and both make the correct prediction, but the areas they focus on are closer to those of the shallow features. This indicates that although deep features and both have stronger discriminative power, It has a closer connection with shallow features and a higher similarity, thus making The learning gap with shallow features is smaller, which is more conducive to information transmission.

[0115] Next, we conducted ablation experiments on multi-scale loss to demonstrate the model accuracy with and without multi-scale loss during feature distillation. Under the same conditions, we used mean squared error (MSE) loss for comparison. It can be seen that the network model using multi-scale loss (MSL) exhibits better performance. This indicates that dividing the information transfer between teacher and student features into different scales can better extract information at different levels (see Table 3).

[0116] Table 3. Accuracy comparison between mean squared error loss and multi-scale loss on CIFAR-100.

[0117] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A self-distillation method for bridging the gap between teachers and students through residual learning, characterized in that: At least the following steps are included: S1: Obtain the original dataset and divide it into training and test sets; S2: Preprocess the original dataset; S3: Divide the network model into multiple exits, introduce a cross-layer feature fusion module, and design a residual learning strategy for multi-exit self-distillation; S4: Construct a complete knowledge distillation loss function, improve the feature loss function based on mean squared error, and propose a multi-scale loss. S5: Input the training set image data into the modified network model, save the model weights after training, and then load the model for inference to obtain the prediction results.

2. The self-distillation method for bridging the gap between teachers and students through residual learning according to claim 1, characterized in that: The S3 includes at least the following steps: S3.1: Divide the network model into multiple stages, each stage being an independent output that can produce prediction results; S3.2: Use the cross-layer feature fusion module to perform cross-layer feature fusion on the features of each stage, and fuse deep features and shallow features stage by stage; S3.3: Treat the fusion feature as the difference between teachers and students, and design residual learning strategies to enable students to learn the differences between teacher and student features.

3. The self-distillation method for bridging the gap between teachers and students through residual learning according to claim 2, characterized in that: S3.1 includes at least the following steps: First, the target neural network is divided into multiple stages based on its depth and original structure; Then, a bottleneck layer and a fully connected layer are set after each shallow stage. These two layers are only used during training and are discarded during inference. Divide the network into The first stage, for the first ( The original characteristics of each stage The scale and deepest features will be obtained by aligning and transforming multiple convolutional layers. Shallow features of the same scale are used for subsequent operations; Each stage can generate prediction results; During the training phase, all shallow stages are considered students by extracting knowledge from deeper stages, while the deepest stage is considered the teacher. For multi-class classification tasks, given a set of values... A dataset with 1 class and 1 training sample and corresponding tags Then the first The first export in The output probability of the class is calculated as follows: in, For the first The logits of each branch; For the corresponding number The exponential form of the predicted scores for a class amplifies the relative differences in higher scores; To sum the predicted scores for all categories, it is guaranteed that the sum of the probabilities for all categories is 1; By minimizing the prediction vector and the ground-truth label Train using the cross entropy loss between them: in, It is the one-hot encoded form of the actual label y; It is the logarithm of the predicted probability, which quantifies the difference between the model's prediction and the actual label.

4. The self-distillation method for bridging the gap between teachers and students through residual learning according to claim 3, characterized in that: S3.2 includes at least the following steps: The first Layer characteristics Features of the previous level Connecting along the channel yields preliminary fusion characteristics; Then, the fused features are used with a channel selection mechanism. To restore the number of channels and ensure subsequent feature fusion, It consists of cascaded pooling layers, convolutional layers, and a sigmoid activation function; In addition, a spatial selection mechanism is used for the two original input features. To emphasize important spatial areas; First, perform one-dimensional convolution on the two features respectively, add them together, and then activate them using the Sigmoid function to obtain important spatial information; By converging the channel information with the spatial system, the final result is obtained. Layer fusion features ; The deepest feature map The fusion features, as the initial input, are ; The feature expression for the subsequent cross-layer fusion output is: in, It is a fusion function from the feature fusion module.

5. The self-distillation method for bridging the gap between teachers and students through residual learning according to claim 4, characterized in that: S3.3 includes at least the following steps: The fusion feature contains information at multiple levels, and student characteristics are incorporated into it, enabling the fusion feature to represent the differences between teachers and students and to provide a foundation for learning with disabilities. Then, for teachers who use the fused features as the shallow features of the corresponding stage, the shallow student features no longer directly learn the difficult-to-understand deep features, but instead learn the residual parts of the fused features, that is, the differences between them. The residual learning representation of the layer is as follows: MSL is the multi-scale loss.

6. The self-distillation method for bridging the gap between teachers and students through residual learning according to claim 5, characterized in that: The knowledge distillation loss function is based on a self-distillation framework, in which all exits should be guided by ground-truth labels. The knowledge distillation loss function is used to transfer information between teachers and students and includes feature distillation and logits distillation.

7. The self-distillation method for bridging the gap between teachers and students through residual learning according to claim 6, characterized in that: The feature distillation method is an information extraction method that pools the scale of the original features by halving it each time, calculates the mean squared error loss for each pooling, and finally summarizes the data to obtain distance information at multiple scales. The feature loss during network training is represented as: .

8. The self-distillation method for bridging the gap between teachers and students through residual learning according to claim 6, characterized in that: The logits distillation is performed by training using the softened teacher output as the student's supervision signal, and the ensemble of all outputs is taken as the teacher, represented as: in, It is the output probability of the teacher model for the b-th sample belonging to the k-th class; Then, the outputs of teachers and students are softened, and the probability formulas for the softened outputs of teachers and students are as follows: in, and These represent the softened outputs of the teacher and student, respectively. For temperature parameters; and These are teacher and student model pairs. logits; Then minimize and The Kullback-Leibler divergence between them, and its loss calculation formula is: Finally, the complete distillation loss function is expressed as: Where total represents the complete loss function; hyperparameters and Weights used to balance the losses of each part; It is the standard cross-entropy loss, which measures the difference between the student model and the true label; It is the distillation loss, which measures the difference in output distribution between the student model and the teacher model; This is an additional loss item.

9. The self-distillation method for bridging the gap between teachers and students through residual learning according to claim 1, characterized in that: S5 includes at least the following steps: The training set is input into the modified self-distillation network for feature extraction, feature fusion, residual learning, and after training, the weight model is saved. Then, the weight model is loaded and input into the test set for accuracy testing.

Images