Self-review knowledge distillation method, system and terminal based on multi-level attention

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a self-review knowledge distillation method, system and terminal based on multi-level attention. The method uses a student model and a converged teacher model to perform prediction processing on sample images in a sample set respectively; obtains feature maps output by the teacher model and the student model at each processing stage, and calculates the attention map of the teacher model and the student model at each processing stage. The attention map of the student model at each processing stage is feature-fused with the attention map of the previous processing stage, and the multi-level attention distillation loss is calculated. The standard cross entropy loss and the basic distillation loss are calculated; the three losses are weighted to obtain the total training loss, and gradient descent and back propagation are performed on the basis of the total training loss, so as to optimize the student model parameters, and then the learning model is continuously optimized using the sample images of the next batch of the sample set until the student model converges. The present invention can effectively improve the accuracy of the student model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a self-review knowledge distillation method, system and terminal based on multi-level attention. Background Art

[0002] Knowledge distillation is a transfer learning method that aims to improve the performance of a smaller student model by transferring knowledge extracted from a larger teacher model. Existing knowledge distillation methods can be divided into two types: logits-based knowledge distillation methods and feature-based knowledge distillation methods.

[0003] Logits-based distillation methods extract knowledge only through the output logits. Existing research proposed the concept of knowledge distillation and guided the training of the student model through the softened teacher model output. After that, many researchers improved the logits-based methods, for example: promoting knowledge distillation by introducing a mutual learning paradigm or additional teacher assistant modules; improving performance by decoupling the distillation process of the teacher model and the student model. Logit distillation methods are simple to implement and can be applied to any situation. However, their performance is usually not as good as feature-based methods.

[0004] Feature-based knowledge distillation methods are a type of method that transfers knowledge by aligning the intermediate feature representations of the teacher model and the student model. Unlike logits-based methods, feature-based methods do not rely solely on the final output logits, but instead utilize the feature representations within the model to capture more intermediate information. More and more work is beginning to study the knowledge of the intermediate layers of neural networks. Some scholars have added auxiliary classifiers to some hidden layers of the teacher network and the student network, and then co-trained the teacher network and the student network, removing these classifiers after training to reduce running time. There are also knowledge distillation methods based on attention transfer, which define the attention map of a specific layer as a spatial map indicating the input region that the model pays the most attention to. However, none of the above methods consider the student's self-review by reviewing their own knowledge. Most of the existing feature-based knowledge distillation methods only use one distillation strategy to refine knowledge at a certain layer of the teacher network, which limits the extraction of more information knowledge and the accuracy of the student model is limited. Summary of the invention

[0005] The technical problem solved by the present invention is how to improve the accuracy of the student model in a feature-based knowledge distillation method.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The present invention discloses a self-review knowledge distillation method based on multi-level attention, comprising:

[0008] The student model and the converged teacher model are used to perform prediction processing on the sample images in the sample set respectively; wherein the prediction processing includes multiple processing stages, and each processing stage is executed by a corresponding network layer in the student model or the teacher model.

[0009] The feature maps output by the teacher model and the student model at each processing stage are obtained, and the attention-based feature maps, i.e., attention maps, of the teacher model and the student model at each processing stage are calculated based on the feature maps.

[0010] The attention map of the student model at each processing stage is feature fused with the attention map of the previous processing stage, and the fusion results are combined to calculate the single distillation loss in the process of transferring knowledge from the teacher model to the student model at each processing stage, thereby calculating the multi-level attention distillation loss.

[0011] According to the output results after prediction processing by the student model and the teacher model, the standard cross entropy loss and the basic distillation loss are calculated; the multi-level attention distillation loss, the standard cross entropy loss and the basic distillation loss are weighted to obtain the total training loss.

[0012] Gradient descent and back propagation are performed based on the total training loss to optimize the student model parameters, and then the learning model is continuously optimized using the sample images of the next batch of the sample set until the student model converges.

[0013] As a further improvement of the above scheme, the calculation formulas of the attention maps of the teacher model and the student model at each processing stage are as follows:

[0014] ;

[0015] ;

[0016] In the formula, For the teacher model i The feature map output by the processing stage is C is the number of channels in the feature map, , K a total number of processing stages processed for said prediction; For the teacher model i The attention map corresponding to the processing stage; ||·||2 represents the second norm of the vector; vec (·) indicates vector operation; For the student model i The feature map output by each processing stage; For the student model i Attention maps corresponding to each processing stage.

[0017] As a further improvement of the above scheme, the attention map of the student model at each processing stage is feature fused with the attention map of the previous processing stage, and the single distillation loss in the process of transferring the knowledge of each processing stage from the teacher model to the student model is calculated by combining the fused features, so as to calculate the multi-level attention distillation loss, including:

[0018] The attention map of the student model at each processing stage is fused with the attention map of the previous processing stage. The calculation formula of the fused feature is as follows:

[0019] ;

[0020] In the formula, Indicates that the student model is i The fusion features of the processing stages are is the set of real numbers, H and W are the width and height of the attention map respectively; represents the initial feature fusion, represents the fusion weight; represents weighted multiplication.

[0021] The single distillation loss in each processing stage is calculated as follows:

[0022] ;

[0023] In the formula, For the i The single distillation loss in each processing stage.

[0024] Calculate the multi-level attention distillation loss based on the single distillation loss of all sample images in the sample set at all processing stages , The calculation formula is as follows:

[0025] ;

[0026] In the formula, represents a sample set; The first Sample images.

[0027] As a further improvement of the above solution, the calculation process of the fusion weight is:

[0028] The initial feature fusion results are respectively input into two convolutional layer processing units, and then the outputs of the two convolutional layer processing units are fused for the second time; each convolutional layer processing unit includes two point-by-point convolutional layers connected in sequence, a ReLu activation function is set between the two point-by-point convolutional layers, a batch normalization process is set before the second fusion, and a Sigmoid activation function is set after the second fusion.

[0029] As a further improvement of the above scheme, the calculation formula of the standard cross entropy loss is as follows:

[0030] ;

[0031] In the formula, represents the standard cross entropy loss; n is the total number of categories, j ∈[1, n ]; For the Hard labels for sample images; log (·) is a logarithmic function; For student models Sample image No. j The predicted probability value of each category.

[0032] The calculation formula of the basic distillation loss is:

[0033] ;

[0034] In the formula, represents the basic distillation loss; For the teacher model Sample image No. j Soft labels for categories; For student models Sample image No. j Soft labels for categories.

[0035] As a further improvement of the above scheme, the calculation formula of the total training loss is as follows:

[0036] ;

[0037] In the formula, represents the total training loss; α and β is the weight coefficient used to balance different distillation losses; is the temperature coefficient.

[0038] The present invention also discloses a self-review knowledge distillation system based on multi-level attention, including: a prediction processing module, a multi-level attention module, a student self-review module, a loss calculation module and a model optimization module.

[0039] The prediction processing module is used to use the student model and the converged teacher model to perform prediction processing on the sample images in the sample set respectively; wherein the prediction processing includes multiple processing stages, and each processing stage is executed by the corresponding network layer in the student model or the teacher model.

[0040] The multi-level attention module is used to obtain the feature maps output by the teacher model and the student model at each processing stage, and calculate the attention-based feature maps, namely the attention maps, of the teacher model and the student model at each processing stage.

[0041] The student self-review module is used to fuse the attention map of the student model at each processing stage with the attention map of the previous processing stage, and combine the fusion results to calculate the single distillation loss in the process of transferring knowledge from the teacher model to the student model at each processing stage, thereby calculating the multi-level attention distillation loss.

[0042] The loss calculation module is used to calculate the standard cross entropy loss and the basic distillation loss according to the output results after prediction processing by the student model and the teacher model; the loss calculation module is also used to weight the multi-level attention distillation loss, the standard cross entropy loss and the basic distillation loss to obtain the total training loss.

[0043] The model optimization module is used to perform gradient descent and back propagation based on the total training loss to optimize the student model parameters, and then continuously optimize the learning model using the next batch of samples in the training set until the student model converges.

[0044] As a further improvement of the above scheme, the knowledge distillation system applies the self-review knowledge distillation method based on multi-level attention as described above.

[0045] The present invention also discloses a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the self-review knowledge distillation method based on multi-level attention as described above are implemented.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. The self-review knowledge distillation method based on multi-level attention disclosed in the present invention integrates the advantages of multi-stage learning and student self-reflection to solve the shortcomings of current knowledge distillation methods. By introducing a multi-level attention mechanism and a self-review mechanism, the method enables the student model to obtain important information from the teacher model at different levels in each training stage, and can review and consolidate previously learned knowledge in each training stage, thereby further enhancing the understanding and mastery of key knowledge. Experimental results show that this method has achieved significant performance improvements on multiple benchmark datasets, verifying its effectiveness and superiority.

[0048] 2. The knowledge distillation system and computer terminal disclosed in the present invention can produce the same beneficial effects as the above-mentioned knowledge distillation method by applying the method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a diagram of the self-review knowledge distillation framework based on multi-level attention in Example 1 of the present invention.

[0050] Figure 2 Schematic diagram of the attention extraction process in Example 1 of the present invention.

[0051] Figure 3 This is a flowchart of the self-review knowledge distillation method based on multi-level attention in Example 1 of the present invention.

[0052] Figure 4 Schematic diagram of the operation of multi-level attention fusion in Example 1 of the present invention.

[0053] Figure 5 This is a t-SNE visualization result diagram of feature learning of the traditional knowledge distillation method in Example 1 of the present invention.

[0054] Figure 6 This is a t-SNE visualization result diagram of feature learning of the method of the present invention in Example 1 of the present invention.

[0055] Figure 7 Schematic diagram of the structure of the self-review knowledge distillation system based on multi-level attention in Example 2 of the present invention.

[0056] Figure 8 This is a schematic diagram of the structure of a computer terminal in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Example 1

[0059] See also Figure 1 ,In response to the problem that the existing knowledge distillation process ignores the key areas of ,samples and that students can conduct multi-level learning, this embodiment ,proposes a novel framework, namely, student self-review knowledge distillation based on ,multi-level attention, which inherits the advantages of multi-level focused learning ,and self-review.

[0060] First, if Figure 2 In the attention extraction process shown in the figure, the present invention designs a multi-level attention mechanism to enable the student model to focus on key knowledge at different levels during the knowledge distillation process. Specifically, the present invention introduces multi-level attention modules, which are embedded in each stage of the teacher model and the student model respectively. At each stage, these attention modules are used to extract important information of the teacher model at different levels and pass it to the student model.

[0061] In addition, the present invention also introduces a self-review mechanism, which passes the attention map of the early stage to the subsequent stage at the end of each stage, so that the student model can self-review and learn through knowledge based on multi-level attention after each stage. This mimics how students in the real world reflect on their own learning, thereby transferring knowledge based on multi-level attention from shallower layers to deeper layers (i.e., the reverse direction of knowledge distillation used in self-distillation). In fact, each layer has its own dark knowledge that needs to be refined for the next layer. Compared with the deep student network, its shallow dark knowledge can be regarded as informative knowledge previously learned from the perspective of human learning, so it can be transferred through self-refinement to facilitate the learning of subsequent layers. In this way, the student model can not only pay attention to the important information of the teacher model at different levels, but also review and consolidate the previously learned knowledge at each stage.

[0062] See also Figure 3 ,This embodiment provides a self-review knowledge distillation method based on multi-level attention, including steps S1 to S5 as shown below.

[0063] S1. Use the student model and the converged teacher model to perform prediction processing on the sample images in the sample set respectively; wherein the prediction processing includes multiple processing stages, and each processing stage is executed by a corresponding network layer in the student model or the teacher model.

[0064] S2. Obtain the feature maps output by the teacher model and the student model at each processing stage, and calculate the attention-based feature maps, i.e., attention maps, of the teacher model and the student model at each processing stage.

[0065] Considering that activation-based attention can focus on the important target areas of training samples, it can well capture the important features of different samples as information knowledge of the distillation process. Therefore, the present invention designs an activation-based attention mechanism to construct attention-based multi-channel features.

[0066] First, given the first i Block , which is the first i The student model i Block , which is the first i processing stage.

[0067] In this embodiment, by obtaining the contribution index of the output data of the teacher model at each processing stage to the training of the student model, the attention graphs of the teacher model and the student model at each processing stage are obtained, and the calculation formulas are as follows:

[0068] ;

[0069] ;

[0070] In the formula, For the teacher model i The feature map output by the processing stage is C is the number of channels in the feature map, , K a total number of processing stages processed for said prediction; For the teacher model i The attention map corresponding to the processing stage; ||·||2 represents the second norm of the vector; vec (·) indicates vector operation; For the student model i The feature map output by each processing stage; For the student model i Attention maps corresponding to each processing stage.

[0071] S3. Feature fusion is performed on the attention map of the student model at each processing stage with the attention map of the previous processing stage, and the single distillation loss of the knowledge transfer from the teacher model to the student model at each processing stage is calculated based on the fusion result, thereby calculating the multi-level attention distillation loss. Step S3 may include steps S31 to S33.

[0072] S31. The attention map of the student model at each processing stage is fused with the attention map of the previous processing stage. The calculation formula of the fused feature is as follows:

[0073] ;

[0074] In the formula, Indicates that the student model is i The fusion features of the processing stages are is the set of real numbers, H and W are the width and height of the attention map respectively; represents the initial feature fusion, represents the fusion weight; represents weighted multiplication.

[0075] How does multi-level attention fusion work? Figure 4 (In the figure r Used for C As shown in the figure, first it adjusts the low-level elements to the same shape as the high-level elements. In this way, the two elements at different levels can be better aligned, resulting in a more accurate association. Next, the two elements are connected to form two H×W attention maps, which are used to weight the two elements to highlight their importance. Finally, the final output is generated by adding the two weighted attention maps. The calculation process of the fusion weight is:

[0076] The initial feature fusion results are respectively input into two convolutional layer processing units, and then the outputs of the two convolutional layer processing units are fused for the second time; each convolutional layer processing unit includes two point-by-point convolutional layers connected in sequence, a ReLu activation function is set between the two point-by-point convolutional layers, a batch normalization process is set before the second fusion, and a Sigmoid activation function is set after the second fusion.

[0077] It should be noted that the fusion weight It consists of real numbers between 0 and 1. This also enables the network to and Soft selection or weighted averaging between.

[0078] This process can dynamically generate different attention maps according to the input features, thereby achieving adaptive aggregation of features. Compared with direct summation, this adaptive method is superior because it takes into account the diversity of attention maps generated at different network stages. Low-level and high-level blocks will focus on different areas, and attention feature fusion can aggregate them more reasonably.

[0079] S32. Calculate the single distillation loss in each processing stage using the following formula:

[0080] ;

[0081] In the formula, For the i The single distillation loss of the processing stage. <1 o'clock, <0.5; when ≥1, ≥0.5. Compared with the square loss, the absolute value of distillation loss It is less sensitive to outliers, and it can enhance important channels and weaken unimportant channels to characterize the indications from each channel. The indications contained in the attention map will be transferred at each stage of the training process.

[0082] S33. Calculate the multi-level attention distillation loss based on the single distillation loss of all sample images in the sample set at all processing stages .

[0083] In order to enable the student model to obtain more comprehensive relational knowledge from the teacher model, multi-level attention distillation loss is involved. , the calculation formula is as follows:

[0084] ;

[0085] In the formula, represents a sample set; The first Sample images.

[0086] Through stage-by-stage attention transfer, the student model can learn more important information of each block / stage channel feature map from the teacher model.

[0087] in addition, Figure 1The structure in shows an elegant design that cleverly uses the concept of residual learning to simplify the distillation process. An example of this approach is that in stage 2, the student features are aggregated with the student features of stage 1 to mimic the teacher features of stage 2. In this way, the student features of stage 2 learn the residual information of the stage 1 features while mimicking the teacher features of stage 2. It is worth noting that the knowledge review of the present invention refines knowledge from shallow to deep layers, which is the opposite direction of standard self-distillation. In this method, the student network is able to constantly review the knowledge acquired so far in each stage to reflect on its learning.

[0088] This process of residual learning is more stable and effective than directly letting the student's high-level features learn from the teacher's low-level features. Through the residual learning framework, the student's high-level features can gradually extract more useful information without being disturbed by the noise or details of the low-level features. This progressive information extraction method makes it easier for the student network to capture important information from the teacher network.

[0089] S4. Calculate the standard cross entropy loss and the basic distillation loss based on the output results after prediction processing by the student model and the teacher model; weight the multi-level attention distillation loss, the standard cross entropy loss and the basic distillation loss to obtain the total training loss.

[0090] The framework of the present invention contains two different target knowledge distillation modules, which work together to implement the principle of multi-level learning from teacher to student, that is, student self-review from the first few layers of the student network to the subsequent layers through offline distillation. The multi-level attention module allows the student model to learn diverse knowledge at different levels, while the student self-review allows the student to continuously improve itself by learning from the past. Therefore, the overall loss function is defined as follows:

[0091] ;

[0092] In the formula, represents the total training loss; represents the standard cross entropy loss; represents the basic distillation loss; α and β is the weight coefficient used to balance different distillation losses; is the temperature coefficient.

[0093] The standard cross entropy loss is calculated as follows:

[0094] ;

[0095] In the formula, n is the total number of categories, j ∈[1,n ]; For the Hard labels for sample images; log (·) is a logarithmic function; For student models Sample image No. j The hard label means that each image has only one certain category label, while the soft label means that each image can have multiple possible category labels, which are used to indicate the similarity or correlation between images.

[0096] The basic distillation loss is calculated as:

[0097] ;

[0098] In the formula, For the teacher model Sample image No. j Soft labels for categories; For student models Sample image No. j Soft labels for categories.

[0099] S5. Perform gradient descent and back propagation based on the total training loss to optimize the student model parameters, and then continue to optimize the learning model using the next batch of sample images in the sample set until the student model converges.

[0100] This example also experiments on various tasks. First, the method of the present invention is compared with other knowledge distillation methods for classification, trying different settings, different architectures and data sets. In addition, the knowledge distillation method of the present invention is applied to object detection and instance segmentation tasks, and the method of the present invention significantly improves the baseline model.

[0101] 1. Dataset and implementation details

[0102] 1.1 Dataset

[0103] This example uses three widely studied datasets: ① CIFAR-100, which has 60,000 images from 100 categories (50,000 for training and 10,000 for validation) with an image resolution of 32×32. ② ImageNet, one of the most important benchmark datasets for image classification, has nearly 1.3 million training images and 50,000 images for validation. These images are from 1,000 categories with high resolution. ③ MS-COCO, a mainstream object detection dataset, has 118,000 training images and 5,000 validation images from 80 categories.

[0104] 1.2 Implementation details

[0105] Specifically, for CIFAR-100, this example trains all models for 240 epochs using SGD (stochastic gradient descent) with a batch size of 64. The initial learning rate for ShuffleNet and MobileNet is 0.01, and the initial learning rate for other architectures consisting of ResNets, WRN, and VGG is 0.05, divided by 10 at 150, 180, and 210 epochs.

[0106] For ImageNet, the model was trained for 100 epochs (batches) with a batch size of 128. The initial learning rate was 0.1 and was divided by 10 every 30 epochs.

[0107] For MS-COCO, the batch size is set to 8 and the base learning rate is set to 0.01.

[0108] In this example, 1 GPU is used to train the model on CIFAR-100, and 8 GPUs are used on ImageNet and MS-COCO. This example tries various representative CNN networks: VGG, ResNet, WideResNet, MobileNet, and ShuffleNet. For fairness, all results of previous methods are either reported in previous papers (this example keeps the same training settings as them) or the training settings of this example are obtained using the code released by the author. All results on CIFAR100 are the average of 5 trials, while all results on ImageNet and COCO are the average of 3 trials.

[0109] 2. Results

[0110] 2.1 Results on the CIFAR-100 dataset

[0111] In this embodiment, the method of the present invention is experimented on CIFAR-100, and the knowledge distillation results of different methods under different teacher-student settings are compared in Tables 1 and 2. Table 1 shows the case where the teacher model and the student model have the same architecture, while Table 2 shows the case where the teacher model and the student model have different architectures.

[0112] Table 1: CIFAR-100 results with the same architecture for both teacher and student models

[0113] ;

[0114] Table 2: CIFAR-100 results with different architectures for teacher and student models

[0115] ;

[0116] In the table, KD refers to existing research: Geoffrey Hinton, “Distilling the knowledge in aneural network,” arXiv preprint arXiv:1503.02531, 2015.

[0117] DKD is existing research: Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and JiajunLiang, “Decoupled knowledge distillation,” in Proceedings of the IEEE / CVFConference on computer vision and pattern recognition, 2022, pp. 11953–11962.

[0118] FitNet is based on existing research: Adriana Romero, Nicolas Ballas, Samira EbrahimiKahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio, “Fitnets: Hints forthin deep nets,” arXiv preprint arXiv:1412.6550, 2014.

[0119] AT is existing research: Sergey Zagoruyko and Nikos Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,” arXiv preprint arXiv:1612.03928, 2016.

[0120] Review of RKD: Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho, “Relational Knowledge Distillation,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 101-1 3967–3976.

[0121] Review of CRD Representation:Yonglong Tian, ​​Dilip Krishnan, and Phillip Isola,“Contrastive representation distillation,” arXiv preprint arXiv:1910.10699,2019.

[0122] OFD Review:Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park,Nojun Kwak, and Jin Young Choi, “A Comprehensive Overhaul of Feature Distillation,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2019, pp. 10-11. 1921–1930.

[0123] ReviewKD Conference: Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia, “Distilling knowledge via knowledge review,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5008–5017.

[0124] CAT-KD is an existing study: Ziyao Guo, Haonan Yan, Hui Li, and Xiaodong Lin, "Class attention transfer based knowledge distillation," in CVPR, 2023.

[0125] Sim-KD is existing research: Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen, “Knowledge distillation with the reused teacherclassifier,” in CVPR, 2022.

[0126] Ours is the self-review knowledge distillation method based on multi-level attention of the present invention.

[0127] It can be seen that this embodiment uses a multi-level attention mechanism combined with a student self-review module to refine key knowledge from the shallow layer to the deep layer, which inherits the advantages of multi-level key learning and self-review. The results in Tables 1 and 2 show that compared with the baseline and classic knowledge distillation methods, the method of the present invention has achieved significant improvements on all teacher-student pairs. Regardless of whether the teacher and student models are homogeneous or heterogeneous, the method of the present invention is superior to most previous knowledge distillation methods.

[0128] 2.2 Results on the ImageNet dataset

[0129] This example experiments the method of the present invention on ImageNet, with the teacher model and the student model in a homogeneous or heterogeneous architecture. Two distillation architecture settings of the teacher model ResNet34 and the student model ResNet18, and the teacher model ResNet50 and the student model MobileNet-V1 were tried respectively. As shown in Table 3, the method of the present invention is compared with the previous knowledge distillation method, and the Top-1 and Top-5 accuracies are reported.

[0130] Table 3: Top-1 and Top-5 accuracy results on ImateNet

[0131] ;

[0132] The results in Table 3 show that the method of the present invention achieves significant improvement compared with other distillation methods.

[0133] 2.3 Results on the COCO dataset

[0134] The representative COCO2017 dataset is used to evaluate the method of the present invention. The implementation of MS-COCO follows the settings in the existing work (Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. arXiv preprint arXiv:2203.08679, 2022.). This embodiment uses the two-stage method FasterRCNN and FPN as feature extractors. ResNet model and MobileNet-V2 are selected as teacher model and student model, and the best pre-trained model provided by Detrctron2 is used as the baseline. As shown in Table 4, AP, AP 50 and AP 75 as evaluation metrics, and the proposed method is compared with previous knowledge distillation methods.

[0135] Table 4: Results on the MS-COCO dataset

[0136] ;

[0137] In the table, FGFI is the existing research: Tao Wang, Li Yuan, Xiaopeng Zhang, and JiashiFeng. “Distilling object detectors with fine-grained feature imitation,” InCVPR, 2019.

[0138] The results in Table 4 show that the knowledge distillation method of the present invention is superior to the existing knowledge distillation methods.

[0139] 3. Analysis

[0140] The above experimental results verify the effectiveness of the knowledge distillation method of the present invention in image classification and object detection. Then, the method of the present invention is further analyzed.

[0141] 3.1 Ablation Experiments on Distillation Framework

[0142] In the previous article, we proposed a knowledge distillation framework that combines multi-level attention and student self-review. In order to verify the effectiveness of each component in the framework, a series of ablation experiments are designed here. First, the multi-level attention module is removed from the overall framework. This module is responsible for extracting different levels of attention features from the image. Without the multi-level attention module, the model can only rely on the basic feature extraction mechanism, which will lead to performance degradation, especially when dealing with complex or fine-grained image tasks. Next, the student self-review module is removed, which allows the model to review previous knowledge to optimize the current feature extraction.

[0143] Through these ablation experiments, this embodiment found that the multi-level attention module is crucial for extracting rich image features, while the student self-review module plays an important role in utilizing past knowledge to enhance model performance. Please refer to Table 5. The dataset used in the experiment is CIFAR-100, the teacher model is WRN-40-2, the student model is WRN-16-2, and the Top-1 accuracy is used as the evaluation condition. The experimental results show that the best results can be obtained when the multi-level attention module and the student self-review module are combined together.

[0144] Table 5: Accuracy results of frame ablation experiments

[0145] ;

[0146] 3.2 Self-review ablation experiment

[0147] Please refer to Table 6. On the CIFAR-100 dataset, this embodiment uses ResNet20 as the student and ResNet56 as the teacher, and uses the Top-1 accuracy as the evaluation condition. ResNet20 and ResNet56 have four stages. This embodiment will gradually reduce the stage of student model review to analyze the effectiveness of cross-stage knowledge transfer.

[0148] Table 6: Accuracy results of self-review ablation experiments

[0149] ;

[0150] The experimental results show that the self-review of students from the previous stage to the current stage through cross-stage self-distillation is beneficial to improve the performance of knowledge distillation. The informative dark knowledge contained in the first few layers of the student network is effectively extracted, which helps promote the learning of subsequent layers in the student network. In addition, the experimental results also show that knowledge from each layer of the network is conducive to self-distillation. Therefore, the student self-review mechanism proposed in this invention is very effective.

[0151] 4. Visualization

[0152] This example uses ResNet32×4 as the teacher and ResNet8×4 as the student to perform t-SNE visualization of knowledge distillation on CIFAR-100. Figure 5 The t-SNE visualization results of feature learning of traditional knowledge distillation methods are shown. Figure 6 The t-SNE visualization results of feature learning of the method of the present invention are shown. The results show that the knowledge distillation method of the present invention has higher separability than the traditional knowledge distillation method.

[0153] 5. Summary

[0154] This embodiment proposes a self-review knowledge distillation method based on multi-level attention, which integrates the advantages of multi-stage learning and student self-reflection to address the shortcomings of current knowledge distillation methods. By introducing a multi-level attention mechanism and a self-review mechanism, this method enables the student model to obtain important information from the teacher model at different levels in each training stage, and to review and consolidate previously learned knowledge in each training stage, thereby further enhancing the understanding and mastery of key knowledge. Experimental results show that this method has achieved significant performance improvements on multiple benchmark datasets, verifying its effectiveness and superiority.

[0155] Example 2

[0156] See also Figure 7 This embodiment provides a self-review knowledge distillation system based on multi-level attention, including: a prediction processing module 11, a multi-level attention module 12, a student self-review module 13, a loss calculation module 14 and a model optimization module 15.

[0157] The prediction processing module 11 is used to use the student model and the converged teacher model to perform prediction processing on the sample images in the sample set respectively; wherein the prediction processing includes multiple processing stages, and each processing stage is executed by the corresponding network layer in the student model or the teacher model.

[0158] The multi-level attention module 12 is used to obtain the feature map output by the teacher model and the student model at each processing stage, and calculate the attention-based feature map, i.e., the attention map, of the teacher model and the student model at each processing stage.

[0159] The student self-review module 13 is used to perform feature fusion of the attention map of the student model at each processing stage with the attention map of the previous processing stage, and calculate the single distillation loss in the process of transferring knowledge from the teacher model to the student model at each processing stage based on the fusion results, thereby calculating the multi-level attention distillation loss.

[0160] The loss calculation module 14 is used to calculate the standard cross entropy loss and the basic distillation loss according to the output results after the prediction processing of the student model and the teacher model; the loss calculation module 14 is also used to weight the multi-level attention distillation loss, the standard cross entropy loss and the basic distillation loss to obtain the total training loss.

[0161] The model optimization module 15 is used to perform gradient descent and back propagation based on the total training loss to optimize the student model parameters, and then continuously optimize the learning model using the next batch of samples in the training set until the student model converges.

[0162] The knowledge distillation system can apply the self-review knowledge distillation method based on multi-level attention described in Example 1.

[0163] Example 3

[0164] This embodiment provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the self-review knowledge distillation method based on multi-level attention as described above are implemented.

[0165] like Figure 8 As shown, the computer terminal provided in this embodiment includes: at least one processor 101, and a memory 102 connected to the at least one processor 101. The specific connection medium between the processor 101 and the memory 102 is not limited in this embodiment. Figure 5 In the example, the processor 101 and the memory 102 are connected via the bus 100. The bus 100 is connected to the memory 102 via the bus 100. Figure 5 The bus 100 is represented by a bold line, and the connection between other components is only for illustration and is not intended to be limiting. The bus 100 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 In the figure, only one thick line is used, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 101 can also be called a controller, and there is no limitation on the name.

[0166] In this embodiment, the memory 102 stores instructions that can be executed by at least one processor 101. The at least one processor 101 can execute the knowledge distillation method discussed in Example 1 by executing the instructions stored in the memory 102. The processor 101 can implement Figure 8 The functions of each module in the system are shown.

[0167] Among them, the processor 101 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 102 and calling data stored in the memory 102, the various functions of the device and process data, the device can be monitored as a whole.

[0168] In one possible design, the processor 101 may include one or more processing units, and the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 101. In some embodiments, the processor 101 and the memory 102 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0169] Processor 101 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment. A general-purpose processor may be a microprocessor or any conventional processor. In combination with the steps of the knowledge distillation method in Example 1, the steps may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in processor 101.

[0170] The memory 102 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 102 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 102 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 102 in this embodiment can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0171] By designing and programming the processor 101, the code corresponding to the knowledge distillation method described in the above embodiment can be fixed into the chip, so that the chip can execute Figure 1 The steps of the knowledge distillation method of the embodiment shown are as follows: How to design and program the processor 101 is a technique known to those skilled in the art and will not be described in detail here.

[0172] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A self-review knowledge distillation method based on multi-level attention, characterized by: include: Using the student model and the converged teacher model, respectively, the sample images in the sample set are predicted and processed; wherein the prediction process includes multiple processing stages, and each processing stage is performed by a corresponding network layer in the student model or the teacher model; The feature maps output by the teacher model and the student model at each processing stage are obtained, and the feature maps based on attention of the teacher model and the student model at each processing stage, namely the attention maps, are calculated based on the feature maps. The calculation formulas are as follows: In the formula, is the feature map output by the teacher model at the i-th processing stage, C is the number of channels in the feature map, i∈[1,K], K is the total number of processing stages of the prediction process; is the attention map corresponding to the teacher model at the i-th processing stage; ||·||2 represents the second norm of the vector; vec(·) represents vector operation; It is the feature map output by the student model at the i-th processing stage; is the attention map corresponding to the student model at the i-th processing stage; The attention map of the student model at each processing stage is feature fused with the attention map of the previous processing stage. The fusion results are combined to calculate the single distillation loss in the process of transferring the knowledge of each processing stage from the teacher model to the student model, thereby calculating the multi-level attention distillation loss. The specific process includes: The attention map of the student model at each processing stage is fused with the attention map of the previous processing stage. The calculation formula of the fused feature is as follows: In the formula, represents the fusion features of the student model at the i-th processing stage, is a set of real numbers, H and W are the width and height of the attention map respectively; represents the initial feature fusion, represents the fusion weight; represents weighted multiplication; The single distillation loss in each processing stage is calculated as follows: In the formula, is the single distillation loss of the i-th processing stage; Calculate the multi-level attention distillation loss based on the single distillation loss of all sample images in the sample set at all processing stages The calculation formula is as follows: In the formula, represents the sample set; x i' is the i'th sample image in the sample set; Calculate the standard cross entropy loss and the basic distillation loss according to the output results of the prediction processing of the student model and the teacher model; weight the multi-level attention distillation loss, the standard cross entropy loss and the basic distillation loss to obtain the total training loss; Gradient descent and back propagation are performed based on the total training loss to optimize the student model parameters, and then the learning model is continuously optimized using the sample images of the next batch of the sample set until the student model converges.

2. The self-review knowledge distillation method based on multi-level attention according to claim 1 is characterized in that: The calculation process of the fusion weight is: The initial feature fusion results are respectively input into two convolutional layer processing units, and then the outputs of the two convolutional layer processing units are fused for the second time; each convolutional layer processing unit includes two point-by-point convolutional layers connected in sequence, a ReLu activation function is set between the two point-by-point convolutional layers, a batch normalization process is set before the second fusion, and a Sigmoid activation function is set after the second fusion.

3. The self-review knowledge distillation method based on multi-level attention according to claim 1, characterized in that: The standard cross entropy loss is calculated as follows: In the formula, represents the standard cross entropy loss; n is the total number of categories, j∈[1,n]; is the hard label of the i'th sample image; log(·) is the logarithmic function; is the probability value predicted by the student model for the jth category of the i'th sample image; The basic distillation loss is calculated as follows: In the formula, represents the basic distillation loss; is the soft label of the jth category of the teacher model for the i'th sample image; is the soft label of the jth category of the student model for the i'th sample image.

4. The self-review knowledge distillation method based on multi-level attention according to claim 3 is characterized in that: The total training loss is calculated as follows: In the formula, represents the total training loss; α and β are weight coefficients used to balance different distillation losses; τ is the temperature coefficient.

5. A self-review knowledge distillation system based on multi-level attention, characterized by: include: A prediction processing module, used to use the student model and the converged teacher model to perform prediction processing on the sample images in the sample set respectively; wherein the prediction processing includes multiple processing stages, each processing stage is executed by a corresponding network layer in the student model or the teacher model; A multi-level attention module is used to obtain the feature maps output by the teacher model and the student model at each processing stage, and calculate the attention-based feature maps of the teacher model and the student model at each processing stage, namely the attention maps; The student self-review module is used to fuse the attention map of the student model at each processing stage with the attention map of the previous processing stage, and calculate the single distillation loss in the process of transferring knowledge from the teacher model to the student model at each processing stage based on the fusion results, thereby calculating the multi-level attention distillation loss; A loss calculation module, used to calculate the standard cross entropy loss and the basic distillation loss according to the output results of the prediction processing of the student model and the teacher model; the loss calculation module is also used to weight the multi-level attention distillation loss, the standard cross entropy loss and the basic distillation loss to obtain a total training loss; A model optimization module, configured to perform gradient descent and back propagation based on the total training loss to optimize the student model parameters, and then continuously optimize the learning model using the next batch of samples in the training set until the student model converges; The system applies the self-review knowledge distillation method based on multi-level attention as described in any one of claims 1 to 4.

6. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the self-review knowledge distillation method based on multi-level attention as described in any one of claims 1 to 4.

Images