A Multidimensional Knowledge Distillation Method Based on Logit Cross-Calibration

Abstract

The present invention discloses a multi-dimensional knowledge distillation method based on Logit cross correction. First, the image to be classified is input into the primary feature extraction modules of the teacher model and the student model to obtain two primary features. Secondly, the two primary features are input into the class result perception module to obtain the adaptively weighted Logit scores and calculate the class result loss. Then, the adaptively weighted Logit scores and the primary features are input into the Logit cross correction module and the class generation perception module to calculate the correction loss and the class generation loss. Finally, the class result loss, the correction loss, and the class generation loss are weighted and fused through variable weights to obtain the final loss value, and the loss value is minimized through backpropagation to complete the multi-dimensional knowledge distillation. The present invention combines feature knowledge and Logit knowledge to solve the problems of poor interpretability of distillation and insufficient utilization of local semantic information in the knowledge distillation task.

Description

Technical Field

[0001] The present invention relates to the field of image classification based on knowledge distillation in deep learning, and particularly to a multi-dimensional knowledge distillation method based on Logit cross correction. Background Art

[0002] Image classification, as an important topic in the field of computer vision, plays a core role in multiple fields. For example, in autonomous driving technology, image classification can help identify roads, traffic signs, pedestrians, other vehicles, etc., to ensure the safe driving of vehicles. In the field of security monitoring, by classifying the monitored video images, face recognition, abnormal behavior detection, dangerous item identification, etc. can be achieved. With the development of image classification models, their internal structures, the number of parameters, and resource consumption have become increasingly complex and large. In some scenarios with limited terminal resources, such as mobile devices, embedded systems, intelligent cameras, etc., they usually do not have powerful computing capabilities. Traditional deep learning models require a large amount of computing resources and may not be directly deployed on these devices. To address the above problems, the knowledge distillation technology in deep learning can achieve model lightweight while maintaining model accuracy.

[0003] Knowledge distillation technology transfers the knowledge of a large-scale teacher model to a smaller student model. By reducing the loss values of the teacher model and the student model, the student model can reduce the computational burden while maintaining a high image classification accuracy. Knowledge distillation technology can be divided into two categories: feature-based knowledge distillation methods and Logit-based knowledge distillation methods. Among them, technologies such as ReviewKD (Distilling Knowledge via Knowledge Review), CAT-KD (Class Attention Transfer Based Knowledge Distillation) which are representative feature distillation methods, and DKD (Decoupled Knowledge Distillation) which is a representative Logit distillation method have achieved obvious effects in model lightweighting and improving the image classification accuracy of small models. However, these methods still have certain limitations in the knowledge transfer process. First, for feature-based distillation methods, the knowledge transferred in this process is often in a black-box state. Researchers cannot determine the specific meaning of high-dimensional features, so they cannot explain the effectiveness of image classification knowledge transfer. Second, for Logit-based distillation methods, the teacher model transfers classification results. This distillation method overly focuses on the main category in the image, lacks attention to the semantic information of local targets, and there is an overfitting phenomenon caused by the extreme output distribution of the model. Finally, existing methods have less research on the complementarity of these two types of knowledge distillation, and there is a lack of efficient fusion of knowledge in these two dimensions of feature-based and Logit-based knowledge distillation. Summary of the Invention

[0004] To address the above technical problems, the present invention proposes a multi-dimensional knowledge distillation method based on Logit cross-correction. This method uses a class result perception module to multi-scale divide the class results into multiple blocks of local information and perform adaptive attention weighting to strengthen the learning of class result knowledge between the teacher and the student, and uses adaptive Logit for cross-correcting the teacher and student features. The intermediate features are transformed into multi-layer class activation maps through a class generation perception module, and a learnable gated feature selection module is used to transfer the knowledge of the class generation process between the teacher and the student. Through the knowledge transfer in two dimensions, the student model can achieve a large accuracy improvement while maintaining lightweight. The present invention includes four modules: a primary feature extraction module, a class generation perception module, a class result perception module, and a Logit cross-correction module.

[0005] For the primary feature extraction module, it is used to extract the features of a specified intermediate layer of the model.

[0006] For the class generation perception module, this module includes 2 class activation map generation modules, 1 gated feature selection module, and 1 loss calculation module. Among them, the class activation map generation module converts the features of the teacher and student models into class activation maps corresponding to each class. In the gated feature selection module, learnable gated weights are generated to perform weighted update or forgetting operations on different features. In the loss calculation module, the loss is calculated by comparing the class attention regions of the teacher and student, enabling the student model to directly learn the class attention region of the teacher model. This method increases the interpretability of knowledge transfer while the student model learns class generation knowledge, solving the problem of low interpretability of knowledge transfer.

[0007] For the class result perception module, this module includes 2 multi-scale Logit generation modules, 1 adaptive Logit attention module, and 1 channel loss calculation module. Among them, the multi-scale Logit generation module divides the feature map into local feature maps of multiple sizes with different partitioning strategies, and generates local Logit scores from each local feature map. In the adaptive Logit attention module, learnable attention weights are generated based on the Logit scores to weight the local Logit scores. In the channel loss calculation module, a loss calculation channel is established between the corresponding local probability distributions of the teacher and student models for one-to-one loss calculation. This method solves the overfitting phenomenon caused by insufficient utilization of local semantics and extreme model output distributions.

[0008] For the Logit cross-correction module, this module includes 1 bidirectional projection module, 1 feature correction module, and 1 loss calculation module. Among them, the bidirectional projection module projects the adaptive Logit and primary features generated in the class result perception module into a common space for comparison. The feature correction module includes a correction matrix generation module and a primary feature correction module for performing Logit correction on the primary features, and the loss calculation module is used to calculate the gap of the bidirectional projection.

[0009] The present invention combines Logit-based adaptive attention and Logit cross-correction, enabling the student model to simultaneously learn the class generation process of the teacher model and the local fine-grained class generation results. Through the fusion of these two types of knowledge, the student model completely perceives multi-dimensional class knowledge, thereby learning more effective knowledge, and further enabling the model to balance lightweight and high precision.

[0010] The specific steps of the method of the present invention include:

[0011] S1 Input the image to be classified into the primary feature extraction modules of the teacher model and the student model to obtain the teacher primary features and the student primary features .

[0012] Among them, the superscript represents the teacher model, and the superscript represents the student model, and the same applies hereinafter.

[0013] S2 inputs the primary features and into the class result perception module to obtain the adaptively weighted Logit scores and calculate the class result loss. The class result perception module includes two multi-scale Logit generation modules, an adaptive Logit attention module, and a channel loss calculation module. The specific steps are as follows:

[0014] S21 The multi-scale Logit generation module contains a multi-scale partitioning layer and a fully connected layer. First, the input features and are respectively input into their respective multi-scale Logit generation modules. After the multi-scale partitioning layer performs the local average pooling operation with an adaptive size, local features of multiple sizes are output. Then, after passing through a fully connected layer, the preliminary local Logit scores are output. Denote the local Logit scores of the teacher model and the student model as and .

[0015] Among them, represents the index of the local features after partitioning.

[0016] S22 inputs and into the adaptive Logit attention module. This module includes an adaptive weight generation layer and an adaptive weighting layer, which perform adaptive attention weighting according to the Logit distribution. First, the local Logits of the teacher model and the student model respectively generate the attention weights adaptively, and then the adaptive weights are weighted to the corresponding local regions. The specific steps are as follows:

[0017] S221 In the adaptive weight generation layer, for each extracted from each local region, first apply the activation function to obtain the preliminary probabilities of each classification. Input these preliminary probabilities into a fully connected layer to obtain learnable weights during backpropagation. Finally, apply the activation function to obtain the final adaptive attention weights .

[0018] S222 In the adaptive weighting layer, weight the adaptive attention weights generated for each local region to the Logit scores and of the corresponding local regions to obtain the corresponding weighted Logit scores and 。

[0019] S23 feeds and into the channel loss calculation module. First, a one-to-one loss calculation is performed on the Logit scores of the corresponding regions of the teacher model and the student model to obtain the local distillation result. Then, the results of all local distillation losses under all partitioning strategies are calculated. The specific steps are as follows:

[0020] S231 For each local Logit score and use the activation function to obtain the classification probability, and then perform KL divergence loss calculation to obtain the loss value of the classification results of the corresponding local regions of the teacher model and the student model 。

[0021] S232 Calculate the overall loss of all local distillation losses under all partitioning strategies to obtain the class result loss 。

[0022] S3 feeds the Logit scores after adaptive weighting of the teacher model generated in step S222 and the primary features generated in step S1 and student features into the Logit cross-correction module to obtain the corrected teacher features and student features and calculate the correction loss.

[0023] S31 The Logit cross-correction module includes a bidirectional projection module, a loss calculation module, and a feature correction module. The specific steps include:

[0024] S311 Project the primary features and respectively with the teacher model's adaptive Logit scores using the projection layer and the activation function to project the primary features and the adaptive Logit scores into the common space for comparison. Denote as the projection vector of the adaptive Logit scores, and as the projection vectors of the primary features and and use to correct and 。

[0025] S312 In the loss calculation module, only project the student model and Adaptive Logit Projection Perform Wasserstein metric calculation, denoted as calibration loss 。

[0026] S313 In the feature calibration module, it includes a calibration matrix generation module and a primary feature calibration module. First, dynamically generate a calibration matrix through the teacher Logit projection, and then respectively calibrate the primary features and The specific steps are as follows:

[0027] S3131 When projecting the adaptive Logit score in step S311, select its projection vector to generate the calibration matrix 。

[0028] S3132 In the primary feature calibration module, use the calibration matrix to respectively calibrate the primary features and and introduce a matrix alignment operation to obtain the calibrated teacher features and student features 。

[0029] S4 Input the calibrated teacher features and student features into the class generation perception module to generate the teacher class activation map and calculate the class generation loss. The class generation perception module includes two class activation map generation modules, a gated feature selection module, and a loss calculation module. The specific steps include:

[0030] S41 Input the calibrated features of the teacher model into the class activation map generation module of the teacher model to generate the teacher class activation map; input the calibrated features of the student model into the class activation map generation module of the student model to generate the student class activation map. The class activation map generation module contains a convolutional layer and an adaptive size average pooling layer. The specific steps of the class activation map generation module include:

[0031] S411 The teacher model inputs its primary features into the class activation map generation module of the teacher model, and after passing through the convolutional layer, the teacher features with channel alignment are denoted as ; at the same time, input the primary features of the student model into the class activation map generation module of the student model, and after passing through the convolutional layer, the student features with channel alignment are denoted as 。

[0032] S412 feeds the aligned features into the adaptive size average pooling layer of the teacher model, and feeds the features into the adaptive size average pooling layer of the student model to obtain their respective class activation maps and .

[0033] S42 feeds the class activation maps of the teacher model and the student model and into the gated feature selection module, which includes a gated weight calculation module and a gated feature weighting module. The specific steps are as follows:

[0034] S421 feeds the class activation map into the gated weight calculation module, where it first passes through a fully connected layer and an activation function layer , obtains the classification probability of each class, and takes the classification probability as the learnable weight denoted as .

[0035] S422 In the gated feature weighting module, first set the gated weights below the threshold to zero, and keep the other part of the weights updated automatically. Then, multiply the new gated weights with the class activation maps input in step S42 and for dot product operation to perform gated feature selection, and obtain the gated class activation maps of the teacher model and the student model and .

[0036] S43 feeds the gated class activation maps of the teacher model and the student model obtained in step S422 and into the loss calculation module. The specific steps are as follows:

[0037] S431 First, the input gated class activation maps and pass through the normalization layer (), standardize the input, and then calculate the mean square error loss of the two results to obtain the loss value .

[0038] S5 weights and fuses the class result loss obtained in step S232, the calibration loss in step S312, and the class generation loss obtained in step S431 through variable weights to obtain the final loss value 。

[0039] S6 minimizes the loss values of the teacher model and the student model through backpropagation , completes multi-dimensional knowledge distillation, and improves the high-precision performance of image classification of the student model while keeping it lightweight by focusing on category knowledge and Logit cross-correction.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] In order to solve the problem of single knowledge content transfer during knowledge distillation between the teacher model and the student model in the field of image classification, the present invention proposes a multi-dimensional knowledge distillation method based on Logit cross-correction.

[0042] First, during the transfer process of category result knowledge, a category result perception module is proposed. The intermediate layer features are divided into multiple local feature maps of various sizes with different partitioning strategies, then the Logit classification scores of local information are generated from each local feature map, and then the Logit classification scores are adaptively weighted through a Logit-based adaptive attention mechanism. Finally, one-to-one loss calculation is performed on the corresponding local Logit classification scores of the teacher and the student, so that more attention is paid to local information during knowledge transfer and the model output is smoothed to prevent overfitting.

[0043] Secondly, the adaptive Logit classification score of the teacher model is used as a correction index to perform Logit correction on the primary features, so that the primary features of the teacher and the student conform to an accurate classification distribution, and more effective learning knowledge can be obtained in subsequent knowledge transfer.

[0044] Then, during the transfer process of category generation knowledge, a category generation perception module is proposed. By converting the black-box and difficult-to-interpret high-dimensional intermediate layer features into interpretable and intuitive class activation maps, and through a gating feature selection mechanism with learnable weights, adaptive weighting or forgetting operations are performed on different features, so that the student model focuses on learning the category attention areas of the teacher model.

[0045] Finally, the losses generated during the knowledge transfer process are weighted and fused, so that the student model can simultaneously learn the category generation process and the category classification results, and efficiently improve the accuracy of the model while keeping the model lightweight. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is the architecture diagram of the multi-dimensional knowledge distillation method based on Logit cross-correction;

[0047] Figure 2 is the structural diagram of the category result perception module;

[0048] Figure 3It is the structural diagram of the Logit cross - correction module;

[0049] Figure 4 It is the structural diagram of the class generation perception module. Specific implementation manners

[0050] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] As a preferred implementation form of the present invention, a multi - dimensional knowledge distillation method based on Logit cross - correction is provided, and its architecture is as Figure 1 shown, including the following steps:

[0052] S1 Input the image to be classified into the primary feature extraction modules of the teacher model and the student model. The primary feature extraction module of the teacher model selects the backbone networks of the ResNet, WRN, and VGG series, and the primary feature extraction module of the student model selects the backbone networks of the ResNet, WRN, VGG, ShuffleNet, and MobileNet series to obtain the teacher's primary features and the student's primary features .

[0053] Among them, the superscript represents the teacher model, and the superscript represents the student model, and the same hereinafter.

[0054] S2 Input the primary features and into the class result perception module.

[0055] S21 Input the teacher's primary features and the student's primary features in step S1 into the class result perception module to perform the transfer of classification result knowledge. The class result perception module includes two multi - scale Logit generation modules, an adaptive Logit attention module, and a channel loss calculation module, and its structure is as Figure 2 shown, and the specific steps include:

[0056] S211 The multi - scale Logit generation module includes a multi - scale division layer and a fully - connected layer. First, the input features and They are respectively input into their respective multi-scale Logit generation modules, and through a multi-scale division layer, an adaptive-size local average pooling operation is performed to output local features of multiple sizes. Then, through a fully connected layer, preliminary local Logit scores are output. Denote the local Logit scores of the teacher model and the student model as and . The specific calculation formula is as follows:

[0057]

[0058] where, and respectively represent the Logit scores generated by the teacher model and the student model on the th local region. According to different division strategies, the feature map is divided into local regions, , the local region index , . and are the weights of the fully connected layer, and represent the feature values of the corresponding teacher feature map and the student feature map at the

[0059] S212 inputs and into the adaptive Logit attention module. This module contains an adaptive weight generation layer and an adaptive weighting layer, and performs adaptive attention weighting according to the Logit distribution. First, the local Logits of the teacher model and the student model respectively adaptively generate attention weights, and then the adaptive weights are weighted to the corresponding local regions. The specific steps are as follows:

[0060] S2121 In the adaptive weight generation layer, for the extracted from each local region, first use the activation function to obtain the preliminary probability of each classification. Input these preliminary probabilities into a fully connected layer to obtain learnable weights in backpropagation, and finally use the activation function to obtain the final adaptive attention weights. The specific calculation formula is as follows:

[0061]

[0062] where, represents the weight adaptively generated on the th local region. represents the weight of the fully connected layer, , where represents the number of dataset categories, which is 100.

[0063] In the adaptive weighting layer, S2122 weights the adaptive weights generated for each local region to the Logit scores of the corresponding local region and to obtain the corresponding weighted Logit scores and . The specific calculation formula is as follows:

[0064]

[0065] , representing the Logit scores after adaptive weighting for the th local region of the teacher model and the student model.

[0066] S213 inputs and into the channel loss calculation module. First, a one-to-one loss calculation is performed on the Logit scores of the corresponding regions of the teacher model and the student model to obtain the local distillation result. Then, the results of all local distillation losses under all partitioning strategies are calculated. The specific steps are as follows:

[0067] S2131 For each local Logit score and , the activation function is used to obtain the classification probability, and then the KL divergence loss is calculated to obtain the loss value of the classification results for the corresponding local regions of the teacher model and the student model . The specific formula is as follows:

[0068]

[0069] S2132 calculates the overall loss of all local distillation losses under all partitioning strategies to obtain the class result loss . The specific calculation formula is as follows:

[0070]

[0071] Among them, represents the partitioning strategy in step S211.

[0072] S3 inputs the Logit scores after adaptive weighting of the teacher model generated in step S2122 and the primary features generated in step S1 and into the Logit cross-correction module.

[0073] The S31 Logit cross - correction module includes a bidirectional projection module, a loss calculation module, and a feature correction module. Its structure is as follows Figure 3 shown, and the specific steps are as follows:

[0074] S311 For the primary feature and respectively perform bidirectional projection with the teacher model's adaptive Logit score using the projection layer and activation function to project the primary feature and the adaptive Logit into a common space for comparison. Denote as the projection vector of the adaptive Logit, and as the projection vectors of the primary feature and . The specific formula is as follows:

[0075]

[0076]

[0077]

[0078] Among them , represents the projection layer weight, represents the bias term. Similarly, represents 's projection layer weight and bias term, represents feature dimension, represents 's projection layer weight and bias term, represents feature dimension.

[0079] S312 In the loss calculation module, only calculate the Wasserstein Distance between the student model projection and the adaptive Logit projection , denoted as the correction loss . The specific formula is as follows:

[0080]

[0081] Among them, is the transfer matrix, represents to 's Euclidean distance, represents all distribution sets that satisfy as the margin.

[0082] In the feature correction module, S313 includes a correction matrix generation module and a primary feature correction module. First, a correction matrix is dynamically generated through the teacher Logit projection, and then the primary features and are subjected to feature correction. The specific steps are as follows:

[0083] When projecting the adaptive Logit in step S311, select its projection vector to generate the correction matrix . The specific formula is as follows:

[0084]

[0085] where represents the transpose matrix of , is the regularization parameter, represents the identity matrix, ensuring that is a non-singular matrix, .

[0086] In the primary feature correction module, use the correction matrix to perform matrix correction on the primary features and respectively, and introduce the matrix alignment operation to obtain the corrected teacher features and student features . The specific formula is as follows:

[0087]

[0088]

[0089] S4 Input the corrected teacher features and student features into the class generation perception module.

[0090] The class generation perception module includes two class activation map generation modules, a gated feature selection module, and a loss calculation module. Its structure is as Figure 4 shown. The specific steps include:

[0091] S411 Input the corrected features of the teacher model into the class activation map generation module of the teacher model to generate the teacher class activation map; input the corrected features of the student model into the class activation map generation module of the student model to generate the student class activation map. The class activation map generation module contains a a convolutional layer and an adaptive average pooling layer. The specific steps of the class activation map generation module are as follows:

[0092] S4111 The teacher model inputs its primary features into the class activation map generation module of the teacher model. After passing through the convolutional layer, the number of channels of is aligned with the number of classes . The aligned teacher features are denoted as . At the same time, the primary features of the student model are input into the class activation map generation module of the student model. After passing through the convolutional layer, the number of channels of is aligned with the number of classes . The aligned student features are denoted as .

[0093] S4112 Input the aligned features into the adaptive average pooling layer of the teacher model, and input the features into the adaptive average pooling layer of the student model. This layer adaptively performs local average pooling operations according to and the sizes of the two features. By dividing the input features into local regions of equal size, each region is mapped to a pixel of the new feature to obtain their respective class activation maps and . For the class activation maps of the th channel, they are respectively denoted as and . The specific calculation formulas for steps S4111 to S4112 are as follows:

[0094]

[0095] where represents and mapped to a new pixel in the region, represents the area of the region, represents and the number of channels, represents the th channel's convolutional kernel weight. represents and in the th channel at the position's feature value.

[0096] S412 Input the class activation maps of the teacher model and the student model and into the gated feature selection module, which includes a gated weight calculation module and a gated feature weighting module. The specific steps are as follows:

[0097] S4121 Input the class activation map into the gated weight calculation module, where first, it passes through a fully connected layer and an activation function layer , and the classification probability of each class is obtained. The classification probability is used as the learnable weight. The specific formula is as follows:

[0098]

[0099] where is the preliminary gated weight, is the weight of the fully connected layer, is the bias term of the fully connected layer.

[0100] S4122 In the gated feature weighting module, first, the gated weights below the threshold are set to zero, and the other part of the weights is automatically updated. Then, the new gated weights are dot-multiplied with the class activation maps input in step S412 and for gated feature selection, strengthening the attention to important features and performing a forgetting operation on low-quality features. The specific formula is as follows:

[0101]

[0102]

[0103] where and respectively represent the initial gated weight and the screened gated weight of the th channel, represents the set threshold of 0.1. represents the class activation map after gated selection.

[0104] S413 Input the gated class activation maps of the teacher model and the student model obtained in step S4122 and into the loss calculation module. The specific steps are as follows:

[0105] S4131 First, input the gated class activation maps and through the normalization layer ( ), standardize the input, and then calculate the mean squared error loss between the two results to obtain the error value of the two inputs, that is, the loss value , and the specific calculation formula is:[[]]END]]

[0106]

[0107] where and respectively represent the gated class activation maps of the teacher model and the student model for the th channel.

[0108] S5 weights and fuses the class result loss obtained in step S2132, the calibration loss in step S312, and the class generation loss obtained in step S4131 through variable weights to obtain the final loss value . The specific calculation formula is as follows:[[]]END]]

[0109]

[0110] where , and respectively represent the weight factors of the class generation loss, the class result loss, and the calibration loss.

[0111] S6 minimizes the loss values of the teacher model and the student model through backpropagation , and by focusing on class knowledge and Logit cross-calibration, the student model is made lightweight while improving its high-precision performance in image classification.

[0112] Example:

[0113] The steps of this example are the same as those in the specific implementation manner, and will not be elaborated here. The following shows some implementation processes and implementation results.

[0114] The technology of the present invention is implemented on the CIFAR-100 dataset, which contains 100 classes and a total of 60,000 general images of 32x32 pixels, specifically divided into 50,000 training images and 10,000 test images, with 500 training images and 100 test images for each class. This model uses a batch size of 64, 240 training epochs, and the SGD (Stochastic Gradient Descent) optimization algorithm, with the learning rate set to 0.05. To verify the effectiveness of the technology of the present invention, it is compared with the following methods respectively: KD (Distilling the Knowledge in a Neural Network): a knowledge distillation based on neural networks, DKD (Decoupled Knowledge Distillation): a knowledge distillation based on decoupling, FitNet (FitNets: Hints for Thin Deep Nets): a knowledge distillation for narrow and deep neural networks, CRD (Contrastive representation distillation): a knowledge distillation based on contrastive representation information, ReviewKD (Distilling Knowledge via Knowledge Review): a knowledge distillation based on knowledge review, CAT-KD (Class Attention Transfer Based Knowledge Distillation): a knowledge distillation based on class attention transfer.

[0115] To verify the effectiveness of the technology of the present invention, Tables 1 and 2 use the metric Accuracy to compare the technology of the invention and the existing technologies. As shown in Tables 1 and 2, the technology of the present invention has achieved good results in both heterogeneous teacher-student pair distillation and homogeneous teacher-student pair distillation, thus proving the effectiveness of the technology of the present invention.

[0116] Table 1 Comparison results between the technology of the present invention and the existing technologies in heterogeneous teacher-student pairs

[0117]

[0118] Table 2 Comparison results between the technology of the present invention and the existing technologies in homogeneous teacher-student pairs

[0119]

Claims

1. A multidimensional knowledge distillation method based on Logit cross correction, characterized in that: The following steps are involved: S1 inputs the image to be classified into the primary feature extraction module of the teacher model and the student model to obtain the teacher primary feature and primary characteristics of students S2 and Input into the category result perception module to obtain the adaptive weighted Logit score and calculate the category result loss; S3 combines adaptively weighted Logit scores and primary features and Input into the Logit cross correction module to obtain the corrected teacher characteristics and student characteristics And calculate the correction loss; S4 will correct the teacher characteristics and student characteristics Input into the category generation perception module to generate the teacher category activation map and calculate the category generation loss; S5 weightedly fuses the category result loss, correction loss and category generation loss through weights to obtain the final loss value, minimizes the loss value through back propagation, and completes multi-dimensional knowledge distillation.

2. The multidimensional knowledge distillation method based on Logit cross correction according to claim 1 is characterized in that: The category result perception module includes two multi-scale Logit generation modules, an adaptive Logit attention module and a channel loss calculation module, which are specifically implemented as follows: The S21 multi-scale Logit generation module contains a multi-scale partitioning layer and a fully connected layer; first, the input feature and They are respectively input into their respective multi-scale Logit generation modules, and then the local average pooling operation of adaptive size is performed through the multi-scale division layer to output local features of various sizes. After that, they are passed through a fully connected layer to output the preliminary local Logit scores. The local Logit scores of the teacher model and the student model are recorded as Logit T (m′) and Logit S (m′), where m′ represents the index of the local feature after division; S22 Logit T (m′) and Logit S (m′) is input into the adaptive Logit attention module, which contains an adaptive weight generation layer and an adaptive weighting layer, and performs adaptive attention weighting according to the Logit distribution; first, the attention weights are adaptively generated by the local Logit of the teacher model and the student model, and then the adaptive weights are weighted to the corresponding local area to obtain the corresponding weighted Logit score and S23 and Input into the channel loss calculation module, first perform a one-to-one loss calculation on the Logit scores of the corresponding areas of the teacher model and the student model to obtain the local distillation results, and then calculate the results of all local distillation losses under all partitioning strategies.

3. The multidimensional knowledge distillation method based on Logit cross correction according to claim 2 is characterized in that: The specific implementation process of step S22 is as follows: In the adaptive weight generation layer S221, for each local region extracted Logit(m′), the activation function f is first used for it. Softmax (), get the preliminary probability of each classification, input these preliminary probabilities into a fully connected layer so that it can obtain learnable weights in back propagation, and finally use the activation function f Softmax (), obtain the final adaptive attention weight; S222 In the adaptive weighting layer, the adaptive attention weight generated by each local area is weighted to the Logit score Logit of the corresponding local area T (m′) and Logit S (m′), and obtain the corresponding weighted Logit score and 4. The multidimensional knowledge distillation method based on Logit cross correction according to claim 3 is characterized in that: The specific implementation process of step S23 is as follows: S231 For each local Logit score and Use the activation function f Softmax (), get the classification probability, and then perform KL divergence loss calculation to get the loss value of the local area classification result corresponding to the teacher model and the student model S232 calculates the overall loss of all local distillation losses under all partitioning strategies and obtains the category result loss 5. The multidimensional knowledge distillation method based on Logit cross correction according to claim 4 is characterized in that: The Logit cross correction module includes a bidirectional projection module, a loss calculation module and a feature correction module. The specific implementation process is as follows: S31 for primary features and Respectively with the teacher model adaptive Logit score Perform bidirectional projection, using the projection layer and activation function f Softmax () Project the primary features and adaptive Logit scores into the common space for comparison. is the projection vector of the adaptive Logit score, and Primary feature and Projection vector, using right and Make corrections; S32 In the loss calculation module, only the student model is projected and Adaptive Logit Projection Perform the Wasserstein metric calculation, denoted as the correction loss The S33 feature correction module includes a correction matrix generation module and a primary feature correction module. First, the correction matrix is ​​dynamically generated through the teacher Logit projection, and then the primary features are respectively and Perform feature correction.

6. The multidimensional knowledge distillation method based on Logit cross correction according to claim 5, characterized in that: The specific implementation process of step S33 is as follows: S3131 in adaptive logit score When projecting, select its projection vector Generate the correction matrix Matrix; the specific formula is as follows: in, express The transposed matrix of , λ is the regularization parameter, and I represents the identity matrix; In the primary feature correction module, the correction matrix Matrix is ​​used to correct the primary features. and Perform matrix correction and introduce matrix alignment operation to obtain the corrected teacher features and student characteristics 7. The multidimensional knowledge distillation method based on Logit cross correction according to claim 6, characterized in that: The category generation perception module includes two category activation map generation modules, a gated feature selection module and a loss calculation module. The specific implementation process is as follows: S41 corrects the features of the teacher model Input into the category activation map generation module of the teacher model to generate the teacher category activation map; the correction features of the student model Input into the class activation map generation module of the student model to obtain the class activation map CAM T and CAM S ; S42 CAM the category activation map of the teacher model and the student model T and CAM S Input into the gated feature selection module to obtain the gated category activation map of the teacher model and the student model and S43 obtains the activation map of the teacher model and the student model gate category and Input to the loss calculation module, input gated category activation map and After the normalization layer, the input is standardized, and then the two results are calculated by the mean square error loss to get the loss value 8. The multidimensional knowledge distillation method based on Logit cross correction according to claim 7, characterized in that: The class activation map generation module includes a convolution layer and an adaptive size average pooling layer, which is specifically implemented as follows: The S411 teacher model takes its primary features Input into the category activation map generation module of the teacher model, after the convolution layer, the channel-aligned teacher feature is obtained and recorded as F T ; At the same time, the primary features of the student model Input into the category activation map generation module of the student model, after the convolution layer, the student features after channel alignment are obtained and recorded as F S ; S412 will feature F T Input to the adaptive size average pooling layer of the teacher model, the feature F S Input to the adaptive size average pooling layer of the student model to obtain the respective category activation map CAM T and CAM S .

9. The multidimensional knowledge distillation method based on Logit cross correction according to claim 8, characterized in that: The gated feature selection module includes a gated weight calculation module and a gated feature weighting module, and the specific implementation process is as follows: S421 inputs the class activation map CAM into the gated weight calculation module, in which the CAM first passes through a fully connected layer and an activation function layer f Softmax (), get the classification probability of each category, and record the classification probability as a learnable weight as G; S422 In the gated feature weighting module, the gated weights below the threshold are first reset to zero, and the other weights are automatically updated; then the new gated weight G is calculated. ′ The class activation map CAM input in step S42 T and CAM S Perform point multiplication and gated feature selection to obtain the category activation maps of the teacher model and the student model. and

Images