Multi-dimensional knowledge distillation method based on Logit cross correction

Abstract

The invention discloses a multi-dimensional knowledge distillation method based on Logit cross correction, and the method comprises the steps: inputting a to-be-classified image into a primary feature extraction module of a teacher model and a primary feature extraction module of a student model, and obtaining two primary features; secondly, inputting the two primary features into a category result sensing module to obtain a self-adaptive weighted Logit score, and calculating category result loss; and then inputting the adaptively weighted Logit score and the primary feature into a Logit cross correction module and a category generation perception module, and calculating correction loss and category generation loss. And finally, performing weighted fusion on the category result loss, the correction loss and the category generation loss through variable weights to obtain a final loss value, and minimizing the loss value through back propagation to complete multi-dimensional knowledge distillation. According to the method, feature knowledge and Logit knowledge are combined, so that the problems of poor distillation interpretability and insufficient utilization of local semantic information in a knowledge distillation task are solved.

Description

Technical Field

[0001] The present invention relates to the field of image classification based on knowledge distillation in deep learning, and in particular to a multidimensional knowledge distillation method based on Logit cross correction. Background Art

[0002] Image classification, as an important topic in the field of computer vision, plays a core role in many fields. For example, in autonomous driving technology, image classification can help identify roads, traffic signs, pedestrians, other vehicles, etc., to ensure safe driving of vehicles. In the field of security monitoring, face recognition, abnormal behavior detection, and dangerous goods identification can be achieved by classifying surveillance video images. With the development of image classification models, their internal structure, parameter quantity, and resource consumption are becoming increasingly complex and large. In some scenarios with limited terminal resources, such as mobile devices, embedded systems, and smart cameras, they usually do not have strong computing power. Traditional deep learning models require a lot of computing resources and may not be directly deployed on these devices. In response to the above problems, the knowledge distillation technology in deep learning can achieve lightweight models while maintaining model accuracy.

[0003] Knowledge distillation technology transfers the knowledge of a large-scale teacher model to a smaller student model. By reducing the loss values ​​of the teacher model and the student model, the student model can reduce the computational burden while maintaining a high image classification accuracy. Knowledge distillation technology can be divided into two categories: feature-based knowledge distillation methods and logit-based knowledge distillation methods. Among them, feature-based distillation methods represented by ReviewKD (Distilling Knowledge via Knowledge Review) and CAT-KD (Class Attention Transfer Based Knowledge Distillation) and logit-based distillation methods represented by DKD (Decoupled Knowledge Distillation) have achieved significant results in model lightweighting and improving the image classification accuracy of small models. However, these methods still have certain limitations in the process of knowledge transfer. First, for feature-based distillation methods, the knowledge transferred in this process is often in a black box state. Researchers cannot determine the specific meaning of high-dimensional features, and cannot explain the effectiveness of image classification knowledge transfer. Secondly, for the Logit-based distillation method, the teacher model transfers the classification results. This distillation method pays too much attention to the main category in the image, lacks attention to the semantic information of local targets, and has overfitting caused by the extreme distribution of model output. Finally, the existing methods have little research on the complementarity of these two types of knowledge distillation, and there is a lack of efficient fusion of the two dimensions of knowledge distillation, feature-based and Logit-based. Summary of the invention

[0004] In view of the above technical problems, the present invention proposes a multi-dimensional knowledge distillation method based on Logit cross correction. This method divides the category results into multiple blocks of local information at multiple scales and performs adaptive attention weighting through a category result perception module, strengthens the category result knowledge learning between teachers and students, and uses adaptive Logit to cross-correct teacher and student features. The intermediate features are converted into multi-layer category activation maps through a category generation perception module, and the knowledge transfer of the category generation process between teachers and students is performed using a learnable gated feature selection module. Through the knowledge transfer in two dimensions, the student model obtains a significant improvement in accuracy while maintaining lightweight. The present invention comprises 4 modules: a primary feature extraction module, a category generation perception module, a category result perception module, and a Logit cross correction module.

[0005] For the primary feature extraction module, it is used to extract features of the specified layer in the middle of the model.

[0006] For the category generation perception module, the module contains two category activation map generation modules, one gated feature selection module and one loss calculation module. Among them, the category activation map generation module converts the features of the teacher and student models into category activation maps corresponding to each category. In the gated feature selection module, learnable gate weights are generated and weighted updates or forgetting operations are performed on different features. In the loss calculation module, loss calculation is performed by comparing the category attention areas of the teacher and the student, so that the student model can directly learn the category attention areas of the teacher model. This method enables the student model to increase the interpretability of knowledge transfer while learning category generation knowledge, solving the problem of low interpretability of knowledge transfer.

[0007] For the category result perception module, the module contains 2 multi-scale Logit generation modules, 1 adaptive Logit attention module and 1 channel loss calculation module. Among them, the multi-scale Logit generation module divides the feature map into local feature maps of various sizes with different partitioning strategies, and generates local Logit scores from each local feature map. In the adaptive Logit attention module, learnable attention weights are generated according to the Logit scores, and the local Logit scores are weighted. In the channel loss calculation module, a loss calculation channel is established between the local probability distributions corresponding to the teacher and student models, and a one-to-one loss calculation is performed. This method solves the overfitting phenomenon caused by insufficient utilization of local semantics and extreme distribution of model output.

[0008] For the Logit cross correction module, the module includes a bidirectional projection module, a feature correction module and a loss calculation module. The bidirectional projection module projects the adaptive Logit and primary features generated in the category result perception module into a common space for comparison. The feature correction module includes a correction matrix generation module and a primary feature correction module for performing Logit correction on the primary features, and the loss calculation module is used to calculate the gap of the bidirectional projection.

[0009] The present invention combines Logit-based adaptive attention and Logit cross-correction to enable the student model to simultaneously learn the category generation process and local fine-grained category generation results of the teacher model. By fusing the two types of knowledge, the student model can fully perceive the multidimensional category knowledge, thereby learning more effective knowledge, thereby making the model both lightweight and high-precision.

[0010] The method of the present invention specifically comprises the following steps:

[0011] S1 Input the image to be classified into the primary feature extraction module of the teacher model and the student model to obtain the teacher primary feature and primary characteristics of students .

[0012] Among them, the superscript represents the teacher model, the superscript Represents the student model, the same below.

[0013] S2 will be the primary feature and Input into the category result perception module to obtain the adaptive weighted Logit score and calculate the category result loss. The category result perception module includes two multi-scale Logit generation modules, an adaptive Logit attention module and a channel loss calculation module. The specific steps include:

[0014] S21 Multi-scale Logit generation module contains a multi-scale partitioning layer and a fully connected layer. First, input feature and They are input into their respective multi-scale Logit generation modules, and then go through the multi-scale division layer to perform local average pooling operations of adaptive size, output local features of various sizes, and then go through a fully connected layer to output preliminary local Logit scores. The local Logit scores of the teacher model and the student model are recorded as and .

[0015] in, Indicates the index of the local feature after division.

[0016] S22 will and Input to the adaptive Logit attention module, which contains an adaptive weight generation layer and an adaptive weighting layer, and performs adaptive attention weighting according to the Logit distribution. First, the attention weights are adaptively generated by the local Logit of the teacher model and the student model, and then the adaptive weights are weighted to the corresponding local areas. The specific steps are as follows:

[0017] S221 In the adaptive weight generation layer, for each local area extracted , first use the activation function , get the preliminary probability of each classification, input these preliminary probabilities into a fully connected layer to make it obtain learnable weights in back propagation, and finally use the activation function , get the final adaptive attention weight .

[0018] S222 In the adaptive weighting layer, the adaptive attention weight generated by each local area is weighted to the Logit score of the corresponding local area and On the top, we get the corresponding weighted Logit score and .

[0019] S23 will and Input into the channel loss calculation module, first, perform a one-to-one loss calculation on the Logit scores of the corresponding areas of the teacher model and the student model to obtain the local distillation results. Then, calculate the results of all local distillation losses under all partitioning strategies. The specific steps are as follows:

[0020] S231 For each local Logit score and Using activation functions , get the classification probability, and then calculate the KL divergence loss to get the loss value of the classification results of the local area corresponding to the teacher model and the student model .

[0021] S232 Calculate the overall loss of all local distillation losses under all partitioning strategies to obtain the category result loss .

[0022] S3 Adaptively weight the Logit score of the teacher model generated in step S222 And the primary features generated in step S1 and Input into the Logit cross correction module to obtain the corrected teacher characteristics and student characteristics , and calculate the correction loss.

[0023] The S31 Logit cross correction module includes a bidirectional projection module, a loss calculation module and a feature correction module. The specific steps include:

[0024] S311 Primary Features and Respectively with the teacher model adaptive Logit score Perform bidirectional projection, using projection layer and activation function Project the primary features and adaptive Logit scores into a common space for comparison. is the projection vector of the adaptive Logit score, and Primary feature and Projection vector, using right and Make corrections.

[0025] S312 In the loss calculation module, only the student model is projected and Adaptive Logit Projection Perform the Wasserstein metric calculation, denoted as the correction loss .

[0026] S313 In the feature correction module, there are a correction matrix generation module and a primary feature correction module. First, the correction matrix is ​​dynamically generated through the teacher Logit projection, and then the primary features are respectively and Perform feature correction. The specific steps are as follows:

[0027] S3131 In step S311, the adaptive Logit score When projecting, select its projection vector Correction Matrix Generation of.

[0028] S3132 In the primary feature correction module, use the correction matrix Primary features and Perform matrix correction and introduce matrix alignment operations , and obtain the corrected teacher characteristics and student characteristics .

[0029] S4 Corrected teacher characteristics and student characteristics Input into the category generation perception module, generate the teacher category activation map, and calculate the category generation loss. The category generation perception module includes two category activation map generation modules, a gated feature selection module and a loss calculation module. The specific steps include:

[0030] S41 corrects the features of the teacher model Input into the category activation map generation module of the teacher model to generate the teacher category activation map; the correction features of the student model The class activation map generation module input to the student model generates a student class activation map. The class activation map generation module includes a convolution layer and an adaptive size average pooling layer. The specific steps of the class activation map generation module include:

[0031] S411 The teacher model takes its primary features Input into the category activation map generation module of the teacher model, after the convolution layer, the channel-aligned teacher feature is obtained and recorded as ; At the same time, the primary features of the student model The input is sent to the category activation map generation module of the student model, and after the convolution layer, the student features after channel alignment are obtained and recorded as .

[0032] S412 Align the features Input to the adaptive size average pooling layer of the teacher model, the feature Input to the adaptive size average pooling layer of the student model to obtain the respective category activation map and .

[0033] S42 The category activation map of the teacher model and the student model and Input to the gated feature selection module, which includes the gated weight calculation module and the gated feature weighting module. The specific steps include:

[0034] S421 Activate the category map Input to the gated weight calculation module, in which First, it passes through a fully connected layer and an activation function layer , get the classification probability of each category, and record the classification probability as the learnable weight as .

[0035] S422 In the gated feature weighting module, firstly, the gated weights below the threshold are reset to zero, and the other weights are automatically updated. Then, the new gated weights are The class activation map input in step S42 and Perform point multiplication operation and gated feature selection to obtain the gated category activation map of the teacher model and the student model and .

[0036] S43 The gated category activation maps of the teacher model and the student model obtained in step S422 and Input into the loss calculation module, the specific steps include:

[0037] S431 First, input the gated category activation map and After the normalization layer (), standardize the input, and then calculate the mean square error loss of the two results to get the loss value .

[0038] S5 Loss of the category result obtained in step S232 , the correction loss in step S312 And the category generation loss obtained in step S431 The final loss value is obtained by weighted fusion through variable weights .

[0039] S6 Minimize the loss of the teacher model and the student model through back propagation , completing multi-dimensional knowledge distillation, by focusing on category knowledge and Logit cross correction, the student model remains lightweight while improving its high-precision performance in image classification.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] In order to solve the problem of single knowledge content being transferred during knowledge distillation of teacher model and student model in the field of image classification, the present invention proposes a multidimensional knowledge distillation method based on Logit cross correction.

[0042] Firstly, in the process of category result knowledge transfer, a category result perception module is proposed, which divides the intermediate layer features into local feature maps of various sizes with different partitioning strategies, and then generates the Logit classification score of local information from each local feature map. Then, a Logit-based adaptive attention mechanism is used to adaptively weight the Logit classification score. Finally, a one-to-one loss calculation is performed on the corresponding local Logit classification scores of the teacher and the student, so that more attention is paid to local information during knowledge transfer and the model output is smoothed to prevent overfitting.

[0043] Secondly, the adaptive Logit classification score of the teacher model is used as a correction indicator to perform Logit correction on the primary features, so that the primary features of the teacher and the student conform to the accurate classification distribution and obtain more effective learning knowledge in the subsequent knowledge transfer.

[0044] Then, in the process of category generation knowledge transfer, a category generation perception module is proposed. It transforms the black-box, difficult-to-interpret high-dimensional intermediate layer features into interpretable and intuitive category activation maps, and adaptively weights or forgets different features through a gated feature selection mechanism with learnable weights, so that the student model can focus on learning the category attention area of ​​the teacher model.

[0045] Finally, the losses generated by the knowledge transfer in the process are weighted and fused, so that the student model can learn the category generation process and category classification results at the same time, which can effectively improve the accuracy of the model while keeping the model lightweight. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is the architecture diagram of the multidimensional knowledge distillation method based on Logit cross correction;

[0047] Figure 2 This is the structural diagram of the category result perception module;

[0048] Figure 3This is the structure diagram of the Logit cross-correction module;

[0049] Figure 4 Generate a perception module structure diagram for the category. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] As a preferred implementation form of the present invention, a multidimensional knowledge distillation method based on Logit cross correction is provided, and its architecture is as follows: Figure 1 As shown, the following steps are included:

[0052] S1 Input the image to be classified into the primary feature extraction modules of the teacher model and the student model. The primary feature extraction module of the teacher model selects the backbone network of the ResNet, WRN, and VGG series, and the primary feature extraction module of the student model selects the backbone network of the ResNet, WRN, VGG, ShuffleNet, and MobileNet series to obtain the primary feature of the teacher. and primary characteristics of students .

[0053] Among them, the superscript represents the teacher model, the superscript Represents the student model, the same below.

[0054] S2 will be the primary feature and Input into the category result perception module.

[0055] S21 The teacher's primary features in step S1 and primary characteristics of students The input is sent to the category result perception module to transfer the classification result knowledge. The category result perception module includes two multi-scale Logit generation modules, an adaptive Logit attention module and a channel loss calculation module. Its structure is as follows: Figure 2 As shown, the specific steps include:

[0056] S211 The multi-scale Logit generation module contains a multi-scale partitioning layer and a fully connected layer. First, the input feature and They are input into their respective multi-scale Logit generation modules, and then pass through a multi-scale division layer to perform local average pooling operations of adaptive size, output local features of multiple sizes, and then pass through a fully connected layer to output preliminary local Logit scores. The local Logit scores of the teacher model and the student model are recorded as and The specific calculation formula is as follows:

[0057]

[0058] in, and Represents the teacher model and the student model in The Logit scores generated on the local areas are divided into local area, , local area index , . and is the weight of the fully connected layer, and express Position corresponding to the teacher's characteristic map and student characteristics graph The characteristic value of .

[0059] S212 and Input to the adaptive Logit attention module, which contains an adaptive weight generation layer and an adaptive weighting layer, and performs adaptive attention weighting according to the Logit distribution. First, the attention weights are adaptively generated by the local Logit of the teacher model and the student model, and then the adaptive weights are weighted to the corresponding local areas. The specific steps are as follows:

[0060] S2121 In the adaptive weight generation layer, for each local area extracted , first use the activation function , get the preliminary probability of each classification, input these preliminary probabilities into a fully connected layer to make it obtain learnable weights in back propagation, and finally use the activation function , and obtain the final adaptive attention weight. The specific calculation formula is as follows:

[0061]

[0062] in, Indicates The weights are adaptively generated on the local regions. represents the weight of the fully connected layer, ,in Indicates that the number of data set categories is 100.

[0063] S2122 In the adaptive weighting layer, the adaptive weight generated for each local region is weighted to the Logit score of the corresponding local region and On the top, we get the corresponding weighted Logit score and , the specific calculation formula is as follows:

[0064]

[0065] , indicating that the teacher model and the student model Logit score after adaptive weighting of local regions.

[0066] S213 will and Input into the channel loss calculation module, first, perform a one-to-one loss calculation on the Logit scores of the corresponding areas of the teacher model and the student model to obtain the local distillation results. Then, calculate the results of all local distillation losses under all partitioning strategies. The specific steps are as follows:

[0067] S2131 For each local Logit score and Using activation functions , get the classification probability, and then calculate the KL divergence loss , get the loss value of the classification results of the local area corresponding to the teacher model and the student model The specific formula is as follows:

[0068]

[0069] S2132 Calculate the overall loss of all local distillation losses under all partitioning strategies to obtain the category result loss The specific calculation formula is as follows:

[0070]

[0071] in, Indicates the division strategy in step S211.

[0072] S3 Adaptively weight the Logit score of the teacher model generated in step S2122 And the primary features generated in step S1 and Input to the Logit Cross Correction module.

[0073] The S31 Logit cross correction module includes a bidirectional projection module, a loss calculation module and a feature correction module, and its structure is as follows: Figure 3 As shown, the specific steps include:

[0074] S311 Primary Features and Respectively with the teacher model adaptive Logit score Perform bidirectional projection, using projection layer and activation function The primary features and adaptive Logit are projected into the common space for comparison. is the projection vector of adaptive Logit, and Primary feature and Projection vector, the specific formula is as follows:

[0075]

[0076]

[0077]

[0078] in , represents the projection layer weight, represents the bias term. Similarly, express The projection layer weights and bias terms, express Feature dimensions, express The projection layer weights and bias terms, express Feature dimension.

[0079] S312 In the loss calculation module, only the student model is projected and Adaptive Logit Projection Calculate the Wasserstein Distance, recorded as the correction loss , the specific formula is as follows:

[0080]

[0081] in, is the transmission matrix, express arrive The Euclidean distance of Indicates that all satisfaction is the set of marginal distributions.

[0082] S313 In the feature correction module, there are a correction matrix generation module and a primary feature correction module. First, the correction matrix is ​​dynamically generated through the teacher Logit projection, and then the primary features are respectively and Perform feature correction. The specific steps are as follows:

[0083] S3131 When the adaptive Logit is projected in step S311, select its projection vector Correction Matrix The specific formula for the generation of is as follows:

[0084]

[0085] in, express The transposed matrix of is the regularization parameter, represents the identity matrix, ensuring is a non-singular matrix, .

[0086] S3132 In the primary feature correction module, use the correction matrix Primary features and Perform matrix correction and introduce matrix alignment operations , and obtain the corrected teacher characteristics and student characteristics , the specific formula is as follows:

[0087]

[0088]

[0089] S4 Corrected teacher characteristics and student characteristics Input into the category generation perception module.

[0090] The S41 category generation perception module includes two category activation map generation modules, a gated feature selection module and a loss calculation module. Its structure is as follows: Figure 4 The specific steps include:

[0091] S411 corrects the features of the teacher model Input into the category activation map generation module of the teacher model to generate the teacher category activation map; the correction features of the student model The class activation map generation module is input to the student model to generate the student class activation map. The class activation map generation module contains a The specific steps of the class activation map generation module include:

[0092] S4111 The teacher model takes its primary features Input into the category activation map generation module of the teacher model, after Convolutional layer, Number of channels With the number of categories Alignment, the teacher characteristics after alignment are recorded as ; At the same time, the primary features of the student model Input into the category activation map generation module of the student model, after Convolutional layer, Number of channels With the number of categories Alignment, the aligned student features are recorded as .

[0093] S4112 Align the features Input to the adaptive size average pooling layer of the teacher model, the feature Input to the adaptive size average pooling layer of the student model. This layer is based on and The sizes of the two features are adaptively localized and averaged, and the input features are divided into local regions of equal size. Each region is mapped to a pixel of the new feature to obtain the respective category activation maps. and , for the The category activation maps of each channel are recorded as and , the specific calculation formula from step S4111 to step S4112 is:

[0094]

[0095] in, express and exist A new pixel in the region map, Indicates the area of ​​the region, express and The number of channels, Indicates The convolution kernel weights for each channel. express and In the In the channel The eigenvalue at the location.

[0096] S412 The category activation map of the teacher model and the student model and Input to the gated feature selection module, which includes the gated weight calculation module and the gated feature weighting module. The specific steps include:

[0097] S4121 Activate the category map Input to the gated weight calculation module, in which First, it passes through a fully connected layer and an activation function layer , get the classification probability of each category, and use the classification probability as the learnable weight. The specific formula is as follows:

[0098]

[0099] in, is the initial gating weight, is the weight of the fully connected layer, is the bias term of the fully connected layer.

[0100] S4122 In the gated feature weighting module, firstly, the gated weights below the threshold are reset to zero, and the other weights are automatically updated. Then, the new gated weights are The class activation map input in step S412 and The dot product operation is used to perform gated feature selection, which focuses on important features and forgets low-quality features. The specific formula is as follows:

[0101]

[0102]

[0103] in, and Respectively represent The initial gating weights of each channel and the screened gating weights, Indicates setting the threshold to 0.1. Represents the class activation map after gate selection.

[0104] S413 The gated category activation maps of the teacher model and the student model obtained in step S4122 and Input into the loss calculation module, the specific steps include:

[0105] S4131 First, input the gated category activation map and After the normalization layer (), standardize the input, and then calculate the mean square error loss of the two results to get the error value of the two inputs, that is, the loss value , the specific calculation formula is:

[0106]

[0107] in and Represents the teacher model and the student model Gated class activation map for each channel.

[0108] S5 Loss of the category result obtained in step S2132 , the correction loss in step S312 And the category generation loss obtained in step S4131 The final loss value is obtained by weighted fusion through variable weights The specific calculation formula is as follows:

[0109]

[0110] in, , and Represent the weight factors of category generation loss, category result loss and correction loss respectively.

[0111] S6 Minimize the loss of the teacher model and the student model through back propagation , by focusing on category knowledge and Logit cross correction, the student model remains lightweight while improving its high-precision performance in image classification.

[0112] Example:

[0113] The steps of this embodiment are the same as those of the specific implementation method, and will not be described in detail here. The following is a demonstration of some implementation processes and implementation results.

[0114] The technology of the present invention is implemented on the CIFAR-100 dataset, which contains 100 categories and a total of 60,000 32x32 pixel general images, which are specifically divided into 50,000 training images and 10,000 test images, with 500 training images and 100 test images for each category. This model uses a batch size of 64, a training round number of 240, and an SGD (Stochastic Gradient Descent) optimization algorithm, with a learning rate set to 0.05. In order to verify the effectiveness of the technology of the present invention, it is compared with the following methods respectively: KD (Distilling the Knowledge in a Neural Network): a knowledge distillation based on neural network, DKD (Decoupled Knowledge Distillation): a knowledge distillation based on decoupling, FitNet (FitNets: Hints for Thin Deep Nets): a knowledge distillation for narrow and deep neural networks, CRD (Contrastive representation distillation): a knowledge distillation based on contrastive representation information, ReviewKD (Distilling Knowledge via Knowledge Review): a knowledge distillation based on knowledge review, CAT-KD (Class Attention Transfer Based Knowledge Distillation): a knowledge distillation based on category attention transfer.

[0115] In order to verify the effectiveness of the technology of the present invention, the indicator Accuracy is used to compare the technology of the present invention with the existing technology in Tables 1 and 2. As shown in Tables 1 and 2, the technology of the present invention has achieved good results in the distillation of teacher-student pairs with different architectures and the distillation of teacher-student pairs with the same architecture, thus proving the effectiveness of the technology of the present invention.

[0116] Table 1 Comparison results between the technology of the present invention and the prior art in heterogeneous teacher-student pairs

[0117]

[0118] Table 2 Comparison results between the technology of the present invention and the prior art in isomorphic teacher-student pairs

[0119]

Claims

1. A multidimensional knowledge distillation method based on Logit cross correction, characterized in that: The following steps are involved: S1 Input the image to be classified into the primary feature extraction module of the teacher model and the student model to obtain the teacher primary feature and primary characteristics of students ; S2 will and Input into the category result perception module to obtain the adaptive weighted Logit score and calculate the category result loss; S3 combines the adaptively weighted Logit scores and primary features and Input into the Logit cross correction module to obtain the corrected teacher characteristics and student characteristics , and calculate the correction loss; S4 Corrected teacher characteristics and student characteristics Input into the category generation perception module to form the teacher category activation map and calculate the category generation loss; S5 weights the category result loss, correction loss, and category generation loss to obtain the final loss value, minimizes the loss value through back propagation, and completes multi-dimensional knowledge distillation.

2. The multidimensional knowledge distillation method based on Logit cross correction according to claim 1 is characterized in that: The category result perception module includes two multi-scale Logit generation modules, an adaptive Logit attention module and a channel loss calculation module, which are specifically implemented as follows: S21 Multiscale Logit Generation Module consists of a multiscale partitioning layer and a fully connected layer; first, the input feature and They are respectively input into their respective multi-scale Logit generation modules, and then the local average pooling operation of adaptive size is performed through the multi-scale division layer to output local features of various sizes. After that, they are passed through a fully connected layer to output the preliminary local Logit scores. The local Logit scores of the teacher model and the student model are recorded as and ,in, Represents the index of the local feature after division; S22 will and Input into the adaptive Logit attention module, which contains an adaptive weight generation layer and an adaptive weighting layer, and performs adaptive attention weighting according to the Logit distribution; first, the attention weights are adaptively generated by the local Logit of the teacher model and the student model, and then the adaptive weights are weighted to the corresponding local area to obtain the corresponding weighted Logit score and ; S23 will and Input into the channel loss calculation module, first perform a one-to-one loss calculation on the Logit scores of the corresponding areas of the teacher model and the student model to obtain the local distillation results, and then calculate the results of all local distillation losses under all partitioning strategies.

3. The multidimensional knowledge distillation method based on Logit cross correction according to claim 2 is characterized in that: The specific implementation process of step S22 is as follows: S221 In the adaptive weight generation layer, for each local area extracted , first use the activation function , get the preliminary probability of each classification, input these preliminary probabilities into a fully connected layer to make it obtain learnable weights in back propagation, and finally use the activation function , obtain the final adaptive attention weight; S222 In the adaptive weighting layer, the adaptive attention weight generated by each local area is weighted to the Logit score of the corresponding local area and On the top, we get the corresponding weighted Logit score and .

4. The multidimensional knowledge distillation method based on Logit cross correction according to claim 3 is characterized in that: The specific implementation process of step S23 is as follows: S231 For each local Logit score and Using activation functions , get the classification probability, and then calculate the KL divergence loss to get the loss value of the classification results of the local area corresponding to the teacher model and the student model ; S232 Calculate the overall loss of all local distillation losses under all partitioning strategies to obtain the category result loss .

5. The multidimensional knowledge distillation method based on Logit cross correction according to claim 4 is characterized in that: The Logit cross correction module includes a bidirectional projection module, a loss calculation module and a feature correction module. The specific implementation process is as follows: S31 Primary Features and Respectively with the teacher model adaptive Logit score Perform bidirectional projection, using projection layer and activation function Project the primary features and adaptive Logit scores into a common space for comparison. is the projection vector of the adaptive Logit score, and Primary feature and Projection vector, using right and Make corrections; S32 In the loss calculation module, only the student model is projected and Adaptive Logit Projection Perform the Wasserstein metric calculation, denoted as the correction loss ; The S33 feature correction module includes a correction matrix generation module and a primary feature correction module. First, the correction matrix is ​​dynamically generated through the teacher Logit projection, and then the primary features are respectively and Perform feature correction.

6. The multidimensional knowledge distillation method based on Logit cross correction according to claim 5, characterized in that: The specific implementation process of step S33 is as follows: S3131 In the adaptive logit score When projecting, select its projection vector Correction Matrix The specific formula is as follows: ； in, express The transposed matrix of is the regularization parameter, represents the identity matrix; S3132 In the primary feature correction module, use the correction matrix Primary features and Perform matrix correction and introduce matrix alignment operation to obtain the corrected teacher features and student characteristics .

7. The multidimensional knowledge distillation method based on Logit cross correction according to claim 6, characterized in that: The category generation perception module includes two category activation map generation modules, a gated feature selection module and a loss calculation module. The specific implementation process is as follows: S41 corrects the features of the teacher model Input into the category activation map generation module of the teacher model to generate the teacher category activation map; the correction features of the student model Input into the category activation map generation module of the student model to obtain the category activation map and ; S42 The category activation map of the teacher model and the student model and Input into the gated feature selection module to obtain the gated category activation map of the teacher model and the student model and , S43 The activation map of the gated categories of the teacher model and the student model is obtained and Input to the loss calculation module, input gated category activation map and After the normalization layer, the input is standardized, and then the two results are calculated by the mean square error loss to get the loss value .

8. The multidimensional knowledge distillation method based on Logit cross correction according to claim 7, characterized in that: The class activation map generation module includes a convolution layer and an adaptive size average pooling layer, which is specifically implemented as follows: S411 The teacher model takes its primary features Input into the category activation map generation module of the teacher model, after the convolution layer, the channel-aligned teacher feature is obtained and recorded as ; At the same time, the primary features of the student model The input is sent to the category activation map generation module of the student model, and after the convolution layer, the student features after channel alignment are obtained and recorded as ; S412 will feature Input to the adaptive size average pooling layer of the teacher model, the feature Input to the adaptive size average pooling layer of the student model to obtain the respective category activation map and .

9. The multidimensional knowledge distillation method based on Logit cross correction according to claim 8, characterized in that: The gated feature selection module includes a gated weight calculation module and a gated feature weighting module, and the specific implementation process is as follows: S421 Activate the category map Input to the gated weight calculation module, in which First, it passes through a fully connected layer and an activation function layer , get the classification probability of each category, and record the classification probability as the learnable weight as ; S422 In the gated feature weighting module, the gated weights below the threshold are first reset to zero, and the other weights are automatically updated; then the new gated weights are The class activation map input in step S42 and Perform point multiplication and gated feature selection to obtain the category activation maps of the teacher model and the student model. and .

Images