Diversified knowledge distillation image recognition method with self-distillation module

Abstract

The invention provides a diversity knowledge distillation image recognition method with a self-distillation module, and belongs to the technical field of computer vision and deep learning. Constructing a student model set comprising a plurality of shared network architectures, and enhancing feature diversity through an attention migration strategy; a self-distillation module is configured for each student, the module is composed of cascaded transposed convolution blocks, deep features are mapped to a shallow layer, and deep semantic knowledge is transmitted to the shallow layer through attention distribution loss and feature map loss; then, an online cooperative training mode is adopted, the total loss is fused with classification loss, diversity loss and self-distillation loss, and all student models are synchronously optimized; and finally, only selecting a single student model with the optimal performance for image recognition reasoning. According to the method, on CIFAR-100 and ImageNet data sets, the recognition accuracy of various architectures such as ResNet and WRN is remarkably improved, only a single model needs to be deployed, resource redundancy of multi-model integration is avoided, and the method is suitable for a high-precision image recognition task of a resource limited scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, and more specifically, relates to an image recognition method with a self-distillation module for diverse knowledge distillation. Background Technology

[0002] The application of deep neural networks in image recognition relies on the high discriminative power of complex models, but the performance of lightweight models often falls short of the requirements. Knowledge distillation techniques improve the performance of lightweight models by transferring knowledge, with online distillation attracting widespread attention due to its elimination of the need for pre-trained teacher models. However, existing online distillation methods have two major limitations: Feature homogenization: Multi-student models tend to converge in collaborative learning (feature maps have highly overlapping regions of interest), resulting in insufficient knowledge complementarity and limited integration performance; Insufficient inter-layer knowledge transfer: The shallow features of the model lack supervision from deep semantic information, resulting in weak feature discriminativeness and affecting the generalization ability of a single model.

[0003] Furthermore, traditional methods either require multiple model deployments (increasing resource overhead) or offer limited performance improvements for individual models, making it difficult to balance accuracy and efficiency. Therefore, there is an urgent need for an online distillation method that can enhance feature diversity, strengthen inter-layer knowledge flow, and offer lightweight deployment. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an image recognition method with a self-distillation module for diverse knowledge distillation, thereby resolving the problems described above.

[0005] An image recognition method with a self-distillation module for diverse knowledge distillation includes the following steps: 1) Construct a network set containing n student models. All models share the same convolutional neural network architecture and each contains M key feature layers (M≤L, where L is the total number of convolutional layers). 2) For each student model, optimize the diversity of its feature maps using a diversity enhancement strategy: 2a) Calculate the attention distribution of the feature map at the m-th layer (1≤m≤M) of each student model, where the attention distribution is the sum of squares of the channel dimensions of the feature map; 2b) Based on the attention distribution, a diverse attention map is generated through an offset strategy, so that the feature maps of different student models focus on different regions of the input image; 2c) Using the aforementioned diverse attention maps as the target, optimize the feature diversity among student models through L2 norm loss; 3) Configure a self-distillation module for each student model. The self-distillation module includes M-1 cascaded blocks. Each cascaded block contains a transposed convolutional layer, a batch normalization layer, and a ReLU activation layer, which are used to map the feature map of the (m+1)th layer to the feature map of the mth layer. 4) The self-distillation module takes the M-th layer feature map of the student model as input and the feature maps of layers 1 to M-1 as supervision targets, and optimizes the module parameters through attention distribution loss and feature map loss; the attention distribution loss is the normalized L2 norm loss of the mapped features and the target features, and the feature map loss is the normalized L2 norm loss of the mapped features and the target features. 5) After training is complete, select the single student model with the best performance for image recognition inference.

[0006] Preferably, in step 1), the convolutional neural network architecture is selected from ResNet, WRN, and DenseNet, and the key feature layer is the last layer of each residual module.

[0007] Preferably, in step 2b), the calculation formula for the offset strategy is: ; in, Let P be the attention distribution of the m-th layer of the i-th student model. The L2 norm, t is The Middle Small values, , Let be the height and width of the feature map at layer m.

[0008] Preferably, in step 2c), the L2 norm loss is calculated using a unidirectional chain: the loss of the i-th student model (i>1) is calculated only with the diversified attention map of the (i-1)-th student model to reduce computational complexity.

[0009] Preferably, the stride and kernel size of the transposed convolutional layer are set according to the size variation of the feature map, so that the height and width of the mapped feature map are consistent with those of the target feature map.

[0010] Preferably, the weight ratio of the attention distribution loss to the feature map loss is adjusted by the hyperparameter α, and the value of α ranges from 0.5 to 2.

[0011] Preferably, the total training loss of the student model includes basic classification loss, diversity enhancement loss, and self-distillation loss, wherein the basic classification loss is the cross-entropy loss between the true label and the predicted probability, and the diversity enhancement loss and self-distillation loss are adjusted by weight parameters.

[0012] Preferably, the weighting parameters include diversity loss weight λ_div (1e-5~1e-3) and self-distillation loss weight λ_self (1e2~1e4).

[0013] Preferably, the self-distillation module is trained using the ADAM optimizer with an initial learning rate of 0.001 and a batch size consistent with that of the student model.

[0014] Preferably, the inference process of the image recognition retains only a single student model, without storing or calling other models, and the FLOPs of the model increase by no more than 5% compared with the baseline model.

[0015] Compared with the prior art, the present invention has the following beneficial effects: Enhancing feature diversity: By using an attention shift strategy, the student model focuses on complementary regions of the image, solving the problem of feature homogenization and improving the richness of integrated knowledge. In the experiment, ResNet-32 achieved a 3.16% performance improvement on CIFAR-100 due to the enhancement of diversity. Enhanced inter-layer knowledge transfer: The self-distillation module maps deep semantic information to shallow layers, making up for the lack of shallow feature supervision. The performance of the single model is improved by 1.87% compared with diversity enhancement alone (WRN-16-2 on CIFAR-100). Lightweight deployment: Ultimately, only a single optimal student model needs to be deployed, with FLOPs consistent with the baseline model (e.g., 412.5M for ResNet-32), avoiding resource redundancy in multi-model integration and reducing computational costs by 66% (compared to multi-model deployment methods such as DML). Excellent generalization ability: It is effective on both small datasets (CIFAR-100) and large datasets (ImageNet), and is compatible with various architectures such as ResNet and DenseNet. The Top1 accuracy is improved by 0.96%-1.2% compared with existing methods (ResNet-34 on ImageNet). Simplified process: No pre-trained teacher model is required. The diversity and self-distillation modules are optimized synchronously through online collaborative learning, resulting in higher training efficiency than the traditional two-stage distillation method. Attached Figure Description

[0016] Figure 1 This invention visualizes the feature maps of two student models using ResNet-32 on CIFAR-100. Figure 2 This is a flowchart of the self-distillation module in this invention; Figure 3 This is a flowchart of the FFSD framework in this invention; Figure 4 This is a comparison diagram of the self-distillation feature map in this invention. Detailed Implementation

[0017] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0018] Please see Figures 1-4 This invention provides an image recognition method with a self-distillation module for diverse knowledge distillation, comprising: Model architecture design: The network ensemble consists of n student models (S1, S2, ..., S...). n All models share the same convolutional neural network architecture (such as ResNet, WRN, DenseNet), which contains L convolutional layers, from which M key feature layers are selected (M≤L, such as the last layer of each residual module). Taking ResNet-32 as an example, 3 key feature layers are selected (layers 10, 20, and 32).

[0019] Self-distillation module: Each student model is configured with an independent self-distillation module, containing M-1 cascaded blocks. Each cascaded block consists of a transposed convolutional layer (3×3 convolutional kernel, stride set according to feature size), a batch normalization layer, and a ReLU activation layer. The m-th cascaded block (1≤m≤M-1) receives the feature map of the (m+1)-th layer and outputs a mapped feature that matches the size of the m-th layer, realizing the transfer of deep knowledge to shallow layers.

[0020] Data preparation and preprocessing: Datasets: CIFAR-100 (32×32 pixels, 100 classes) or ImageNet (224×224 pixels, 1000 classes) are used, and the training and test sets are divided in the same way as usual.

[0021] Preprocessing: Standardization: CIFAR-100 uses mean [0.5071, 0.4867, 0.4408] and standard deviation [0.2675, 0.2565, 0.2761]; ImageNet uses mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225]. Data augmentation: During the training phase, random horizontal flipping (probability 0.5), random cropping (CIFAR-100 padding 4 pixels and then cropping to 32×32, ImageNet cropping to 224×224), and random color dithering (brightness ±0.1, contrast ±0.1) are used.

[0022] Training process: Diversity enhancement strategies: Attention distribution calculation: For student model S i The m-th layer feature map (size ), calculate the attention distribution along the channel dimension: ; in, for The attention map has a higher value, indicating that the region is more important for recognition.

[0023] Diverse attention generation: Generating attention maps that complement those of other students through an offset strategy. The formula is: ; Where, P= (L2 norm), t is The Middle Small values ​​(control the proportion of the offset area).

[0024] Diversity loss: A unidirectional chained computation is used (to reduce complexity), and the i-th student is only compared with the (i-1)-th student. ; in =1e-5 (weight parameter).

[0025] Self-distillation module training: Mapping Feature Generation: The self-distillation module takes the feature map of layer M as input, maps it sequentially through M-1 cascaded blocks, and generates mapping features that match layers 1 to M-1. .

[0026] Self-distillation loss: includes attention distribution loss and feature map loss. ; in, For mapping features Attention distribution, weight parameters =1e3, the weight ratio of attention loss to feature map loss is α=1.

[0027] Total loss of student model: The total loss function combines the basic classification loss, diversity loss, and self-distillation loss: ; in, Let the true label y and the predicted probability P be... i The cross-entropy loss.

[0028] Optimizer: SGD is used, with an initial learning rate of 0.1 (CIFAR-100) / 0.8 (ImageNet), momentum of 0.9, and weight decay of 1e-4; training cycles are 300 epochs (CIFAR-100) / 90 epochs (ImageNet), and the learning rate decays to 1 / 10 in 1 / 2 and 3 / 4 of the cycles.

[0029] Deployment of Reasoning: After training, the performance of n student models is evaluated on a validation set, and the single model with the highest accuracy is selected for inference. The input image to be recognized is forward-propagated through this model, and the predicted probability is taken as the category corresponding to the highest probability.

[0030] Experimental verification: Experimental setup: Hardware: Intel Xeon Gold 6226 CPU, NVIDIA RTX 3090 GPU, PyTorch v1.10 implementation; Parameters: Number of students n=3, key feature layers M=3, batch size 128 (CIFAR-100) / 256 (ImageNet).

[0031] Experimental results: Table 1: Performance Comparison on CIFAR-100 (Accuracy, %, Mean ± Standard Deviation of Three Runs):

[0032] Table 2: Comparison with existing methods (ResNet-32 on CIFAR-100):

[0033] Table 3: Performance on ImageNet (Top-1 accuracy %)

[0034] In summary, the image recognition method with a self-distillation module proposed in this invention effectively solves the problems of feature homogenization and insufficient inter-layer knowledge transfer in existing knowledge distillation methods through the synergistic design of a "diversity enhancement strategy" and a "self-distillation module." This method constructs a multi-student model ensemble, guiding different students to focus on complementary regions of the image through an attention shift strategy, thereby enhancing feature diversity. Simultaneously, a self-distillation module is configured for each student, mapping deep semantic knowledge to shallower layers and improving the feature discriminative power of a single model. During online collaborative training, no pre-trained teacher model is required; the model is optimized by fusing diversity loss and self-distillation loss, ultimately deploying only the single student model with the best performance.

[0035] Experimental results demonstrate that this method achieves significant performance improvements across various architectures, including ResNet, WRN, and DenseNet, on the CIFAR-100 and ImageNet datasets (e.g., a 5.64% improvement in accuracy for ResNet-32 on CIFAR-100 and a 2.1% improvement in Top-1 accuracy for ResNet-34 on ImageNet). Furthermore, deployment requires only a single model, with FLOPs consistent with the baseline, avoiding resource redundancy associated with multi-model integration. Its core advantages lie in: enhancing the complementarity of knowledge across multiple learners through diversity enhancement and strengthening inter-layer knowledge flow within a single model through self-distillation. This significantly improves image recognition accuracy without increasing deployment costs, making it suitable for scenarios with high performance requirements and limited resources (such as smart terminals and real-time monitoring), and possessing significant practical value.

[0036] The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. An image recognition method for diverse knowledge distillation with a self-distillation module, characterized in that, Includes the following steps: 1) Construct a network set containing n student models. All models share the same convolutional neural network architecture and each contains M key feature layers (M≤L, where L is the total number of convolutional layers). 2) For each student model, optimize the diversity of its feature maps using a diversity enhancement strategy: 2a) Calculate the attention distribution of the feature map at the m-th layer (1≤m≤M) of each student model, where the attention distribution is the sum of squares of the channel dimensions of the feature map; 2b) Based on the attention distribution, a diverse attention map is generated through an offset strategy, so that the feature maps of different student models focus on different regions of the input image; 2c) Using the aforementioned diverse attention maps as the target, optimize the feature diversity among student models through L2 norm loss; 3) Configure a self-distillation module for each student model. The self-distillation module includes M-1 cascaded blocks. Each cascaded block contains a transposed convolutional layer, a batch normalization layer, and a ReLU activation layer, which are used to map the feature map of the (m+1)th layer to the feature map of the mth layer. 4) The self-distillation module takes the M-th layer feature map of the student model as input and the feature maps of layers 1 to M-1 as supervision targets, and optimizes the module parameters through attention distribution loss and feature map loss; the attention distribution loss is the normalized L2 norm loss of the mapped features and the target features, and the feature map loss is the normalized L2 norm loss of the mapped features and the target features. 5) After training is complete, select the single student model with the best performance for image recognition inference.

2. The image recognition method with a self-distillation module for diverse knowledge distillation as described in claim 1, characterized in that, In step 1), the convolutional neural network architecture is selected from ResNet, WRN, and DenseNet, and the key feature layer is the last layer of each residual module.

3. The image recognition method with a self-distillation module for diverse knowledge distillation as described in claim 2, characterized in that, In step 2b), the calculation formula for the offset strategy is: ； in, Let P be the attention distribution of the m-th layer of the i-th student model. The L2 norm, t is The Middle Small values, , Let be the height and width of the feature map at layer m.

4. The image recognition method with a self-distillation module for diverse knowledge distillation as described in claim 3, characterized in that, In step 2c), the L2 norm loss is calculated using a one-way chain: the loss of the i-th student model (i>1) is calculated only with the diversified attention map of the (i-1)-th student model to reduce computational complexity.

5. The image recognition method with a self-distillation module for diverse knowledge distillation as described in claim 4, characterized in that, The stride and kernel size of the transposed convolutional layer are set according to the size variation of the feature map, so that the height and width of the mapped feature map are consistent with those of the target feature map.

6. The image recognition method with a self-distillation module for diverse knowledge distillation as described in claim 5, characterized in that, The weight ratio of the attention distribution loss to the feature map loss is adjusted by the hyperparameter α, which ranges from 0.5 to 2.

7. The image recognition method with a self-distillation module for diverse knowledge distillation as described in claim 6, characterized in that, The total training loss of the student model includes basic classification loss, diversity enhancement loss, and self-distillation loss, where the basic classification loss is the cross-entropy loss between the true label and the predicted probability, and the diversity enhancement loss and self-distillation loss are adjusted by weight parameters.

8. The image recognition method with a self-distillation module for diverse knowledge distillation as described in claim 7, characterized in that, The weighting parameters include the diversity loss weight λ_div (1e-5~1e-3) and the self-distillation loss weight λ_self (1e2~1e4).

9. The image recognition method with a self-distillation module for diverse knowledge distillation as described in claim 8, characterized in that, The self-distillation module is trained using the ADAM optimizer with an initial learning rate of 0.001 and a batch size consistent with the student model.

10. The image recognition method with a self-distillation module for diverse knowledge distillation as described in claim 9, characterized in that, The inference process for image recognition retains only a single student model, without storing or calling other models, and the model's FLOPs increase by no more than 5% compared to the baseline model.

Images