Model lightweight method for image classification

Abstract

The invention discloses an image classification-oriented model lightweight method, which comprises the following steps of: (1) training an image classification task by using a large and complex neural network (teacher network); (2) fusing intermediate features of a student model from a deep layer to a shallow layer and from the shallow layer to the deep layer in sequence; (3) correcting or strengthening a prediction structure of the student model by using a correction mechanism, performing classified knowledge distillation on the prediction structure and a prediction result of the teacher model, and performing multi-scale knowledge distillation with an intermediate feature of the teacher model, namely calculating characteristic distillation loss; and (4) updating parameters of each layer of the network through a back propagation error, and continuously iterating to finally minimize the loss value. According to the method, efficient image classification can be realized in light-weight equipment with limited resources, and new possibility is provided for realizing balance between accuracy and efficiency by application of the technology.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification and relates to a method for lightweighting a model for image classification. Background Art

[0002] Image classification is an important task in the field of computer vision, aiming to accurately classify images into predefined categories. However, the image classification task faces many challenges, such as diverse image datasets, degraded image quality, and computational resource limitations. To address the challenge of computational resource limitations, the application of knowledge distillation technology in image classification has been proposed. Knowledge distillation aims to transfer the knowledge of a complex teacher model to a simplified student model to improve the performance of the student model.

[0003] In image classification, the application method of knowledge distillation is as follows: First, a large and complex teacher model is used for image classification training to obtain its output probability distribution, that is, soft targets. These soft targets contain confidence information for each category. Then, the soft targets are used to guide the training of a simplified student model so that it can benefit from the knowledge of the teacher model. The student model learns by minimizing the distance from the output of the teacher model, usually using the cross-entropy loss function. Through knowledge distillation, the student model can obtain the knowledge of the teacher model, including accurate class discrimination ability and robust feature representation. Compared with the teacher model, the student model has lower computational and storage requirements and is suitable for resource-constrained environments. Knowledge distillation technology provides an effective model compression method by transferring the knowledge of the teacher model to the student model to achieve efficient image classification in resource-limited environments. The application of this technology provides new possibilities for achieving a balance between accuracy and efficiency. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for lightweighting a model for image classification, which solves the problem that it is difficult for existing neural network models to achieve efficient image classification on lightweight devices with limited resources.

[0005] The technical solution adopted by the present invention is a lightweight model image classification method based on knowledge distillation. A teacher model is used to train the image classification task; the intermediate features of the student model are successively fused from deep to shallow and from shallow to deep; then knowledge distillation is performed on multiple scales with the intermediate features of the teacher model, that is, calculating the feature distillation loss; a correction mechanism is used to correct or strengthen the prediction structure of the student model and perform classification knowledge distillation with the prediction result of the teacher model, that is, calculating the classification distillation loss; the parameters of each layer of the network are updated through backpropagation error, and the loss value is minimized through continuous iteration.

[0006] The features of the present invention also lie in:

[0007] Specifically, it includes the following steps:

[0008] Step 1: Preprocess the training dataset;

[0009] Step 2: Train the teacher model;

[0010] Step 3: Use the trained teacher model to guide the training of the student model.

[0011] The specific process of Step 1 is as follows:

[0012] Step 1.1: Use the Cifar-100 dataset as the test sample. The training images and test images in this dataset are used as the training set and test set respectively. For the Cifar-100 dataset, augment the training images by randomly rotating them 90 degrees, 180 degrees, 270 degrees, and randomly cropping and flipping them.

[0013] Step 1.2: Load the classification annotation information corresponding to each image in the training dataset to provide the ground truth for calculating the loss function during the training process, denoted by Y.

[0014] The specific process of Step 2 is as follows:

[0015] Step 2.1: Send the training set image X into the teacher network to extract image features through a series of convolutional and activation functions. This process is expressed as:

[0016] F = f(X) (1)

[0017] where f(·) represents the feature extraction function, that is, all activation functions and convolutional operations, and F represents the extracted image features;

[0018] Step 2.2: Input the extracted image features into the fully connected layer FC for image classification prediction. A Softmax function follows the fully connected layer. This process is expressed as:

[0019] P = Softmax(F fc ) (2)

[0020] where P represents the predicted probability values for 100 classes, and F fc represents the feature output after passing through the fully connected layer;

[0021] Step 2.3: Calculate the loss function Loss using the cross entropy between the obtained probability prediction value P and the correct class information Y obtained in Step 1.2, and then perform backpropagation of the loss function gradient to update the neural network weight parameters. Optimize the teacher model through multiple iterative calculations and updates to complete the pre-training of the teacher model;

[0022] The loss function Loss is expressed as:

[0023] Loss = L CE (P, Y) (3)

[0024] where L CE (·) represents the cross-entropy loss.

[0025] The specific process of step 3 is as follows:

[0026] Step 3.1, knowledge review and preview distillation;

[0027] Step 3.2, knowledge correction mechanism;

[0028] Step 3.3: Calculate the error gradient direction and update the parameters through the stochastic gradient descent optimizer.

[0029] The specific process of step 3.1 is as follows:

[0030] Step 3.1.1, perform feature fusion using the attention-based feature fusion module. The attention-based fusion module F ABF (F b , F a ) is expressed as follows:

[0031] F ABF (F b , F a ) = Conv(G(F a )) * G(F a ) + Conv(F b ) * F b (4)

[0032] where G(·) represents convolution and interpolation operations, F a is the feature representation from the upper or lower layer, F b is the feature representation of this layer, and Conv(·) represents the 1x1 convolution operation;

[0033] Step 3.1.2, perform the progressive fusion feature knowledge review process in a forward manner, expressed as:

[0034]

[0035] where represents the first-order fusion feature, represents the second-order fusion feature. Through equation (5), the second-order fusion feature of each intermediate layer of the student model is obtained At this time, the second-order feature contains the intermediate features from all layers;

[0036] In step 3.1.3, after the ABF module fuses the features of the student model at different scales and dimensions, the HCL module uses pyramid pooling to process the features of the teacher and student models. This process is shown as follows:

[0037] F P =P(F) (6)

[0038] Among them, F P is the feature after pyramid pooling, and P(·) represents the pyramid pooling process. At this time, the teacher and student model features at multiple scales are obtained, and the loss function is calculated using the L2 distance, which is expressed as follows:

[0039]

[0040] Where D(·) is the L2 loss, They are the student and teacher features after pyramid pooling, L RP For final knowledge review and preview loss.

[0041] The specific steps of step 3.2 are:

[0042] In the process of processing the model output, the predicted probability value of the model is smoothed by the softmax function containing the temperature coefficient τ. The process is expressed as follows:

[0043]

[0044] Among them, p i is the class probability prediction for the i-th class, n is the total number of classes, τ is the temperature coefficient, represents the class probability prediction after smoothing;

[0045] The final loss function of original knowledge distillation is expressed as:

[0046] L KD =αL CE (X S ,Y)+(1-α)τ 2 H(X T ,X S ) (9)

[0047] Among them, L KD Represents distillation loss.

[0048] Take the student model's prediction for the i-th sample as P i ={p1,p2,…,p cls ,…,p n}, and then add the maximum predicted value in the predicted distribution to the predicted value of the correct category, which is expressed as follows:

[0049] p rc = p cls + max(P i ) (10)

[0050] where p cls is the predicted probability value of the correct class, i.e., correction. According to equation (10), P rc = {p1, p2, …, p rc , …, p n}, which is Prediction_RC. Then, let the student model learn from the corrected labels. The knowledge correction loss is expressed as follows:

[0051]

[0052] where k is the number of samples, and is the knowledge correction loss;

[0053] Use λ and μ as hyperparameters to balance different loss functions. The overall loss function is expressed as follows:

[0054] L ALL = L KD + λL RP + μL RC (12)

[0055] where λ and μ are constant factors used to balance each loss function and are used to assign weights to each loss.

[0056] The beneficial effects of the present invention are as follows:

[0057] 1. The overall network structure is composed of a large neural network model (which we call the teacher model) and a lightweight neural network model (which we call the student model). The student model is trained under the guidance of the teacher model to achieve the transfer of image classification knowledge from the teacher model to the student model.

[0058] 2. Knowledge review and preview distillation can extract and transfer more informative feature-based knowledge, thus overcoming the problem of difficult access to high-quality intermediate features. It first performs reverse feature fusion from deep to shallow layers, and then further performs forward fusion of the obtained fused features from shallow to deep layers. The finally fused features contain the knowledge of each intermediate layer and have extremely rich information.

[0059] 3. Knowledge correction mines the student's own knowledge and makes better use of it. When the correction mechanism corrects or strengthens the student's prediction distribution, the maximum predicted probability value in the prediction distribution is added to the predicted probability value of the correct class, effectively reducing the negative impact caused by the gap between the student and the teacher model. Brief Description of the Drawings

[0060] Figure 1 It is the structural diagram of the lightweight network of the image classification model in the method for classifying images of the lightweight model based on knowledge distillation according to the present invention;

[0061] Figure 2 It is the structural diagram of the attention-based feature fusion module in the method for classifying images of the lightweight model based on knowledge distillation according to the present invention;

[0062] Figure 3 It is the structural diagram of the knowledge review and preview module in the method for classifying images of the lightweight model based on knowledge distillation according to the present invention;

[0063] Figure 4 It is the structural diagram of the multi-level context loss module in the method for classifying images of the lightweight model based on knowledge distillation according to the present invention;

[0064] Figure 5 It is the schematic diagram of the knowledge correction mechanism in the method for classifying images of the lightweight model based on knowledge distillation according to the present invention. Detailed Embodiments

[0065] The present invention will be described in detail below with reference to the drawings and specific embodiments.

[0066] Embodiment 1

[0067] The method for classifying images of the lightweight model based on knowledge distillation according to the present invention uses multi-level feature progressive fusion and multi-level feature loss for feature knowledge transfer, and strengthens the image classification ability of the model through classification prediction error correction. This method consists of a large model image classification network (referred to as the teacher model) and a lightweight model image classification network (referred to as the student model) in the network model structure, and knowledge distillation is carried out from the intermediate layer and the output layer. The structural diagram is shown in Figure 1. First is the pre-training stage. The teacher model needs to complete its own pre-training before participating in the training of the student model. At this time, the teacher model has strong image classification ability and can provide relatively accurate and more acceptable 'knowledge' (specifically referring to intermediate features and classification labels in the method of the present invention) for the student model, enabling the lightweight student model to achieve an effect similar to or even better than that of the large model in classification ability. In the training stage, the pre-trained teacher shares the same input with the student model and conducts knowledge distillation from two levels, the intermediate layer and the output layer, to the student model. For knowledge distillation of the intermediate layer, in order to obtain highly information-rich intermediate features to guide the student model, the present invention proposes a method of knowledge review and preview distillation. First, reverse feature fusion from deep to shallow is performed, which is called knowledge review; then, the obtained fused features are further fused from shallow to deep, which is called knowledge preview. Finally, the features fused bidirectionally contain the knowledge from each intermediate layer, greatly improving the efficiency of feature learning of the student model. To obtain more focused features, the present invention embeds a coordinate attention module in the network. For knowledge distillation of the output layer, a response correction mechanism is proposed to correct or enhance the prediction results of the student model, make more full use of the knowledge of the student model itself, and effectively alleviate the negative impact brought by the gap between the teacher model and the student model.

[0068] The present invention uses a teacher model with stronger image classification ability to guide the training of a lightweight student model.

[0069] Specifically, it is implemented according to the following steps:

[0070] Step 1: Preprocessing of the training dataset.

[0071] Step 1 is specifically implemented according to the following steps:

[0072] Step 1.1, using the commonly used Cifar-100 dataset as the test sample. This dataset has a total of 60,000 images with a size of 64×64, including 50,000 training images and 10,000 test images, which are used as the training set and test set images respectively. For the Cifar-100 dataset, the training images are augmented by randomly rotating 90 degrees, 180 degrees, 270 degrees and randomly cropping and flipping.

[0073] Step 1.2, loading the classification annotation information corresponding to each image in the training dataset to provide the groundtruth for calculating the loss function during the training process, denoted by Y.

[0074] Step 2: Model training, that is, the teacher model learns in advance the image classification knowledge required by the student model. Model training can be divided into three processes: image feature extraction, classification prediction, and loss function gradient backpropagation.

[0075] Step 2 is specifically implemented according to the following steps:

[0076] In Step 2.1, first, image classification prediction is performed. The training set image X is fed into the teacher network, and image feature extraction is carried out through a series of convolutional and activation functions. This process can be expressed as:

[0077] F = f(X) (1)

[0078] where f(·) represents the feature extraction function, that is, all activation functions and convolutional operations, and F represents the extracted image features.

[0079] In Step 2.2, classification prediction is performed. The extracted image features are input into the fully connected layer (FC) for image classification prediction, and a Softmax function follows the fully connected layer. This process can be expressed as:

[0080] P = Softmax(F fc ) (2)

[0081] where P represents the predicted probability values for 100 classes, and F fc represents the feature output after passing through the fully connected layer.

[0082] In Step 2.3, the gradient of the loss function is backpropagated. The obtained probability prediction value P and the correct class information Y obtained in Step 1.2 are used to calculate the loss function with cross-entropy, which is expressed as:

[0083] Loss = L CE (P, Y) (3)

[0084] where L CE (·) represents the cross-entropy loss.

[0085] Then, the gradient of the loss function is backpropagated to update the neural network weight parameters. The network model is optimized through multiple iterative calculations and updates to complete the pre-training of the teacher model.

[0086] In Step 3, the training stage of the student model. After completing the pre-training, the trained teacher model is used to guide the training of the student model. The training stage of the student model includes two important processes: knowledge review and preview distillation, and knowledge correction.

[0087] Step 3 is specifically implemented according to the following steps:

[0088] Let the student model be S. Referring to Equation (1) in Step 2.1, the feature extraction process of the student model can be expressed as:

[0089] F S = S(X) (4)

[0090] where F SRepresents the characteristics of the student model, and F S ={f1, f2, …, f n}, where n represents the number of network layers. Similarly, the feature extraction process of the teacher model T can be expressed as:

[0091] F T = T(X) (5)

[0092] where F T represents the characteristics of the student model.

[0093] Step 3.1, Knowledge Review and Preview Distillation. This process can be divided into three parts: the attention-based feature fusion module, knowledge review and preview, and multi-level context loss.

[0094] Step 3.1.1, Use the attention-based feature fusion module (Attention Based Fusion, ABF) for feature fusion. See specifically Figure 2 .

[0095] First, make the features from different layers alignable through convolution and interpolation. This process can be expressed as:

[0096] F trans = G(F a ) (6)

[0097] where F trans represents the feature representation after convolution and interpolation, G(·) represents the convolution and interpolation operations, and F a is the feature representation from the upper or lower layer. The deformed feature representation F trans is consistent with the feature representation F b of this layer in terms of channels and dimensions.

[0098] Then a 1×1 convolution is used to obtain the feature attention maps corresponding to features F a and F b respectively, which can be expressed as follows:

[0099] F att = Conv(F) (7)

[0100] where F att represents the attention map, and Conv(·) represents the 1×1 convolution operation.

[0101] Finally, multiply and add the features from different levels and their corresponding attention maps to obtain the fused features. This process can be expressed as:

[0102] F fuse = Conv(F trans ) * Ftrans +Conv(F b )*F b (8)

[0103] Among them, F fuse represents the fused feature.

[0104] In summary, the attention-based fusion module F ABF (F b , F a ) can be expressed as follows:

[0105] F ABF (F b , F a ) = Conv(G(F a )) * G(F a ) + Conv(F b ) * F b (9)

[0106] Step 3.1.2, Knowledge review and preview.

[0107] During the knowledge review process, for the intermediate features of the student model, feature fusion is performed through the ABF module mentioned in Step 3.1.1. The backward fusion is expressed as follows:

[0108]

[0109] Then, the process of progressive fusion of feature knowledge in a forward-to-backward manner can be expressed as:

[0110]

[0111] Among them, represents the first-order fused feature. Through Equation (11), we can obtain the first-order fused feature of each intermediate layer of the student model At this time, the first-order feature contains features from deeper layers than the current i-th layer.

[0112] Similar to Equation (10), the forward fusion can be expressed as:

[0113] f i ·f i+1 = F ABF (f i+1 , f i ) (12)

[0114] f i represents the feature of the i-th layer, and f i+1 represents the feature of the (i + 1)-th layer. Then, the process of progressive fusion of feature knowledge in a forward-to-backward manner can be expressed as:

[0115]

[0116] Among them, represents the second-order fusion feature. Through Equation (13), the second-order fusion features of each intermediate layer of the student model can be obtained The second-order features at this time contain the intermediate features from all layers. See specifically Figure 3 .

[0117] Step 3.1.3, Hierarchical Context Loss (HCL).

[0118] The ABF module fuses the features of the student model at different scales and dimensions. However, directly performing feature distillation will result in insufficient extraction of feature information. This problem is effectively solved by the HCL module. See specifically Figure 4 . This module uses pyramid pooling to process the features of the teacher and student models. This process can be expressed as follows:

[0119] F P = P(F) (14)

[0120] Among them, F represents the feature, P(·) represents the pyramid pooling process, and F P is the feature after pyramid pooling. At this time, the features of the teacher and student models at multiple scales can be obtained, and the L2 distance is used to calculate the loss function, which can be expressed as follows:

[0121]

[0122] Among them, D(·) is the L2 loss. are the student and teacher features after pyramid pooling respectively, and L RP is the final knowledge review and preview loss.

[0123] Knowledge review and preview distillation draws on the stacking idea of the residual network to ensure that the features of each layer become very rich after multiple rounds of fusion, greatly improving the effect of feature knowledge distillation.

[0124] Step 3.2, Knowledge Correction Mechanism. Response Correction is proposed to correct the output of the student model. When the prediction is incorrect, the student is corrected to let the student learn from their own mistakes; while when the prediction is correct, it is hoped that the student can be more confident. See specifically Figure 5 .

[0125] During the process of processing the model output, the predicted probability values of the model are smoothed through the softmax function containing the temperature coefficient τ. This process can be expressed as follows:

[0126]

[0127] Among them, p i is the class probability prediction for the i-th class, n is the total number of classes, τ is the temperature coefficient, Denotes the class probability prediction after smoothing. The loss of the original knowledge distillation consists of two parts: one is the cross entropy loss between the classification prediction of the student model and the hard label Y, which is the same as Equation (3) in step 2.3.

[0128] The other part is to calculate the KL divergence of the soft classification probability between the teacher and student models using the same temperature coefficient τ, which can be expressed as follows:

[0129]

[0130] in, That is the soft label, That is prediction, H(·) represents the KL function, L KL represents the KL divergence loss.

[0131] In summary, the final loss function L of the original knowledge distillation KD It can be expressed as:

[0132]

[0133] Where α is a hyperparameter that balances the weight of the loss function.

[0134] Take the student model's prediction for the i-th sample as P i ={p1,p2,…,p cls ,…,p n The maximum predicted value in the prediction distribution is then added to the predicted value for the correct class. This process can be expressed as follows:

[0135] p rc =p cls +max(P i ) (19)

[0136] where p cls is the predicted probability value of the correct category, that is, correction. According to equation (19), we can get P rc ={p1,p2,…,p rc ,…,p n}, which is Prediction_RC, see Figure 5 Then let the student model learn from the corrected labels, and the knowledge correction loss can be expressed as follows:

[0137]

[0138] where k is the number of samples, and L RC is the knowledge correction loss.

[0139] In this method, we construct a combined loss, including knowledge distillation loss, knowledge review and preview distillation loss, and knowledge correction loss. We use λ and μ as hyperparameters to balance different loss functions, and the overall loss function can be expressed as follows:

[0140] L ALL = L KD + λL RP + μL RC (20)

[0141] where λ and μ are constant factors used to balance each loss function and assign weights to each loss.

[0142] Step 3.3, calculate the error gradient direction. Parameter updates are performed through the Stochastic Gradient Descent (SGD) optimizer. However, during the training of the student model, the teacher network no longer updates the gradient parameters.

[0143] During the training process, standard data augmentation schemes are used, including random cropping, rotation, and flipping. All models are trained for 240 epochs, with a sample batch size of 128 for each cycle of training and a momentum of 0.9.

[0144] Example 2

[0145] To verify the effectiveness of the method of the present invention, various types of deep networks are used as the backbone network, including WideResNet, ResNet, ShuffleNetV1, and ShuffleNetV2, and are compared with state-of-the-art methods, including Basic Knowledge Distillation (Vanilla KD), Attention Transfer, FT, CRD, ReviewKD, Spot-adaptive Distillation, ICKD-C, and Decouple KD. The model lightweighting results on the homogeneous network are shown in Table 1:

[0146] Table 1: Comparison of classification accuracies of image classification model lightweighting methods on homogeneous networks

[0147] Teacher model Resnet56 Resnet110 Resnet32x4 wrn_40_2 wrn_40_2 Student model Resnet20 Resnet32 Resnet8x4 wrn_16_2 wrn_40_1 Vanilla KD 70.84 73.08 73.25 74.77 73.54 AT 70.96 73.11 73.36 75.23 72.77 FT 70.44 73.37 73.64 75.10 73.04 CRD 71.50 73.57 75.44 75.51 74.14 Review KD 71.89 73.89 75.53 76.01 75.09 CRD+SAKD 71.53 73.42 75.60 75.85 74.67 ICKD-C 71.71 73.72 74.80 75.58 74.32 Decouple KD 71.94 74.11 76.32 76.24 74.81 The method of the present invention 72.54 74.21 76.91 76.24 75.19

[0148] After comparison, the method of the present invention is superior to several other mainstream comparison methods in terms of classification accuracy.

[0149] Example 3

[0150] To further prove the effectiveness of the method of the present invention, we tested our method on a more challenging heterogeneous network, and the results are shown in Table 2 as follows:

[0151] Table 2: Comparison of classification accuracies of image classification model lightweight methods on heterogeneous networks

[0152] Teacher model Resnet32x4 wrn_40_2 wrn_40_2 Student model ShuffleNetV1 ShuffleNetV2 ShuffleNetV1 Vanilla KD 74.11 74.36 74.69 AT 73.62 73.57 73.80 FT 72.58 72.73 72.21 CRD 75.14 75.60 75.67 Review KD 77.34 77.50 77.04 CRD+SAKD 75.33 75.82 75.92 ICKD-C 74.69 74.56 74.63 Decouple KD 76.39 76.94 76.49 The method of the present invention 77.15 77.01 77.43

[0153] After comparison, the method of the present invention still has strong competitiveness in terms of classification accuracy compared with several other mainstream comparison methods.

Claims

1. A model lightweight method for image classification, characterized in that: Train the image classification task using the teacher model; fuse the intermediate features of the student model successively from deep to shallow and from shallow to deep; then perform knowledge distillation on multiple scales with the intermediate features of the teacher model, that is, calculate the feature distillation loss; use the correction mechanism to correct or strengthen the prediction structure of the student model and perform classification knowledge distillation with the prediction results of the teacher model, that is, calculate the classification distillation loss; update the parameters of each layer of the network through backpropagation error, and continuously iterate to finally minimize the loss value.

2. The lightweight model image classification method based on knowledge distillation according to claim 1, wherein: Specifically, it includes the following steps: Step 1, preprocess the training data set; Step 2, train the teacher model; Step 3, use the trained teacher model to guide the training of the student model.

3. The lightweight model image classification method based on knowledge distillation according to claim 2, characterized in that: The specific process of the said Step 1 is: Step 1.1, use the Cifar-100 data set as the test sample, and the training images and test images in this data set are used as the training set and test set respectively. For the Cifar-100 data set, augment the training images by randomly rotating 90 degrees, 180 degrees, 270 degrees and random cropping and flipping; Step 1.2, load the classification annotation information corresponding to each image in the training data set to provide the ground truth for calculating the loss function during the training process, denoted by Y.

4. The lightweight model image classification method based on knowledge distillation according to claim 2, wherein: The specific process of the said Step 2 is: Step 2.1, send the training set image X into the teacher network, and perform image feature extraction through a series of convolutional and activation functions. This process is expressed as: F = f(X) (1) where f(·) represents the feature extraction function, that is, all activation functions and convolutional operations, and F represents the extracted image features; Step 2.2, input the extracted image features into the fully connected layer FC for image classification prediction. A Softmax function follows the fully connected layer. This process is expressed as: P = Softmax(F fc ) (2) Among them, P represents the predicted probability value for 100 categories, and F fc represents the feature output after passing through the fully connected layer; Step 2.3, calculate the loss function Loss using the cross entropy between the obtained probability prediction value P and the correct class information Y obtained in Step 1.2, and then perform the backpropagation of the loss function gradient to update the neural network weight parameters. The teacher model is optimized through multiple iterations of calculation and update to complete the pre-training of the teacher model; The loss function Loss is expressed as: Loss=L CE (P,Y) (3) Among them, L CE (·) represents the cross-entropy loss.

5. The lightweight model image classification method based on knowledge distillation according to claim 4, characterized in that: The specific process of the said Step 3 is: Step 3.1, knowledge review and preview distillation; Step 3.2, knowledge correction mechanism; Step 3.3, calculate the error gradient direction and update the parameters through the stochastic gradient descent optimizer.

6. The method for lightweight model image classification based on knowledge distillation according to claim 5, characterized in that: The specific process of the said Step 3.1 is: Step 3.1.1, perform feature fusion using an attention-based feature fusion module, and the attention-based fusion module F ABF (F b , F a ) is expressed as follows: F ABF (F b ,F a ) = Conv(G(F a )) * G(F a ) + Conv(F b ) * F b (4) Among them, G(·) represents convolution and interpolation operations, and F a is the feature representation from the upper or lower layer, and F b is the feature representation of this layer; Step 3.1.2, perform a progressive fusion feature review process in a forward manner, expressed as: Among them, represents the second-order fusion feature. Through Equation (5), the second-order fusion features of each intermediate layer of the student model are obtained The second-order features at this time contain the intermediate features from all layers; Step 3.1.3, the ABF module fuses the features of the student model at different scales and dimensions. This module uses pyramid pooling to process the features of the teacher and student models. This process is expressed as follows: F P = P(F) (6) Among them, F P is the feature after pyramid pooling processing. At this time, the teacher and student model features at multiple scales are obtained, and the L2 distance is used to calculate the loss function, which is expressed as follows: where D(·) is the L2 loss, are the student and teacher features after pyramid pooling respectively, and L RP is the final knowledge review and preview loss.

7. The lightweight model image classification method based on knowledge distillation according to claim 6, characterized in that: The specific steps of the said Step 3.2 are: During the processing of the model output, smooth the prediction probability value of the model through the softmax function containing the temperature coefficient τ. This process is expressed as follows: where p i is the class probability prediction for the i-th sample, n is the total number of classes, and τ is the temperature coefficient, represents the class probability prediction after smoothing; The final loss function of the original knowledge distillation is expressed as: Take the prediction of the student model for the i-th sample as P i = {p1, p2, …, p cls , …, p n}, and then add the maximum predicted value in the predicted distribution to the predicted value of the correct class. This process is expressed as follows: p rc = p cls + max(P i ) (10) where p cls is the predicted probability value of the correct category, i.e., correction. According to Equation (10), P rc ={p1, p2, …, p rc , …, p n}, i.e., Prediction_RC. Then, let the student model learn from the corrected labels. The knowledge correction loss is expressed as follows: where k is the number of samples, and L RC is the knowledge correction loss; Use λ and μ as hyperparameters to balance different loss functions, and the overall loss function is expressed as follows: L ALL = L KD + λL RP + μL RC (12) Among them, λ and μ are constant factors used to balance each loss function and are used to assign weights to each loss.

Images