An image classification method and device based on knowledge distillation of model output difference matrix

Abstract

The present invention discloses an image classification method and device based on knowledge distillation of model output difference matrices, comprising: (1) inputting training set images into a student model to be trained and a trained teacher model, and outputting corresponding logical classification values ​​respectively; (2) calculating the difference between each pair of logical classification values ​​and constructing two difference matrices; (3) calculating the discrimination between the two difference matrices as a distillation loss function; (4) simultaneously calculating the cross entropy of the logical classification value of the student model and the true classification label of the image as the loss function of the classification task, and adding the cross entropy to the distillation loss function to obtain the final loss function of the trained student model; (5) forwarding the gradient to update the parameters of the student model, while the parameters of the teacher model remain unchanged; (6) repeating the above steps for training, and performing image classification on the trained student model. By using the present invention, the effect of knowledge distillation on image classification tasks can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision image classification, and in particular relates to an image classification method and device based on knowledge distillation of model output difference matrix. Background Art

[0002] Knowledge distillation is an effective method for compressing high-performance, large-scale models. This method primarily utilizes a pre-trained, high-performance, and large-parameter teacher model to assist in the training of a smaller, poorly performing student network. This transfers the "knowledge" learned by the teacher model to the student model, improving its performance and ultimately achieving model compression.

[0003] The concept of knowledge distillation was first proposed by Hinton's team in the paper "Distilling the knowledge in a neural network." Pages 2 to 4 of the paper propose that before performing softmax on the output of the classification model, the classifier outputs of the teacher model and the student model should be "softened" using the hyperparameter "temperature". This can amplify the output of non-target values ​​to a certain extent, thereby transferring more "knowledge" to the student model. The loss function of this method can be described as follows:

[0004]

[0005] L kd =KLD(P s ,P t )

[0006] Among them, z is the logical output of the model, T is the hyperparameter temperature, p represents the predicted output of the model, s and t respectively represent the student model and the teacher model, and KLD is the Kullback-Leibler divergence loss function. This method only distills the final output of the model, and subsequent related research can be classified as a knowledge distillation method based on logical output. Subsequent work has proposed distillation methods based on features of other parts of the model. For example, in the paper "FitNets: Hints for Thin Deep Nets" published at the International Conference on Learning Representations in 2015, pages 2 to 5 of the paper achieved good results by aligning an intermediate feature layer of the teacher model and the student model to perform knowledge distillation; in the paper "Distilling Knowledge via Knowledge Review" published at the Conference on Computer Vision and Pattern Recognition in 2021, a novel intermediate structure was designed to distill multiple intermediate feature layers, which has a very good effect.

[0007] Generally speaking, distillation methods based on intermediate features can often achieve better distillation effects, because intermediate features generally contain more information, but the intermediate structure design is generally more complex and the computational cost is higher; while the distillation method based on the final output of the model, although relatively simple and with almost no additional overhead, is generally difficult to achieve the best results. However, some studies in recent years have also proved that the distillation method based on the final output of the model still has certain potential. For example, "Knowledge Distillation with the Reused Teacher Classifier" published in the Conference on Computer Vision and Pattern Recognition in 2022 attempts to reuse the classifier of the teacher model on the student model, thereby improving the performance of the student model; "Knowledge Distillation from AStrongerTeacher" published in the Conference on Neural Information Processing Systems in 2022 replaced the KL divergence as the loss function by the Pearson correlation coefficient and achieved good results; "Decoupled knowledge distillation" published in the top international computer vision conference Conference on Computer Vision and Pattern Recognition in 2022 proposed an effective method to decouple the method based on "Distilling the knowledge in a neural network." and then perform distillation, which greatly improved the effect of the original method, and surpassed the distillation method based on the intermediate feature layer in many indicators, and to a certain extent revealed some shortcomings of KL divergence as a distillation loss function. The downside is that although this method has strong performance and is computationally simple, it also introduces additional hyperparameters. In actual use, it is necessary to adjust the ratio between the loss function of the task itself, the target distillation loss, and the non-target distillation loss function. In addition, the "temperature" hyperparameter also needs to be adjusted. In actual use, parameter adjustment is still relatively cumbersome. Summary of the Invention

[0008] The present invention provides an image classification method and device based on knowledge distillation of the model output difference matrix. The difference matrix between different categories is constructed by the difference between each category of the final logical output of the model, and knowledge distillation is performed by aligning the difference matrices of the teacher model and the student model, so as to achieve a better distillation effect and thus improve the image classification effect.

[0009] An image classification method based on knowledge distillation of model output difference matrix includes the following steps:

[0010] (1) Input the training set images into the student model to be trained. The images are processed by the middle layer of the student model, and the corresponding logical classification values ​​are output by the classifier of the student model.

[0011] The same training set images are input into the trained teacher model. The images are processed by the teacher model's intermediate layer, and the teacher model's classifier outputs the corresponding logical classification value.

[0012] (2) According to the logical classification values ​​output by the student model and the teacher model, the pairwise differences between each logical classification value are calculated and two difference matrices are constructed;

[0013] (3) Calculate the discrimination between the difference matrix of the student model and the difference matrix of the teacher model, and use it as the distillation loss function;

[0014] (4) Calculate the cross entropy between the logical classification value of the student model and the real classification label of the image at the same time, and use it as the cross entropy loss function of the classification task. Add the distillation loss function and the cross entropy loss function according to the set ratio to serve as the final loss function for training the student model.

[0015] (5) Pass the gradient forward to update the parameters of the student model, and the parameters of the teacher model remain unchanged;

[0016] (6) Repeat the above steps (1) to (5) until the training is completed after the preset number of training times is reached, and then use the trained student model for image classification.

[0017] Furthermore, in step (1), the logical classification values ​​obtained by the student model and the teacher model are represented as Z s and Z t , both have the same shape B×C, B is the number of images in a batch during training, and C is the number of categories.

[0018] In step (2), the difference matrix constructed is of shape B×C×C, and the formula is as follows:

[0019] D b,i,j =Z b,i -Z b,j ,b∈[0,B),i,j∈[0,C)

[0020] Where, the dimension of the D matrix is ​​B×C×C, D b,i,j Represents the value at position (b,i,j) in the D matrix; the dimension of X is B×C, and Z b,i and Z b,jRepresent the values ​​of the logical classification value Z at (b,i) and (b,j), respectively, where b∈[0,B) represents the bth picture, and i,j∈[0,C) represent the i,j categories respectively.

[0021] In step (3), the difference matrix between the student model and the teacher model is calculated by L2 loss, and the resulting distillation loss function is:

[0022]

[0023] Where, L kd is the distillation loss function, Represents the value of the difference matrix D of the student model at position b, i, j, Represents the value of the difference matrix value of the teacher model at position b,i,j.

[0024] In step (4), the final loss function of training the student model is:

[0025] Loss=α·CE(Z s ,y)+β·L kd

[0026] Among them, CE is the cross entropy loss function, y is the true label of the image, and α and β are used as hyperparameters to adjust the relative size of the two loss functions.

[0027] An image classification device based on knowledge distillation of a model output difference matrix includes a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned image classification method.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. This method abandons the use of the softmax function and the hyperparameter temperature to soften the logical classification values, and directly operates on the logical classification values, which is more direct. Compared with other existing methods, it has fewer hyperparameters and is easier to debug.

[0030] 2. The present invention improves the performance of the student model by aligning the differences between the categories of the teacher model and the student model, which is equivalent to improving the performance of the student model by distilling the relationship between different categories of the teacher model. Compared with the existing method of directly aligning the final prediction value of the model, it has fewer constraints and can more effectively extract the knowledge of the model to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of an image classification method based on knowledge distillation of model output difference matrix according to the present invention;

[0032] Figure 2 Schematic diagram of a framework of an image classification method based on knowledge distillation of model output difference matrix according to the present invention;

[0033] Figure 3 Schematic diagram of difference matrix calculation in the present invention. DETAILED DESCRIPTION

[0034] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0035] like Figure 1 and Figure 2 As shown, an image classification method based on knowledge distillation of model output difference matrix includes the following steps:

[0036] S01: Input a commonly used image classification dataset (e.g., CIFAR-100) into the student model to be trained and the teacher model that has completed training. On the CIFAR-100 dataset, the present invention generally sets the batch size of training images to 64, while the number of categories contained in the dataset is 100. Therefore, the student model and the teacher model will output a 64×100 logical classification value matrix P respectively. s and P t .

[0037] S02, use the logical classification values ​​obtained in S01 to calculate the difference between different categories to construct a difference matrix. Figure 3 Above, you can see the specific schematic diagram of the difference matrix calculation, which is an antisymmetric matrix. Calculating the logical classification values ​​of the student model and the teacher model can obtain the corresponding difference matrix D s and D t , both have the same shape, 64 × 100 × 100. The size of the difference matrix is ​​only related to the batch size B and the number of categories C of the dataset, so it is easy to extend to other datasets.

[0038] S03, transfer the "knowledge" of the teacher model to the student model by aligning the difference matrix between the teacher model and the student model. The present invention uses the Euclidean distance between each element of the two matrices to represent the difference between the two, and uses this as the loss function of knowledge distillation to assist the student model training. The difference between the logical classification values ​​of different categories The reason why it works is that the difference represents the relationship between the two categories. If the logical classification value is processed by the softmax function according to the classic method and the final prediction value (that is, the possible probability of each category) is output, the difference between the logical classification values It can express the multiple difference between the probability prediction values ​​of different categories as follows:

[0039]

[0040]

[0041] Other existing methods often perform knowledge distillation by directly aligning the logical classification values ​​or final prediction values ​​of the teacher model and the student model through some methods. The present invention indirectly aligns the outputs of the teacher and student models by aligning the relationship between the logical classification values.

[0042] In step S04, the loss function for calculating the distance between the difference matrices of the teacher and student models and the classification loss function of the student model are added in a certain ratio to obtain the final loss function. The gradient is passed back to the student model to update the parameters of the student model, while the parameters of the teacher model remain unchanged.

[0043] Repeat the above steps until the preset number of training times is reached, evaluate the accuracy of the model obtained in each epoch, and select the model with the highest accuracy as the final model.

[0044] To verify the effectiveness of this invention, we conducted comparative experiments on the CIFAR-100 dataset. The CIFAR-100 dataset contains 60,000 images with 100 classes. Each class has 600 32×32 color images, 500 of which are used as a training set and 100 as a test set. This dataset is commonly used for image classification. Furthermore, we conducted experiments on multiple teacher-student model pairs to evaluate the performance of models with the same structure and models with different structures.

[0045] This example compares the best published methods on the test set. The comparison results are shown in Tables 1 and 2. Table 1 shows the distillation results on CIFAR-100 for models with similar structures, and Table 2 shows the distillation results on CIFAR-100 for models with different structures.

[0046] Table 1

[0047]

[0048] Table 2

[0049]

[0050] As can be seen, our method outperforms the current state-of-the-art distillation method, DKD, based on the model's final output, across all models with varying structures. For models with identical structures, it outperforms DKD in half of the models. Compared to distillation methods based on intermediate eigenvalues, our method also outperforms the current state-of-the-art model in a subset of models. Overall, our method is simpler than the current state-of-the-art method and performs well across a wide range of models.

[0051] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image classification method based on knowledge distillation of model output difference matrix, characterized in that: The following steps are involved: (1) Input the training set images into the student model to be trained. The images are processed by the middle layer of the student model, and the corresponding logical classification values ​​are output by the classifier of the student model. The same training set images are input into the trained teacher model. The images are processed by the teacher model's intermediate layer, and the teacher model's classifier outputs the corresponding logical classification value. (2) According to the logical classification values ​​output by the student model and the teacher model, the pairwise differences between each logical classification value are calculated and two difference matrices are constructed; (3) Calculate the discrimination between the difference matrix of the student model and the difference matrix of the teacher model, and use it as the distillation loss function; (4) Calculate the cross entropy between the logical classification value of the student model and the real classification label of the image at the same time, and use it as the cross entropy loss function of the classification task. Add the distillation loss function and the cross entropy loss function according to the set ratio to serve as the final loss function for training the student model. (5) Pass the gradient forward to update the parameters of the student model, and the parameters of the teacher model remain unchanged; (6) Repeat the above steps (1) to (5) until the training is completed after the preset number of training times is reached, and then use the trained student model for image classification.

2. The image classification method based on knowledge distillation of model output difference matrix according to claim 1, characterized in that: In step (1), the logical classification values ​​obtained by the student model and the teacher model are represented as Z s and Z t , both have the same shape B×C, B is the number of images in a batch during training, and C is the number of categories.

3. The image classification method based on knowledge distillation of model output difference matrix according to claim 2, characterized in that: In step (2), the difference matrix constructed is of shape B×C×C, and the formula is as follows: D b,i,j =Z b,i -Z b,j ,b∈[0,B),i,j∈[0,C) Where, the dimension of the D matrix is ​​B×C×C, D b,i,j Represents the value at position (b,i,j) in the D matrix; the dimension of Z is B×C, Z b,i and Z b,j Represent the values ​​of the logical classification value Z at (b,i) and (b,j), respectively, where b∈[0,B) represents the bth picture, and i,j∈[0,C) represent the i,j categories respectively.

4. The image classification method based on knowledge distillation of model output difference matrix according to claim 3, characterized in that: In step (3), the difference matrix between the student model and the teacher model is calculated by L2 loss, and the resulting distillation loss function is: Where, L kd is the distillation loss function, Represents the value of the difference matrix D of the student model at position b, i, j, Represents the value of the difference matrix value of the teacher model at position b,i,j.

5. The image classification method based on knowledge distillation of model output difference matrix according to claim 4, characterized in that: In step (4), the final loss function of training the student model is: Loss=α·CE(Z s ,y)+β·L kd Among them, CE is the cross entropy loss function, y is the true label of the image, and α and β are used as hyperparameters to adjust the relative size of the two loss functions.

6. An image classification device based on knowledge distillation of model output difference matrix, characterized in that: The method comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, the method is used to implement the image classification method according to any one of claims 1 to 5.

Images