Two-dimensional normalized knowledge distillation method, system and terminal based on class relationship

Abstract

The present invention relates to the field of artificial intelligence technology, and discloses a two-dimensional normalized knowledge distillation method, system and terminal based on class relationship. The method inputs samples in a training set batch into a teacher model and a student model respectively, obtains the output of the last hidden layer of each teacher model and the student model, namely, teacher logits and student logits; performs inter-class dimension normalization correction and intra-class dimension normalization correction on each logit in the teacher logits and the student logits respectively, and calculates the inter-class normalization loss and the intra-class normalization loss according to the correction results; trains the student logits and the original true labels of the samples, and calculates the classification loss; weights to obtain the total training loss, performs gradient descent and back propagation based on the total training loss, thereby optimizing the student model parameters, and then uses the next batch of samples in the training set to continuously optimize the learning model until the student model converges. The present invention can improve the accuracy and training efficiency of logit knowledge distillation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a two-dimensional normalized knowledge distillation method based on class relations, as well as a system and a computer terminal applying the method. Background Art

[0002] Deep convolutional neural networks are increasingly being used in computer vision applications such as image classification, object detection, and semantic segmentation. While convolutional neural networks help improve model performance, the complexity of these neural network models is also increasing, requiring more computing resources and storage costs. This makes it difficult to directly deploy convolutional neural network-based methods on mobile devices and embedded platforms. Knowledge distillation, as one of the important methods for model compression, has been widely used in various tasks.

[0003] Knowledge distillation uses a high-performance but difficult-to-deploy teacher model to improve and enhance the performance of a lightweight student model, so that the student's performance can approach or even exceed the teacher. The current mainstream distillation methods include logit distillation and feature distillation. The performance of class-based distillation that directly learns the teacher's output is often not ideal. This is because in the traditional logit distillation method, the student is required to accurately match the teacher's prediction, but due to the difference in capacity and structure between the teacher and student models, the student cannot produce the same distribution as the teacher. In addition, traditional methods are limited to knowledge transfer between instances, that is, the transfer of inter-class relationships, but ignore the contextual information between different samples, which affects the generalization ability of the model, thereby limiting the accuracy and training efficiency of logit knowledge distillation. Summary of the invention

[0004] The technical problem solved by the present invention is how to improve the accuracy and training efficiency of logit knowledge distillation.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The present invention discloses a two-dimensional normalized knowledge distillation method based on class relationship, comprising:

[0007] Input the samples, i.e., images, in the training set batch into the teacher model and the student model respectively, and obtain the outputs of the last hidden layer of the teacher model and the student model, i.e., teacher logits and student logits; wherein, teacher logits and student logits are two-dimensional matrices of the same dimension, each element, i.e., logit, in the two-dimensional matrix represents the confidence score of the model for each category, and the number of rows and columns of the two-dimensional matrix corresponds to the number of samples and the number of categories respectively;

[0008] Perform inter-class dimension normalization correction and intra-class dimension normalization correction on each logit in the teacher logits and student logits, and calculate the inter-class normalization loss and intra-class normalization loss based on the correction results;

[0009] Train the student logits with the original true labels of the samples and calculate the classification loss;

[0010] The inter-class normalization loss, the intra-class normalization loss and the classification loss are weighted to obtain the total training loss, and gradient descent and back propagation are performed based on the total training loss to optimize the student model parameters. Subsequently, the learning model is continuously optimized using the next batch of samples in the training set until the student model converges.

[0011] The present invention also discloses a two-dimensional normalized knowledge distillation system based on class relations, comprising: a sample input module, a loss calculation module and a student model output module.

[0012] The sample input module is used to input the samples, i.e., images, in the training set batches into the teacher model and the student model, respectively, to obtain the outputs of the last hidden layers of the teacher model and the student model, i.e., teacher logits and student logits; wherein, teacher logits and student logits are two-dimensional matrices of the same dimension, each element in the two-dimensional matrix, i.e., logit, represents the confidence score of the model for each category, and the number of rows and columns of the two-dimensional matrix correspond to the number of samples and the number of categories, respectively;

[0013] The loss calculation module is used to perform inter-class dimension normalization correction and intra-class dimension normalization correction on each logit in the teacher logits and the student logits, and calculate the inter-class normalization loss and the intra-class normalization loss according to the correction results; the loss calculation module is also used to train the student logits with the original true labels of the samples to calculate the classification loss; the loss calculation module is also used to weight the inter-class normalization loss, the intra-class normalization loss and the classification loss to obtain the total training loss;

[0014] The student model output module is used to perform gradient descent and back propagation based on the total training loss to optimize the student model parameters, and then continuously optimize the learning model using the next batch of samples in the training set until the student model converges.

[0015] The present invention also discloses a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the two-dimensional normalized knowledge distillation method based on class relations as described above are implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1. The two-dimensional normalized knowledge distillation method based on class relations disclosed in the present invention can reduce the performance problems caused by the capacity gap between the student model and the teacher model and improve the distillation efficiency. In addition, this method also overcomes the limitations of traditional knowledge distillation by combining inter-class information with intra-class correlation and reconstructing the logits of the student model and the teacher model through normalization correction, so that the student model can pay more attention to the relative relationship between logits, thereby better imitating the teacher model. This method is simple to implement but has significant effects. A large number of experiments have proved its effectiveness in image classification and target detection tasks, and it has significant advantages in training efficiency.

[0018] 2. The knowledge distillation system and computer terminal disclosed in the present invention can produce the same beneficial effects as the above method by applying the above method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of a two-dimensional normalized knowledge distillation framework based on class relations in Example 1 of the present invention.

[0020] Figure 2 This is a flow chart of the two-dimensional normalized knowledge distillation method based on class relations in Example 1 of the present invention.

[0021] Figure 3 This is a comparison chart of the training time and Top-1 accuracy of various methods in Example 1 of the present invention on the CIFAR-100 dataset.

[0022] Figure 4 This is a t-SNE feature visualization result diagram of the student model of the traditional knowledge distillation method in Example 1 of the present invention.

[0023] Figure 5 This is a t-SNE feature visualization result diagram of the student model of the method of the present invention in Example 1 of the present invention.

[0024] Figure 6 It is a correlation matrix distance diagram between the teacher model and the student model of the traditional knowledge distillation method in Example 1 of the present invention.

[0025] Figure 7 This is a correlation matrix distance diagram between the teacher model and the student model of the method of the present invention in Example 1 of the present invention.

[0026] Figure 8 This is a framework diagram of the two-dimensional normalized knowledge distillation system based on class relations in Example 2 of the present invention.

[0027] Fig. 9This is a schematic diagram of the structure of a computer terminal in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Example 1

[0030] This embodiment provides a two-dimensional normalized knowledge distillation method based on class relations, aiming to narrow the capacity gap between the student model and the teacher model and improve the distillation performance.

[0031] like Figure 1 As shown, the distillation framework of this embodiment is described by taking 3 samples and 5 categories as an example. Each row of the matrix in the figure represents a sample, and each column represents a category. The sample is a picture of a computer vision task to be performed, and the types of computer vision tasks may include image classification, object detection, and semantic segmentation. Figure 1 The samples in are pictures corresponding to "cat", "dog" and "bird" respectively.

[0032] See also Figure 2 ,The knowledge distillation method includes the following steps, namely S1~S5.

[0033] S1. Input the samples in the training set batch, i.e., images, into the teacher model and the student model respectively, and obtain the output of the last hidden layer of the teacher model and the student model, i.e., teacher logits and student logits. Among them, teacher logits and student logits are two-dimensional matrices of the same dimension, and each element in the two-dimensional matrix, i.e., logit, represents the confidence score of the model for each category.

[0034] Figure 1 The teacher logits and student logits in are both two-dimensional matrices with dimensions of 5×3.

[0035] S2. Perform inter-class dimension normalization correction and intra-class dimension normalization correction on each logit in the teacher logits and student logits, and calculate the inter-class normalization loss and intra-class normalization loss based on the correction results.

[0036] The calculation method of the inter-class normalization loss includes the following steps, namely S2a1~S2a3.

[0037] S2a1. Perform inter-class normalization correction on the logit of each sample in each category in the batch and calculate the first correction result.

[0038] The teacher's prediction scores indicate the teacher's confidence in all categories. In traditional knowledge distillation, the student is required to accurately match the teacher's predictions, but due to the differences in capacity and structure between the teacher and student models, the student cannot produce the same distribution as the teacher. In particular, when the teacher's confidence in a certain class is extremely high, the average confidence of the remaining non-target classes is close to 0. In this case, the temperature T The softening effect of the soft label is very small, the soft label effect is almost equivalent to the hard label, and some effective information is lost during the training process. In fact, what students want to obtain from the teacher is not only the absolute probability value of each class, but also the relative relationship between the classes, that is, the relative ranking between the classes. In order to alleviate the impact of the above situation, the present invention proposes to perform a normalized correction on logits, and the calculation formula of the first correction result is:

[0039] ;

[0040] In the formula, Indicates the batch i The sample in j logit on categories, , B is the number of samples in the batch, K is the number of categories, R is the set of real numbers; express The corresponding first correction result; Indicates i The mean of the logit of samples in all categories; Indicates i The variance of the logit of samples across all categories.

[0041] S2a2. Cool down the first correction result and perform Softmax function processing to obtain the first probability of each sample in each category in the batch. The expression formula of the first probability is:

[0042] ;

[0043] In the formula, For the i The sample in j The first probability of each category; e is a natural constant; T is the distillation temperature, T >1; 1≤ n ≤ K .

[0044] S2a3. Calculate inter-class normalized loss based on the first probability :

[0045] ;

[0046] In the formula, is the first i The sample in j The first probability of each category; is the first i The sample in j The first probability of each category; log(·) is the logarithmic function.

[0047] Through this method, the original arbitrarily distributed logits of students and teachers are transformed and mapped to a Gaussian distribution. In this Gaussian distribution, the mean is set to 0, so the category with the highest prediction confidence now receives the largest positive score, while the category with the lowest confidence receives the smallest negative score. Since this transformation process is linear, the rankings between different classes remain unchanged, which ensures that the relationship between classes can be correctly preserved. In addition, by utilizing the variance, all predicted values ​​are limited to a certain range, which is different from the traditional method of using temperature alone. T Softening, the variance can be adaptively adjusted according to the amplitude of the predicted value, combined with the temperature T By controlling the basic range, the entire logits distribution is smoother, which effectively narrows the performance gap between teachers and students due to structural differences.

[0048] In addition, traditional methods are limited to the transfer of knowledge between instances, that is, the transfer of inter-class relationships. However, the present invention believes that intra-class correlation is crucial, which contains contextual information between samples. Through this information, students can learn the reasons for their high and low confidence. For example, a picture A labeled "apple" has the highest confidence. When knowledge is transferred in the inter-class dimension, A can learn that it belongs to the "apple" category, not "banana" or "orange". When knowledge is transferred in the intra-class dimension, A can learn why it is more like an "apple" than pictures with lower confidence than itself, and can also learn why it is less like an "apple" than pictures with higher confidence than itself. Based on the above analysis, the present invention proposes to integrate intra-class correlation into the learning process, and the specific method is to apply normalization correction to the intra-class dimension.

[0049] The calculation method of the intra-class normalization loss includes the following steps, namely S2b1~S2b3.

[0050] S2b1. Perform intra-class normalization correction on the logit of each sample in each category in the batch and calculate the second correction result.

[0051] The calculation formula for the second correction result is:

[0052] ;

[0053] express The corresponding second correction result; Indicates that all samples in j The mean of the logit on the categories; Indicates that all samples in j The variance of the logit over the categories.

[0054] S2b2. Perform cooling and Softmax function processing on the second correction result to obtain the second probability of each sample in the batch in each category.

[0055] The expression formula of the second probability is:

[0056] ;

[0057] In the formula, For the i The sample in j The second probability of the category.

[0058] S2b3. Based on the second probability, calculate the intra-class normalization loss .

[0059] Intra-class normalization loss The calculation formula is:

[0060] ;

[0061] In the formula, is the first i The sample in j The second probability of the categories; is the first i The sample in j The second probability of the category.

[0062] After the above processing, the logits of each instance incorporates the information of other instances in the batch. When the intra-class predictions of the teacher and student models are adjusted on the same scale, this will affect the probability distribution of a single instance across all classes, providing students with more subtle details and differences, enabling them to learn important but implicit structural knowledge. In addition, since each logits value obtains contextual information, this can effectively alleviate the problem of the teacher's overconfidence in a certain class. In summary, the introduction of intra-class correlation effectively explores the limitations of traditional knowledge distillation and brings considerable performance improvements to the knowledge distillation method of the present invention.

[0063] S3. Train the student logits with the original true labels of the samples and calculate the classification loss.

[0064] The calculation formula of the classification loss is:

[0065] ;

[0066] In the formula, L cls is the classification loss; L CE (·) is the cross entropy loss function; l is the true label; p stu predicting outcomes for students; is the true label j The probability of each category; Predict the first j The probability of a class.

[0067] S4. Weighting the inter-class normalization loss, the intra-class normalization loss and the classification loss to obtain a total training loss.

[0068] The total training loss is calculated as:

[0069] ;

[0070] In the formula, L total is the total training loss; α , β , γ is a hyperparameter for balancing loss. Preferably, α , β , γ All three can take values ​​of 1.0.

[0071] S5. Perform gradient descent and back propagation based on the total training loss to optimize the student model parameters, and then continue to optimize the learning model using the next batch of samples in the training set until the student model converges.

[0072] The method of the present invention combines the inter-class relationship with the intra-class correlation in a simple and effective strategy, and reconstructs the logits in two dimensions through normalization correction. By learning new representations, students successfully get rid of blind imitation of teachers and establish a deeper understanding of the inter-class and intra-class relationships. This deep understanding ability can effectively narrow the gap between the student model and the teacher model, and improve the robustness and generalization ability of students.

[0073] In order to verify the effectiveness of the knowledge distillation method disclosed in the present invention, this embodiment also provides the following experiments:

[0074] 1. Data Setting

[0075] This example conducts experiments on three widely used datasets, including CIFAR-100, ImageNet, and MS-COCO. First, CIFAR-100 contains 60,000 color images with a resolution of 32×32, divided into 100 categories, of which 50,000 are used for training and 10,000 are used for validation. ImageNet is one of the most challenging datasets for image classification, which includes more than 1 million high-resolution images covering 1,000 categories. MS-COCO is a mainstream benchmark dataset for object detection, which contains 80 object categories and more than 110,000 images.

[0076] 2. Network Settings

[0077] This embodiment conducts experiments on a variety of network architectures, including ResNet, WideResNet, VGG, MobileNetV2, ShuffleNetV1, and ShuffleNetV2. In the object detection experiment conducted on the MS-COCO dataset, this embodiment uses Faster-RCNN and FPN as feature extractors. In the experimental design, this embodiment considers two architecture configurations: one is that the teacher network and the student network have the same structure (for example, ResNet50 and ResNet20), and the other is that the teacher network and the student network have different structures (for example, ResNet50 and MobileNetV2).

[0078] Implementation Details

[0079] The experimental setting of this embodiment follows the same setting as the existing work: Zhao, B., Cui, Q., Song, R., Qiu,Y., Liang, J.: Decoupled knowledge distillation. In: Proceedings of the IEEE / CVF Conference on computer vision and pattern recognition. pp. 11953–11962(2022). For the CIFAR-100 dataset, the batch size is 64, the initial learning rate is set to 0.05, and it is divided by 10 at 150, 180, and 210 rounds, respectively, and SGD is used for training for 240 epochs. For the ImageNet dataset, the batch size is 128, the initial learning rate is set to 0.2, and it is divided by 10 at 30, 60, and 90 rounds, respectively, and SGD is used for training for 100 epochs. For the MS-COCO dataset, the batch size is 8 and the initial learning rate is 0.01. All experiments were completed on one GPU. On the CIFAR-100 dataset and the ImageNet dataset, for the hyperparameters in the calculation formula of the total training loss, this embodiment sets On the MS-COCO dataset, this embodiment sets , .

[0080] IV. Main Results

[0081] This example benchmarks two representative tasks, image classification and object detection, and compares them with previous logit distillation methods to evaluate the performance of the method of the present invention. In order to highlight the advantages and potential of the method of the present invention, this example also compares with a variety of mainstream feature distillation methods, although these methods require more computing resources.

[0082] 4.1. Image classification of CIFAR-100 dataset:

[0083] This embodiment evaluates the method of the present invention on the CIFAR-100 dataset and compares it with the existing methods. Tables 1 and 2 show the verification accuracy of different methods under different teacher-student architectures. Table 1 shows the case where the teacher model and the student model have the same architecture. Table 2 shows the case where the teacher and the student use different network architectures. The results marked with "+ +" in the cell represent the best results, and the results marked with "+" represent the second best results.

[0084] Table 1. Top-1 accuracy results of CIFAR-100 dataset (same teacher-student architecture)

[0085] ;

[0086] Table 2. Top-1 accuracy results of CIFAR-100 dataset (different teacher-student architectures)

[0087] ;

[0088] FitNet is existing research: Romero A, Ballas N, Kahou SE, et al. Fitnets: Hints for thin deep nets[J]. arXiv preprint arXiv:1412.6550, 2014.

[0089] OFD is existing research: Heo B, Kim J, Yun S, et al. A comprehensive overhaul offeature distillation[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 1921-1930.

[0090] CRD is an existing study: Tian Y, Krishnan D, Isola P. Contrastive representation distillation[J]. arXiv preprint arXiv:1910.10699, 2019.

[0091] ReviewKD is existing research: Chen P, Liu S, Zhao H, et al. Distilling knowledgevia knowledge review[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition. 2021: 5008-5017.

[0092] CAT-KD is an existing research: Guo Z, Yan H, Li H, et al. Class attention transferbased knowledge distillation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 11868-11877.

[0093] KD is an existing study: Hinton G. Distilling the Knowledge in a Neural Network[J]. arXiv preprint arXiv:1503.02531, 2015.

[0094] TAKD is existing research: Mirzadeh SI, Farajtabar M, Li A, et al. Improvedknowledge distillation via teacher assistant[C] / / Proceedings of the AAAIconference on artificial intelligence. 2020, 34(04): 5191-5198.

[0095] CTKD is existing research: Li Z, Li

[0096] DKD is an existing study: Zhao B, Cui Q, Song R, et al. Decoupled knowledge distillation[C] / / Proceedings of the IEEE / CVF Conference on computer vision and pattern recognition. 2022: 11953-11962.

[0097] LSKD is existing research: Sun S, Ren W, Li J, et al. Logit standardization inknowledge distillation[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. 2024: 15731-15740.

[0098] Ours is the knowledge distillation method of the present invention.

[0099] It can be seen from Table 1 and Table 2 that in all 11 groups of experiments, the present invention won the first place in 7 groups of experiments and the second place in 4 groups of experiments. It is worth noting that compared with the distillation method based on class relations, the method of the present invention achieved the best results in all experiments. Compared with the traditional knowledge distillation method, the method of the present invention achieved at least 1% accuracy improvement in all experimental settings, which strongly proves the effectiveness of the method of the present invention. In addition, even compared with the mainstream feature distillation method, the method of the present invention only has one set of experimental accuracy that is second best in the case of teacher-student isomorphism, which shows that the method of the present invention has comparable or even better performance than the feature distillation method, achieving a balance between distillation performance and training efficiency.

[0100] 4.2. Image classification of ImageNet dataset:

[0101] Please see Table 3, which shows the experimental results on the ImageNet dataset.

[0102] Table 3. Two evaluation results of ImageNet dataset

[0103] ;

[0104] AT is existing research: Zagoruyko S, Komodakis N. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer[J]. arXiv preprint arXiv:1612.03928, 2016.

[0105] As can be seen from Table 3, this embodiment adopts two network structure designs: teacher-student homogeneous (ResNet34 and ResNet18) and heterogeneous (ResNet50 and MobileNetV2). Under the two evaluation criteria of Top-1 and Top-5, the method of the present invention has achieved the best results in all benchmark models. Even in the face of such a complex large-scale data set, the method of the present invention still performs well, which fully reflects its strong generalization ability and robustness.

[0106] 4.3. Object detection of MS-COCO dataset:

[0107] In order to explore the versatility of the method of the present invention, this embodiment applies it to the target detection task and conducts a series of experiments. Please refer to Table 4. The experiment adopts two network structure designs: teacher-student homogeneous (ResNet101 and ResNet18) and heterogeneous (ResNet50 and MobileNetV2), and evaluates it on the MS-COCO dataset, and uses Faster-RCNN-FPN as the feature extractor.

[0108] Table 4. Three evaluation results of the MS-COCO dataset

[0109] ;

[0110] FGFI is an existing research: Wang T, Yuan L, Zhang X, et al. Distilling objectdetectors with fine-grained feature imitation[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2019: 4933-4942.

[0111] As shown in Table 4, the method of the present invention has the following advantages: 50 , A.P. 75 Under all three evaluation indicators, the proposed method surpassed the previous knowledge distillation method based on class relations, and obtained the best results under the teacher-student isomorphic structure, surpassing the mainstream feature knowledge distillation method. Under the teacher-student heterogeneous structure, the proposed method also achieved at least the second place, showing a very competitive performance compared with the mainstream feature knowledge distillation method.

[0112] 5. Expansion

[0113] The above experiments prove the remarkable results of the method of the present invention in image classification and target detection tasks. In order to more clearly illustrate the advantages of the method of the present invention, it is now expanded from the following four aspects. First, this embodiment designs an ablation experiment to analyze the role of each component of the method. Then, this embodiment tests the sensitivity of hyperparameters. In addition, the training efficiency of the method of the present invention is compared with the mainstream KD method. Finally, some visual data are used to prove the superiority of the method of the present invention.

[0114] 5.1 Ablation Experiment

[0115] The method of the present invention normalizes and corrects logits in two dimensions, namely, normalization and correction of the inter-class dimension and normalization and correction of the intra-class dimension. In order to verify the effectiveness of the two, this embodiment also separately trains a student model using only one correction. The teacher model is ResNet56, the student model is ResNet20, the data set is CIFAR-100, and the Top-1 evaluation standard is adopted. The results are shown in Table 5.

[0116] Table 5. Ablation experiment results

[0117] ;

[0118] The results in Table 5 prove that using either inter-class dimension normalization or intra-class dimension normalization alone can significantly improve performance, and using both together can further improve performance and achieve the best results.

[0119] 5.2 Hyperparameter Sensitivity

[0120] This embodiment also adds the hyperparameters in the above total training loss calculation formula α , β , γ A sensitivity test was performed to determine the best hyperparameter configuration. The teacher model and student model are ResNet32×4 and ResNet8×4 respectively, the dataset is CIFAR-100, and the Top-1 evaluation standard is used. The results are shown in Table 6.

[0121] Table 6. Hyperparameter sensitivity test results

[0122] ;

[0123] 5.3 Training Efficiency

[0124] In order to verify the high efficiency of the method of the present invention, this example compares the training efficiency with various methods. Experiments were conducted on the teacher-student isomorphic model (ResNet32×4 and ResNet8×4) on the CIFAR-100 dataset. The results are as follows: Figure 3As shown in the figure, the horizontal axis is the shortest training time for each batch, and the vertical axis is the Top-1 accuracy. The upper left corner of the figure represents higher efficiency, and the lower right corner represents lower efficiency. When the method of the present invention achieves the best accuracy, the training time is comparable to that of the traditional knowledge distillation method, and the training efficiency is the highest. This is mainly due to the fact that the method of the present invention only requires simple processing of logits without introducing additional training parameters. In contrast, although other feature knowledge distillation methods can achieve better results than traditional KD, they usually require complex transformation of features, so they require more time and computational cost.

[0125] 5.4 Visualization

[0126] In order to intuitively demonstrate the effect of the method of the present invention, t-SNE is used to display the characteristics of the student model of the present invention, and the distance between the teacher and student logits correlation matrices is visualized. The teacher model uses ResNet32×4, the student model uses ResNet8×4, and the data set is CIFAR-100. The t-SNE visualization results are as follows: Figure 4 and Figure 5 As shown, the student model distilled by the method of the present invention has more obvious features between categories and more compact feature distribution within the same category, indicating that the student model has a clearer understanding of the relationship between and within categories.

[0127] Figure 6 and Figure 7 The difference between the correlation matrix of the teacher and the student is shown. Compared with the traditional knowledge distillation, the difference between the student model trained by the method of the present invention and the teacher is smaller, that is, the logits of the student model are closer to the logits of the teacher model. This shows that the method of the present invention can effectively reconstruct logits, so that the student model can better imitate the behavior of the teacher model and learn its ability to discriminate data.

[0128] Example 2

[0129] See also Figure 8 This embodiment provides a two-dimensional normalized knowledge distillation system based on class relations, including: a sample input module, a loss calculation module and a student model output module. The system can apply the knowledge distillation method in Example 1.

[0130] The sample input module is used to input the samples, i.e., images, in the training set batches into the teacher model and the student model, respectively, to obtain the outputs of the last hidden layers of the teacher model and the student model, i.e., teacher logits and student logits; wherein, teacher logits and student logits are two-dimensional matrices of the same dimension, each element in the two-dimensional matrix, i.e., logit, represents the confidence score of the model for each category, and the number of rows and columns of the two-dimensional matrix correspond to the number of samples and the number of categories, respectively;

[0131] The loss calculation module is used to perform inter-class dimension normalization correction and intra-class dimension normalization correction on each logit in the teacher logits and the student logits, and calculate the inter-class normalization loss and the intra-class normalization loss according to the correction results; the loss calculation module is also used to train the student logits with the original true labels of the samples to calculate the classification loss; the loss calculation module is also used to weight the inter-class normalization loss, the intra-class normalization loss and the classification loss to obtain the total training loss;

[0132] The student model output module is used to perform gradient descent and back propagation based on the total training loss to optimize the student model parameters, and then continuously optimize the learning model using the next batch of samples in the training set until the student model converges.

[0133] Example 3

[0134] The present invention also discloses a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the two-dimensional normalized knowledge distillation method based on class relations described in Example 1 are implemented.

[0135] like Fig. 9 As shown, the computer terminal provided in this embodiment includes: at least one processor 101, and a memory 102 connected to the at least one processor 101. The specific connection medium between the processor 101 and the memory 102 is not limited in this embodiment. Figure 5 In the example, the processor 101 and the memory 102 are connected via the bus 100. The bus 100 is connected to the memory 102 via the bus 100. Figure 5 The connections between other components are shown by thick lines, which are only schematic and not limiting. The bus 100 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 5 In the figure, only one thick line is used, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 101 can also be called a controller, and there is no limitation on the name.

[0136] In this embodiment, the memory 102 stores instructions that can be executed by at least one processor 101. The at least one processor 101 can execute the knowledge distillation method discussed in Example 1 by executing the instructions stored in the memory 102. The processor 101 can implement Figure 8 The functions of each module in the system are shown.

[0137] Among them, the processor 101 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 102 and calling data stored in the memory 102, the various functions of the device and process data, the device can be monitored as a whole.

[0138] In one possible design, the processor 101 may include one or more processing units, and the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 101. In some embodiments, the processor 101 and the memory 102 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0139] Processor 101 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment. A general-purpose processor may be a microprocessor or any conventional processor. In combination with the steps of the knowledge distillation method in Example 1, the steps may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in processor 101.

[0140] The memory 102 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 102 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 102 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 102 in this embodiment can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0141] By designing and programming the processor 101, the code corresponding to the knowledge distillation method described in the above embodiment can be fixed into the chip, so that the chip can execute Figure 1 The steps of the knowledge distillation method of the embodiment shown are as follows: How to design and program the processor 101 is a technique known to those skilled in the art and will not be described in detail here.

[0142] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A two-dimensional normalized knowledge distillation method based on class relations, characterized in that: include: Input the samples, i.e., images, in the training set batch into the teacher model and the student model respectively, and obtain the outputs of the last hidden layer of the teacher model and the student model, i.e., teacher logits and student logits; wherein, teacher logits and student logits are two-dimensional matrices of the same dimension, each element, i.e., logit, in the two-dimensional matrix represents the confidence score of the model for each category, and the number of rows and columns of the two-dimensional matrix corresponds to the number of samples and the number of categories respectively; Perform inter-class dimension normalization correction and intra-class dimension normalization correction on each logit in the teacher logits and student logits, and calculate the inter-class normalization loss and intra-class normalization loss based on the correction results; Train the student logits with the original true labels of the samples and calculate the classification loss; The inter-class normalization loss, the intra-class normalization loss and the classification loss are weighted to obtain a total training loss, and gradient descent and back propagation are performed based on the total training loss to optimize the student model parameters, and then the learning model is continuously optimized using the next batch of samples in the training set until the student model converges; The step of performing inter-class dimension normalization correction and intra-class dimension normalization correction on each logit in the teacher logits and the student logits includes: Perform inter-class normalization correction on the logit of each sample in each category in the batch, and calculate the first correction result; perform intra-class normalization correction on the logit of each sample in each category in the batch, and calculate the second correction result; the calculation formulas of the first correction result and the second correction result are respectively: In the formula, z ij represents the logit of the i-th sample in the batch on the j-th category, z ij ∈R B×K , B is the number of samples in the batch, K is the number of categories, and R is a real number set; z ij (x) represents z ij The corresponding first correction result; M i· represents the mean of the logit of the i-th sample in all categories; V i· represents the variance of the logit of the i-th sample in all categories; z ij (y) represents z ij The corresponding second correction result; M ·j represents the mean of the logit of all samples in the jth category; V ·j Represents the variance of the logit of all samples in the jth category; The calculating of the inter-class normalization loss and the intra-class normalization loss according to the correction result includes: The first correction result and the second correction result are simultaneously subjected to cooling and Softmax function processing, so as to obtain the first probability and the second probability of each sample in each category in the batch respectively; the expression formulas of the first probability and the second probability are respectively: Where P ij (x) is the first probability of the i-th sample in the j-th category; e is a natural constant; T is the distillation temperature, T>1; 1≤n≤K; P ij (y) is the second probability of the i-th sample in the j-th category; Based on the first probability and the second probability, the inter-class normalization loss L is calculated respectively. xnorm and intra-class normalization loss L ynorm , the calculation formulas are: In the formula, is the first probability of the i-th sample in the teacher logits on the j-th category; is the first probability of the i-th sample in the student logits on the j-th category; is the second probability of the i-th sample in the teacher logits on the j-th category; is the second probability of the i-th sample in the student logits in the j-th category; log(·) is the logarithmic function.

2. The two-dimensional normalized knowledge distillation method based on class relationship according to claim 1 is characterized in that: The calculation formula of the classification loss is: Where, L cls is the classification loss; L CE (·) is the cross entropy loss function; l is the true label; p stu Predict results for students; j is the probability of the true label on the jth category; Predict the probability of the jth category for the student model.

3. The two-dimensional normalized knowledge distillation method based on class relationship according to claim 2 is characterized in that: The total training loss is calculated as: L total =αL cls +βL xnorm +γL ynorm ; Where, L total is the total training loss; α, β, and γ are the hyperparameters of the balanced loss, and their values ​​are all 1.

0.

4. The two-dimensional normalized knowledge distillation method based on class relationship according to claim 1, characterized in that: The samples are pictures of computer vision tasks to be performed, and the types of computer vision tasks include image classification, object detection and semantic segmentation.

5. A two-dimensional normalized knowledge distillation system based on class relations, characterized in that: Applying the knowledge distillation method as claimed in claim 1, the system comprises: The sample input module is used to input the samples, i.e., images, in the training set batches into the teacher model and the student model respectively, and obtain the outputs of the last hidden layers of the teacher model and the student model, i.e., teacher logits and student logits; wherein, teacher logits and student logits are two-dimensional matrices of the same dimension, each element, i.e., logit, in the two-dimensional matrix represents the confidence score of the model for each category, and the number of rows and columns of the two-dimensional matrix corresponds to the number of samples and the number of categories respectively; A loss calculation module is used to perform inter-class dimension normalization correction and intra-class dimension normalization correction on each logit in the teacher logits and the student logits, and calculate the inter-class normalization loss and the intra-class normalization loss according to the correction results; the loss calculation module is also used to train the student logits with the original true labels of the samples to calculate the classification loss; the loss calculation module is also used to weight the inter-class normalization loss, the intra-class normalization loss and the classification loss to obtain the total training loss; The student model output module is used to perform gradient descent and back propagation based on the total training loss to optimize the student model parameters, and then continuously optimize the learning model using the next batch of samples in the training set until the student model converges.

6. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the two-dimensional normalized knowledge distillation method based on class relations as described in any one of claims 1 to 4.

Images