Knowledge distillation method and system based on view alignment

Abstract

The invention relates to the technical field of machine learning, in particular to a knowledge distillation method and system based on view alignment. The invention aims to solve the main limitations in the traditional distillation technology, including the problems of excessive confidence of a teacher model, confirmation deviation and the like. The invention provides a novel knowledge distillation framework based on logarithmic probability, which is called KDVA; specifically, z-score standardization is applied to the logarithmic probability of a model for smooth output, so that transmission of more teacher implicit knowledge is promoted. In addition, a same-view-angle and cross-view-angle alignment mechanism is introduced into the KDVA, and comparison information of weak and strong image transformation is utilized to enlighten a student model to obtain more knowledge. In addition, the student model is supervised by using the real label of the sample, so that the student model can obtain more information about each sample target category. According to the method, when different network architectures are applied to different data sets, high effectiveness and stability are shown.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a knowledge distillation method and system based on view alignment. BACKGROUND

[0002] Deep neural networks (DNNs) have shown remarkable performance in image classification, brain disease exploration, image segmentation, language recognition, and object detection. However, their high performance is often accompanied by excessive model size and computational requirements, which limits their application on resource-limited devices. To address these challenges, knowledge distillation (KD) technology has emerged. This method, initially proposed by Hinton et al., aims to reduce model size by transferring the representation information learned by a complex high-capacity teacher network to a more compact student model. Over the past decade, knowledge distillation technology has made significant progress, with feature-based and logits-based algorithms developed for different tasks. Advanced knowledge distillation methods have significantly narrowed the performance gap between teacher and student models.

[0003] However, existing distillation algorithms still face the following problems: (1) Overconfidence of the teacher: Although the "dark information" of non-target classes learned by the teacher is valuable to the student, the teacher's excessively high prediction confidence can suppress such information. (2) Confirmation bias: The teacher's predictions are not absolutely correct, and existing logit-based distillation methods require students to blindly follow the teacher's supervision, which can exacerbate bias due to false pseudo-labels.

[0004] In summary, based on the above technical problems, the present application introduces KDVA, a logits knowledge distillation framework based on view alignment. We propose the KDVA method to simultaneously adopt intra-view and cross-view alignment mechanisms to encourage the student model to acquire representative knowledge from the teacher model. In addition, we also utilize sample label information to further guide the student model to acquire more discriminative knowledge for target classes. The present application method does not introduce additional network parameters, thus being simple, effective, and easy to implement. SUMMARY

[0005] The main purpose of the present application is to provide a knowledge distillation method and system based on view alignment to overcome the problems of overconfidence and confirmation bias in existing distillation algorithms. By introducing KDVA, a logits knowledge distillation framework based on view alignment, we ensure high performance and stability of the output student model when applying different network architectures on different datasets.

[0006] The specific technical solutions are as follows:

[0007] A knowledge distillation method based on view alignment, specifically comprising the following steps:

[0008] S1: Teacher model training, based on the selected data set and network architecture, using standard supervised learning to train the teacher model;

[0009] S2: Knowledge distillation, based on the selected data set input of the original sample image, after differential enhancement, the original sample image and the enhanced differential image are respectively input into the trained teacher model and the selected student model, and the differential view results are output; after processing, the output logits are processed using standard scores, and then the loss is calculated based on the alignment standard of the same view and cross view of the differential image, and the loss is calculated by using the real label to supervise the output of the student model on different differential views and adjusting the weight of the hyperparameter; then the student model is optimized based on the total loss, and the distillation process is repeated until the student model converges;

[0010] S3: Based on the selected same data set, the student model is evaluated after convergence, and the performance parameters of the model are output as evaluation reference.

[0011] Further, the differential enhancement of S2 includes strong enhancement and weak enhancement; the strong enhancement is random enhancement of the original sample image with random amplitude, and auxiliary random cropping, random horizontal flipping or random masking; the weak enhancement is random cropping and random horizontal flipping of the original sample image.

[0012] Further, the differential view result of S2 includes the original view logits result output by the teacher model, the weak enhancement view logits result and the strong enhancement view logits result, and the original view logits result output by the student model, the weak enhancement view logits result and the strong enhancement view logits result.

[0013] Further, the processing using standard scores in S2 is to use simple z-score to regularize the logits result, and the formula is defined as: Wherein, σ represents the variance of logits corresponding to the test sample, and μ represents the mean of logits corresponding to the test sample.

[0014] Further, the loss calculation based on the alignment standard of the same view and cross view of the differential image in S2, wherein the alignment loss of the same view is calculated as follows: Wherein, KL represents the Kullback-Leibler divergence, i.e. relative entropy, p t and p srespectively represent the probability distribution obtained by the normalized exponential function from the logits of the teacher model and the student model, and the subscripts o, w, s represent the original sample, the weakly augmented sample and the strongly augmented sample respectively; wherein the cross-view alignment loss is calculated as follows: , each symbol has the same meaning as in the same view.

[0015] Further, the cross-view alignment loss calculation uses different alignment regularization methods, specifically: the teacher model and the student model are required to generate probability distributions as similar as possible when receiving different views of the same sample.

[0016] Further, S2 uses the real label to supervise the output of the student model on different differentiated views and calculates the loss, and the loss function is represented as follows: , wherein CE represents the standard cross-entropy loss, and y represents the real label.

[0017] Further, the total loss of S2 can be represented as: , wherein beta and lambda are hyperparameters of S2, the value range of beta can be 1-7, and the value range of lambda can be 1-4.

[0018] The application also provides a view alignment-based knowledge distillation system, characterized in that the system comprises a data set reading module, a teacher model training module, a knowledge distillation module and a model evaluation module; the data set reading module is responsible for extracting the required number of data subsets from the data set according to different task requirements; the teacher model training module is responsible for training the teacher model based on the selected data set, network framework and task target using standard supervised learning; the knowledge distillation module is responsible for knowledge distillation from the teacher model to the student model; the model evaluation module is responsible for performance evaluation of the output student model based on the selected data set.

[0019] Further, the knowledge distillation module comprises an image enhancement submodule, an output processing submodule, a view alignment submodule, a hard label supervision submodule, a loss calculation submodule and a student model optimization submodule.

[0020] Compared with the prior art, the application has the following beneficial effects:

[0021] (1) The application introduces a same-view and cross-view alignment mechanism to solve the problems of overfitting and excessive confidence of the teacher model in knowledge distillation.

[0022] (2) We use simple z-score to regularize the prediction of the teacher model, thereby reducing the confirmation bias in knowledge distillation.

[0023] (3) We propose a KDVA knowledge distillation framework structure, which is simple, general and efficient, and can effectively improve the performance of student models. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Figure 1 is a schematic diagram of the KDVA knowledge distillation framework in the knowledge distillation method based on view alignment of the present application;

[0025] Figure 2 Figure 2 is a structural schematic diagram of the knowledge distillation system based on view alignment of the present application. DETAILED DESCRIPTION

[0026] The following embodiments further explain and illustrate the technical solutions of the present application. It is particularly pointed out that each specific embodiment is a specific embodiment and explanation of the technical solution, and should not be regarded as a limitation on the protection scope of the present application. Those skilled in the art still have the right to modify the technical solutions of these embodiments, to equivalently replace part or all of the technical features, and these modifications or replacements do not change the essence of the corresponding technical solutions, and do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions described in the present application.

[0027] The present application proposes a knowledge distillation method and system based on view alignment, as shown in the accompanying drawings Figure 1 Figure 1 is a schematic diagram of the KDVA knowledge distillation framework in the knowledge distillation method of the present application, and the specific key technical solution design is as follows:

[0028] 1. Improvement of standard knowledge distillation method: The standard knowledge distillation method guides the student model to capture the implicit knowledge of the teacher model by minimizing the Kullback-Leibler divergence (KL) between the logits (i.e. the original score of the model without normalization) of the student model and the teacher model. In the classification task, the real label is usually used for supervision through the cross-entropy function during the training process. The mathematical expression of the loss function is:

[0029] , where L CE represents the cross-entropy loss, and its expression is:

[0030] , α is the weight term for balancing the distillation loss, and y represents the real label of the sample;

[0031] L KD represents the distillation loss, and its expression is: , where and respectively represent the logits of the teacher model and the student model, σ(•) is the softmax function (normalized exponential function), and C represents the number of classes in the sample.

[0032] The distillation method of the present application refers to the standard knowledge distillation method, and specifically can refer to document 【1】Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network[J]. arXiv preprint arXiv:1503.02531, 2015. On this basis, in order to obtain a smoother probability distribution, we choose to use a simple z-score to regularize logits, and the formula is defined as: Where σ represents the variance of logits corresponding to the sample, and μ represents the mean of logits corresponding to the sample.

[0033] 2. Multi-view alignment scheme based on view alignment of semi-supervised learning algorithm: View alignment is crucial to the success of semi-supervised learning algorithm. It involves imposing an invariant representation in different views of the same unlabeled input image, thereby enhancing the generalization ability of the learned representation to unknown data and distribution. Based on this method, the present application generates multiple views of the same unlabeled image, each of which is input into the neural network to obtain logits. With the prior knowledge that these views have the same high-level semantics (e.g., object class), alignment regularization constraints are imposed between the prediction results.

[0034] In the KDVA framework (logits distillation framework based on view alignment) constructed by the present application, we use a strong view and a weak view to lay the foundation for our same-view and cross-view alignment criteria. Specifically, our strong data augmentation strategy combines RandAugment (random augmentation) with random amplitude, supplemented by random cropping, random horizontal flipping, or Cutout (random masking). For weak augmentation, we only apply random cropping and random horizontal flipping. The method of forming different images with respect to the above image processing is a standard practice in previous logits-based knowledge distillation methods, and specifically can refer to 【2】Zhao B, Cui Q, Song R, et al. Decoupled knowledge distillation C] / / CVPR, 2022:11953-11962. The present application respectively represents the transformation functions of the weak view and the strong view as T W (•) and T S (•).

[0035] Specifically, given the original sample view x o , we generate the weak augmented view x W and the strong augmented view x S through T w (•) and T sThe three generated view images are processed by the teacher model and the student model respectively to obtain original view logits, weak enhanced view logits and strong enhanced view logits output by the teacher model, and original view logits, weak enhanced view logits and strong enhanced view logits output by the student model. Then the same view alignment regularization is defined as the alignment standard between the prediction results of the teacher model and the student model under the same original view, weak view or strong view. Thus, the same view alignment target loss is calculated as follows: , wherein p t and p s respectively represent the probability distribution obtained by the softmax function from the logits of the teacher model and the student model, and the subscripts o, w and s represent the original, weak enhancement and strong enhancement. For cross-view processing, the present application proposes a novel cross-view alignment regularization method. This method requires the teacher model and the student model to generate probability distributions as similar as possible when receiving different angles of the same sample. The mathematical expression of the cross-view loss function based on this is as follows: In order to make full use of sample label information, the present application also uses the real label to supervise the output of the student in different views, and the loss function is shown as follows: , wherein CE represents the standard cross-entropy loss, and y represents the real label. In summary, the total loss can be represented as: , wherein β and λ are weight adjustment parameters.

[0036] In summary, the general process of a standard knowledge distillation method includes: (1) teacher model training, that is, using a large and complex but excellent teacher model to train a large and complex but excellent teacher model on a large data set using standard supervised learning method, and freezing the weights of the teacher model after training, and the output of the teacher model is not updated thereafter; (2) knowledge distillation, using the same or related training data set as training the teacher model, performing forward propagation, that is, for each batch of samples in the training set, inputting them into the teacher model and the student model at the same time, obtaining their outputs and incorporating the "smoothness" of the output probability distribution controlled by the temperature hyperparameter; then calculating the loss function, that is, calculating the distillation loss, the student loss and the total loss; then performing back propagation and optimization, calculating the gradient of the total loss with respect to the parameters of the student model, and updating the parameters of the student model using the optimizer according to the gradient; repeat the above forward propagation to back propagation and optimization steps until the student model converges; (3) student model evaluation, evaluate the student model on an independent test set, and output the final performance indicators of the student model including accuracy, etc.

[0037] Compared with the distillation method in the standard and prior art, the present application makes the following improvements: (1) in the processing of logits output values of the teacher model and the student model, based on the consideration of obtaining a smoother probability distribution, the logits are regularized by using a simple z-score; (2) in the construction of the distillation framework, the present application constructs a logits distillation framework based on view alignment, which enables the student model to learn the knowledge of the teacher model from different view domains through data enhancement technology, by enhancing the input image into strong and weak views and inputting them into the teacher model and the student model respectively, thereby generating six prediction results for the same sample, and then using the same view and cross-view alignment mechanism to guide the student model to capture representative knowledge from the teacher model, and further guiding the student model to obtain differentiated knowledge of the target class by using sample label information. Among them, in the processing of cross-view, the present application also proposes a novel cross-view alignment regularization method to further improve the effect of the distillation method of the present application.

[0038] Embodiment 1

[0039] A knowledge distillation method based on view alignment, specifically as follows:

[0040] The data set used: CIFAR-100, CIFAR-100 is composed of 60000 color images, each with a size of 32x32 pixels, divided into 100 categories, and each category contains 600 images; among them, 50000 images are allocated for training, and the remaining 10000 images are used for testing;

[0041] S1: teacher model training, based on the above selected data set, the teacher model network architecture VGG13 is trained in the ubantu system based on the 3.8 version of Python environment using the standard supervised learning method.

[0042] S2: knowledge distillation, using the KDVA framework constructed by the present application, i.e. after strong and weak enhancement of the input original sample image, the original sample image, the weak enhanced image and the strong enhanced image are respectively input into the teacher model and the student model (using network architecture VGG8) for processing, and the z-score is used for regularization of the output logits, then the loss of the two is calculated based on the alignment standards of the same view and cross-view, and the loss is calculated by using the real label to supervise the student's output on different views, and the total loss is calculated after adding the hyperparameter adjustment weight; the student model is optimized based on the total loss, and the student model can obtain information about the target class of the sample according to the real label, the same view mechanism and the cross-view strategy, and the distillation process is repeated until the student model is trained.

[0043] S3: based on the same data set for training, the student model is evaluated, and the accuracy of the output model is used as the evaluation reference.

[0044] Example 2

[0045] The distillation method of Example 1 is referred to, except that:

[0046] In the S2 distillation process, the output of the student model is only supervised by the real label, the student model only learns from the hard label, and the total loss calculation only includes the loss of this item; other steps are the same.

[0047] Example 3

[0048] The distillation method of Example 1 is referred to, except that:

[0049] In the S2 distillation process, on the one hand, the output of the student model is supervised by the real label, and on the other hand, the student model also learns more implicit knowledge from the logit of the teacher model by forced alignment of multiple same views, and the total loss calculation includes the loss of the output after real label supervision and the loss under the same view learning; other steps are the same.

[0050] Example 4

[0051] The distillation method of Example 1 is referred to, except that:

[0052] In the S2 distillation process, on the one hand, the output of the student model is supervised by the real label, and on the other hand, the student model also learns more implicit knowledge from the logit of the teacher model by forced alignment of multiple same views, and the total loss calculation includes the loss of the output after real label supervision and the loss under the same view learning; other steps are the same.

[0053] Example 5

[0054] The distillation method of Example 1 is referred to, except that:

[0055] The combination of the teacher model network architecture VGG13 and the student model network architecture MobileNet V2 is used in S2; other steps are the same.

[0056] Example 6

[0057] The distillation method of Example 2 is referred to, except that:

[0058] The combination of the teacher model network architecture VGG13 and the student model network architecture MobileNet V2 is used in S2; other steps are the same.

[0059] Example 7

[0060] The distillation method of Example 3 is referred to, except that:

[0061] The combination of the teacher model network architecture VGG13 and the student model network architecture MobileNet V2 used in S2; the other steps are the same.

[0062] Example 8

[0063] Referring to the distillation method of Example 4, the difference is that:

[0064] The combination of the teacher model network architecture VGG13 and the student model network architecture MobileNet V2 used in S2; the other steps are the same.

[0065] Based on the above Examples 1-8, the accuracy of the student model output is tested, and the experimental settings follow the standard experimental settings of the classical method, keeping the same hyperparameters as the CRD method in the literature 【3】 Yang Z, Zeng A, Li Z, et al. From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels [C] / / ICCV, 2023: 17185-17194. such as learning rate, batch size and training rounds. The baseline result is obtained by training the student model using cross-entropy loss based on the output of the weak view. The specific experimental results are shown in Table 1.

[0066] Table 1 Comparison of experimental results of Examples 1-8

[0067]

[0068] Note: “ ” indicates the knowledge acquisition of the student model under the same view alignment; “ ” indicates the knowledge acquisition of the student model under cross-view alignment; “∇” indicates the improvement of the experimental results compared with the baseline; “x” indicates that the corresponding knowledge acquisition method is not used; “√” indicates that the corresponding knowledge acquisition method is used.

[0069] From the above comparison results, it can be seen that the real label supervision is beneficial because it can enable the student to obtain more implicit knowledge from the hard label; and the introduction of the same view alignment helps to more effectively transfer the representation knowledge from the teacher model to the student model; and the cross-view alignment enables the student model to learn more useful implicit knowledge from the teacher model, thereby enhancing its generalization ability. The combination of real label supervision and same view and cross view alignment mechanisms achieves the best experimental results, and it can be seen that these mechanism components are complementary and indispensable, and each component positively contributes to the final performance, and their combination always achieves the maximum benefit; they jointly ensure the most comprehensive and effective transfer of knowledge, highlighting the importance of integrating the two alignment strategies and real label supervision methods for the optimal performance of the student model. In addition, the example gives the comparison experimental results of two types of homogeneous and heterogeneous teacher-student network architectures, and stable benefits are obtained, verifying the effectiveness and universality of the method proposed in the application. The experimental results show that the knowledge distillation method of the application successfully alleviates the problems related to teacher overconfidence and confirmation bias.

[0070] Embodiment 9

[0071] The distillation method of Embodiment 1 is referred to, except that:

[0072] The combination of the teacher model network architecture ResNet32x4 and the student model network architecture ResNet8x4 used in S1; other steps are the same.

[0073] Embodiments 10-36

[0074] The distillation method of Embodiment 9 is referred to, except that:

[0075] The combination of the total loss used in S2 is different, according to the calculation formula in the above It can be seen that the superparameters include the weight superparameter β of the same view and the weight superparameter λ of the cross view; the specific combination is shown in Table 2:

[0076] Table 2 Total loss superparameter combination table used in Embodiments 9-36

[0077]

[0078] Based on the above Embodiments 9-36, the accuracy of the student model output is tested, and the experimental settings are the same as those of the aforementioned Embodiments 1-8. The specific experimental results are shown in Tables 3 and 4.

[0079] Table 3 Comparison table of experimental results of Embodiments 9-36

[0080]

[0081] Table 4 Experimental results of Examples 9-36 (%)

[0082]

[0083]

[0084] From the above comparison results, it can be seen that the student model obtained by the method of the present application performs stably under a variety of different super parameter configurations, and most of the accuracy values are concentrated in a narrow interval of 77.3% to 78.3%. The best performance (78.30%) appears in the combination of β=6 and λ=2, and other adjacent combinations also achieve considerable accuracy. These results show that although different super parameter settings can cause slight performance fluctuations, the overall accuracy remains relatively stable, proving that the model is not sensitive to small changes in these specific super parameters. This robustness makes the method more feasible in practical applications.

[0085] Example 37

[0086] The present application also provides a knowledge distillation system based on view alignment, and the specific structure is shown in the accompanying Figure 2 The data set reading module is responsible for extracting the required number of data subsets from the data set according to different task requirements. The teacher model training module trains a teacher model based on the selected data set, network framework and task target using standard supervised learning methods. The knowledge distillation module is responsible for knowledge distillation from the teacher model to the student model, which includes an image enhancement submodule, an output processing submodule, a view alignment submodule, a hard label supervision submodule, a loss calculation submodule, and a student model optimization submodule. Finally, a model evaluation module is also provided to evaluate the performance of the output student model based on the selected data set.

[0087] Comparative Example 1

[0088] The method of the present application is compared with a variety of mainstream distillation methods. The same data set as in Example 1 is used to evaluate the accuracy of the student model using a variety of network combinations of isomorphic and heterogeneous teachers and students. The robustness of the method of the present application is comprehensively tested. The specific comparison of the mainstream distillation methods is as follows:

[0089] (1) KD, reference [4] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network [J]. arXiv preprint arXiv:1503.02531, 2015.

[0090] (2) CTKD, Ref. [5] Li Z, Li X, Yang L, et al. Curriculum temperature for knowledge distillation [C] / / AAAI, 2023, 37(2): 1504-1512.

[0091] (3) DKD, Ref. [6] Zhao B, Cui Q, Song R, et al. Decoupled knowledge distillation [C] / / CVPR, 2022: 11953-11962.

[0092] (4) NKD, Ref. [7] Yang Z, Zeng A, Li Z, et al. From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels [C] / / ICCV, 2023: 17185-17194.

[0093] (5) SP, Ref. [8] Tung F, Mori G. Similarity-preserving knowledge distillation [C] / / ICCV, 2019: 1365-1374.

[0094] (6) AT, Ref. [9] Zagoruyko S, Komodakis N. Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer [C] / / ICLR, 2017.

[0095] (7) VID, Ref.

[10] Ahn S, Hu S X, Damianou A, et al. Variational information distillation for knowledge transfer [C] / / CVPR, 2019: 9163-9171.

[0096] (8) CRD, Ref.

[11] Li Z, Li X, Yang L, et al. Curriculum temperature for knowledge distillation [C] / / AAAI, 2023, 37(2): 1504-1512.

[0097] (9) CCKD, Ref.

[12] Peng B, Jin X, Liu J, et al. Correlation congruence for knowledge distillation [C] / / ICCV, 2019: 5007-5016.

[0098] (10) RKD, Ref.

[13] Park W, Kim D, Lu Y, et al. Relational knowledge distillation [C] / / CVPR. 2019: 3967-3976.

[0099] The adopted homogeneous teacher-student network architecture combinations include: ResNet32x4 and ResNet8x4, WRN-40-2 and WRN-40-1, VGG13 and VGG8, WRN-40-2 and WRN-16-2, ResNet56 and ResNet20.

[0100] The adopted heterogeneous teacher-student network architecture combinations include: ResNet32x4 and ShuffleNetV2, WRN-40-2 and ResNet8x4, VGG13 and MobileNetV2, WRN-40-2 and MobileNetV2.

[0101] The specific experimental results are shown in Tables 5 and 6.

[0102] Table 5 Comparison table of accuracy (%) experimental results of homogeneous teacher-student network architecture of Comparative Example 1

[0103] Table 6 Comparison table of accuracy (%) experimental results of heterogeneous teacher-student network architecture of Comparative Example 1

[0104]

[0105] Note: “-” means no available data.

[0106] From the above comparison results, for the architecture optimization of the isomorphic student-teacher network architecture, our method always maintains the best student model accuracy in all test network pairs. This shows that our decoupling strategy can effectively transfer the structural knowledge within the same architecture family. In the heterogeneous network environment where the student network and the teacher network architectures are different, our method also shows excellent performance. In particular, it is worth noting that on the ResNet32x4 and ShuffleNetV2 double model architectures, our method achieves an accuracy of 78.82%, which is the highest among all methods; in the harsh test of MobileNetV2 lightweight architecture, our method can still improve the student network accuracy to 70.50% and 69.91%, which is 0.79% and 0.53% higher than the second place respectively. These outstanding performances not only prove the strong adaptability of the method to architecture differences, but also highlight its practical value in real deployment scenarios - even if there are differences in the design of the teacher network and the student network, it can still maintain good compatibility.

[0107] Comparative Example 2

[0108] In the horizontal comparison of the method of the present application with a plurality of representative distillation baseline methods, Comparative Example 2 is evaluated on the ImageNet dataset, and the teacher and student network models used are ResNet34 and ResNet18 respectively. As shown in Table 7, it is the method comparison result of Top-1 and Top-5 classification accuracy of the present application on the ImageNet dataset. Top-1 represents whether the highest probability class predicted by the model is consistent with the true label, which refers to the scenario of a unique answer; Top-5 represents whether the real class is contained in the top five classes, allowing the model to choose the correct answer among multiple similar classes.

[0109] Table 7 Comparative Example 2 Different Classification Accuracy (%) Experimental Result Comparison Table

[0110] Method KD AT CRD RKD DKD The present invention Top-1 70.66 70.69 71.17 70.40 71.70 71.86 Top-5 89.88 90.01 90.13 89.78 90.41 90.54

[0111] From the above comparison results, the method of the present application achieves a Top-1 accuracy of 71.86%, which is significantly better than other methods; in terms of Top-5 accuracy, we also surpass the baseline method with a result of 90.54%; these results fully prove that the method of the present application effectively improves the knowledge transfer ability on large-scale datasets, and highlights its strong generalization ability in challenging scenarios with fine-grained class distinction.

Claims

1. A view alignment based knowledge distillation method, characterized in that, Specifically comprising the following steps: S1: teacher model training, based on the selected data set and network architecture, using standard supervised learning to train the teacher model; S2: knowledge distillation, based on the selected data set input of the original sample image, after differential enhancement, the original sample image and the enhanced differential image are respectively input into the trained teacher model and the selected student model to output the differential view result; after processing, the output logits are processed using the standard score, and then the loss is calculated based on the alignment standard of the same view and cross view of the differential image, and the loss is calculated based on the real label supervision of the output of the student model in different differential views, and the total loss is calculated after adding the weight of the hyperparameter; then the student model is optimized based on the total loss, and the distillation process is repeated after optimization until the student model converges; S3: based on the selected same data set, the student model is evaluated after convergence, and the performance parameters of the model are output as the evaluation reference.

2. The view alignment based knowledge distillation method of claim 1, wherein, The differential enhancement of S2 includes strong enhancement and weak enhancement; the strong enhancement is random enhancement of the original sample image with random amplitude, and any one or combination of random cropping, random horizontal flipping or random masking; the weak enhancement is random cropping and random horizontal flipping of the original sample image.

3. The view alignment based knowledge distillation method of claim 1, wherein, The differential view result of S2 includes the original view logits result output by the teacher model, the weak enhancement view logits result and the strong enhancement view logits, and the original view logits result, the weak enhancement view logits result and the strong enhancement view logits result output by the student model.

4. The view alignment based knowledge distillation method of claim 1, wherein, The processing using the standard score described in S2 is a simple z-score regularization processing on logits results, and the formula definition is: wherein σ represents the variance corresponding to the logits of the test sample, and μ represents the mean corresponding to the logits of the test sample.

5. The view alignment based knowledge distillation method of claim 1, wherein, S2 calculates the loss based on the alignment criterion of the same view and the cross view of the differentiated image, wherein the alignment loss of the same view is calculated as follows: , wherein KL represents the Kullback-Leibler divergence, i.e., the relative entropy, p t and p s respectively represent the probability distribution obtained by the normalized exponential function from the logits of the teacher model and the student model, and the subscripts o, w and s respectively represent the original sample, the weakly enhanced sample and the strongly enhanced sample; wherein the cross view alignment loss is calculated as follows: , wherein the symbols have the same meaning as in the same view.

6. The view alignment based knowledge distillation method of claim 5, wherein, The alignment loss calculation of cross view uses different alignment regularization methods, specifically: the probability distribution generated by the teacher model and the student model when receiving different views of the same sample should be as similar as possible.

7. The view alignment based knowledge distillation method of claim 1, wherein, S2 said using real label to supervise the output of student model on different differentiated views and calculate the loss, the loss function is represented as follows: where CE represents the standard cross-entropy loss, and y represents the real label.

8. The view alignment based knowledge distillation method of claim 1, wherein, The total loss described in S2 can be expressed as: where β and λ are hyperparameters described in S2, β can have a value range of 1-7, and λ can have a value range of 1-4.

9. The view alignment based knowledge distillation system of any one of claims 1-8, wherein, The system includes a data set reading module, a teacher model training module, a knowledge distillation module and a model evaluation module; the data set reading module is responsible for extracting the required number of data subsets from the data set according to different task requirements; the teacher model training module is responsible for training the teacher model based on the selected data set, network framework and task target using standard supervised learning; the knowledge distillation module is responsible for knowledge distillation from the teacher model to the student model; the model evaluation module is responsible for performance evaluation of the output student model based on the selected data set.

10. The view alignment based knowledge distillation system of claim 9, wherein, The knowledge distillation module includes an image enhancement submodule, an output processing submodule, a view alignment submodule, a hard label supervision submodule, a loss calculation submodule and a student model optimization submodule.

Images