Self-enhanced dynamic knowledge distillation method
By introducing self-reinforcement technology and dynamic knowledge distillation methods in the learning process of student model, the problem of deploying large-scale convolutional neural network models in resource-constrained environments is solved, and the effect of model compression is achieved while maintaining model performance, which significantly improves the accuracy of student models.
Patent Information
- Application Number
- CN202510134835.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-07
- Publication Date
- 2025-04-29
AI Technical Summary
In resource-confined environments, such as embedded systems, mobile devices, or edge computing nodes, large convolutional neural network models are difficult to deploy, and knowledge distillation technology challenges model compression while maintaining model performance.
A self-reinforced dynamic knowledge distillation method is designed. By introducing self-reinforced technology in the learning process of student model, the distance calculation between feature maps is used to dynamically adjust the learning content, avoid knowledge forgetting, and improve the performance of student model.
Experiments on CIFAR-10 and SVHN datasets show that the self-reinforced dynamic knowledge distillation method can significantly improve the accuracy of student models, reaching 77.11% and 88.76%, providing strong support for model deployment in resource-constrained environments.
Smart Images
Figure CN119903908A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine learning, and more specifically to a model compression method that transfers knowledge in a teacher model to a student model through knowledge distillation to enhance student performance. Background Art
[0002] With the continuous improvement of computer hardware performance and the rapid development of deep learning algorithms, deep learning technology has become the mainstream method for processing various machine learning tasks such as image classification, object detection, and semantic segmentation. In terms of image classification, deep learning technology can accurately classify images into different categories in various complex scenarios. This achievement is attributed to the ability of convolutional neural networks to perform hierarchical feature extraction and gradually learn complex features and details in images, thereby achieving efficient recognition of object categories. However, this high level of abstraction and feature extraction process requires a large number of model parameters to ensure its accuracy. In addition, training large convolutional neural networks also requires powerful computing resources and sufficient storage space to support the complexity and scale of model training. However, in practical applications, especially in resource-constrained environments such as embedded systems, mobile devices, or edge computing nodes, large convolutional neural network models face deployment difficulties. Mobile devices have limited processing power, embedded systems have limited computing resources, and edge computing nodes need to process large amounts of real-time data in a distributed environment. These limitations require models to be optimized and compressed during deployment to adapt to limited computing resources and storage capacity.
[0003] Researchers have proposed a variety of techniques for model optimization. On the one hand, they are committed to developing lightweight model structures, such as deep separable convolutional networks, sparse convolutional networks, and compact neural networks, to improve the running efficiency of models on embedded and mobile devices by reducing the number of parameters and computational complexity in the model. On the other hand, researchers are also exploring a series of new algorithms and techniques to further optimize the model. For example, the model quantization technology is used to reduce the accuracy of numerical representation in the model to reduce the storage requirements and computing resource consumption of the model. The network pruning technology is used to remove unimportant neurons and connections to simplify the model structure. Although these techniques can simplify the model, they usually sacrifice the overall performance of the model. Knowledge distillation transfers the knowledge in a complex large model to a lightweight model. Therefore, it can compress the model parameters while maintaining the overall performance of the model. In addition, knowledge distillation can be applied to almost all types of network structures and can be seamlessly integrated with other optimization techniques, with good simplicity and practicality.
[0004] The present invention designs a self-reinforcement dynamic knowledge distillation method, which introduces self-reinforcement technology when the student model learns label knowledge, so that the student model can review what it has learned in time to avoid knowledge forgetting and further improve its performance.
[0005] The effectiveness of the self-reinforced dynamic knowledge distillation method was verified on the CIFAR-10 dataset and the SVHN dataset. The results show that under the teacher-student isomorphic experimental setting, the student model trained by the scheme in this paper achieved an accuracy of 77.11% on the CIFAR-10 dataset and 88.76% on the SVHN dataset. Therefore, the scheme invented in this paper shows excellent performance and provides strong support for model compression in various application scenarios. Summary of the invention
[0006] The present invention designs a self-reinforced dynamic knowledge distillation method. When the student model calculates the distance between the predicted result and the true label, the method also calculates the distance between the deep feature map and the shallow feature map, thereby avoiding forgetting important knowledge in the shallow features in the process of learning new knowledge and improving the learning performance of the student model.
[0007] A self-reinforced dynamic knowledge distillation method includes two parts: dynamic distillation of features and self-reinforcement of learning effects. (1) Dynamic distillation of features The feature map is the intermediate layer representation generated by the model when processing input data, which deeply reflects the model's ability to capture data features. By comparing the differences in feature maps between the student model and the teacher model, the feature extraction ability of the student model can be effectively evaluated. The present invention dynamically distills features: first, a loss threshold is set, and the distance between the feature graph of the student model and the feature graph of the teacher model is calculated. When the distance between the two is less than the set threshold, it means that the feature graph of the student model is close to that of the teacher model, and the feature extraction ability of the student model is close to that of the teacher model, and there is no need to continue learning the feature graph knowledge of the teacher model. When the distance between the two is greater than the set threshold, it means that the feature graph of the student model is far away from that of the teacher model, and the feature extraction ability of the student model is weak at this time, and it is necessary to continue learning the feature graph knowledge of the teacher model to further improve its feature extraction ability. (2) Self-reinforcement of learning outcomes Self-reinforcement technology is introduced when the student model learns label knowledge, so that the student model can review what it has learned in a timely manner, avoid knowledge forgetting, and further improve its performance. When the student model calculates the distance between the predicted result and the true label, it also calculates the distance between the deep feature map and the shallow feature map, thereby avoiding forgetting important knowledge in the shallow features in the process of learning new knowledge and self-reinforcing the learning effect. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 Comparison of distillation effects of the method of the present invention under different data sets; DETAILED DESCRIPTION
[0008] A self-reinforcing dynamic knowledge distillation method, characterized by: (1) Dynamic distillation of features For the student model S and the teacher model T (assuming that both the teacher model and the student model are divided into K blocks), the features are dynamically distilled: Among them, MSE stands for the mean square error loss function, which is used to measure the feature map extracted by the i-th block of the student model Feature maps generated by the teacher model The distance between Used to indicate whether the student model selects the i-th block for feature learning; when the distance between the feature map of the i-th block of the student model and the feature map of the i-th block of the teacher model is greater than the pre-set loss threshold (the loss threshold used in this method is 0.1), the flag is set to 1, and the student model will select the i-th block for feature learning; otherwise, the flag is set to 0, and the student model will not select the i-th block for feature learning; (2) Self-reinforcement of learning outcomes When the student model calculates the distance between the predicted result and the true label, it also calculates the distance between the deep feature map and the shallow feature map, so as to avoid forgetting important knowledge in the shallow features in the process of learning new knowledge and self-reinforce the learning effect: Among them, CE is the cross entropy loss function: ; represents the prediction result produced by the student model, and y represents the true label of the data; represents the feature map extracted by the i-th block of the student model, represents the feature map extracted by the jth block of the student model; MSE represents the mean square error loss function, which is used to measure the feature map extracted by the i-th block of the student model And the feature map extracted by the jth block The distance between. Validity Verification
[0009] The validity verification experiment of the present invention is as follows: (1) Experimental setup The present invention verifies the effectiveness of the scheme under the condition of teacher-student isomorphism on the CIFAR-10 dataset and the SVHN dataset. Teacher-student isomorphism means that the teacher model and the student model use the same type of network architecture. In the teacher-student isomorphism, the teacher model we use is AlexNet and the student model is Tiny-AlexNet. (2) Experimental results and analysis Combination Figure 1 It can be seen that under the teacher-student isomorphic experimental setting, the student model trained by this scheme achieves an accuracy of 77.11% on the CIFAR-10 dataset and 88.76% on the SVHN dataset. The scheme proposed in this paper shows excellent performance and provides strong support for model compression in various application scenarios.
Claims
1. A self-reinforcing dynamic knowledge distillation method, characterized by: (1) Dynamic distillation of features For the student model S and the teacher model T (assuming that both the teacher model and the student model are divided into K blocks), the features are dynamically distilled: in, MSE stands for mean square error loss function, which is used to measure the feature map extracted by the i-th block of the student model Feature maps generated by the teacher model The distance between Used to indicate whether the student model selects the i-th block for feature learning; the student When the distance between the feature map of the i-th block of the model and the feature map of the i-th block of the teacher model is greater than the pre-set loss threshold (the loss threshold used in the experiment of this method is 0.1), the flag is set to 1, and the student model will select the i-th block for feature learning; On the contrary, if the flag is set to 0, the student model will not select the i-th block for feature learning; (2) Self-reinforcement of learning outcomes When the student model calculates the distance between the predicted result and the true label, it also calculates the distance between the deep feature map and the shallow feature map, so as to avoid forgetting important knowledge in the shallow features in the process of learning new knowledge and self-reinforce the learning effect: Among them, CE is the cross entropy loss function: ; represents the prediction result produced by the student model, and y represents the true label of the data; represents the feature map extracted by the i-th block of the student model, represents the feature map extracted by the jth block of the student model; MSE represents the mean square error loss function, which is used to measure the feature map extracted by the i-th block of the student model And the feature map extracted by the jth block The distance between.