Knowledge distillation method based on teacher model output reconstruction

Abstract

The invention relates to the field of machine learning, in particular to a knowledge distillation method based on teacher model output reconstruction, which comprises the following steps of: preprocessing a data set to obtain a preprocessed data set capable of being directly input into a teacher model, inputting the teacher model and a student model, respectively obtaining classification results corresponding to the teacher model and the student model and outputting logit, and calculating the output logit of the teacher model and the student model; obtaining the probability distribution of the output logit after the teacher model is corrected, the probability distribution of the non-target category, and the probability distribution of the output logit of the student model; and performing knowledge distillation based on the probability distribution of the output logit corrected by the teacher model, the probability distribution of the non-target category and the probability distribution of the output logit of the student model, and training the student model to obtain the required student model. According to the method, the output probability distribution of the teacher model can be reconstructed, the knowledge quality is improved, the knowledge type is perfected, the student model learning is more accurate, the model compression performance is optimized, and the generalization ability of the student model is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a knowledge distillation method based on reconstruction of teacher model output. Background Art

[0002] Deep learning has achieved remarkable performance in computer vision tasks such as image classification, object detection, and semantic segmentation. With the continuous exploration of the performance and capacity of deep neural network models, the scale and capacity of neural network models continue to increase, and their application scenarios are correspondingly limited, and the computing cost is also increasing. Smart mobile devices generally have limited computing power, insufficient storage capacity, and difficulty in deploying high-performance large-scale deep neural network models, which limits the development of deep learning on smart mobile devices. Studies have found that by compressing deep neural network models, the model size can be reduced while retaining the model performance as much as possible. Among them, knowledge distillation is one of the hot research directions for deep neural network model compression. It aims to train a lightweight student model through supervision by a larger teacher model, and transfer the knowledge of the better-performing teacher model to the smaller student model, thereby achieving compression of the deep neural network model.

[0003] In existing knowledge distillation, the teacher model generally has an error rate higher than 20%. The errors output by the teacher model will be learned by the student model, resulting in a decline in the performance of the student model, which in turn affects the performance of the model compression. It can be seen that how to correct the classification errors of the teacher model, find out the misclassified samples and reconstruct the probability distribution of the classification is of great significance for enhancing the generalization ability of the student model and improving the model compression performance. Summary of the invention

[0004] In order to avoid the classification errors in the teacher model affecting the learning performance of the student model and degrading the compression performance of the model, and to improve the quality of knowledge transfer from the teacher model to the student model, the present invention provides a knowledge distillation method based on teacher model output reconstruction, the method comprising the following steps: Preprocessing the data set to obtain a preprocessed data set that can be directly input into the teacher model; inputting the preprocessed data set into the teacher model and the student model respectively, and obtaining the classification results and output logits corresponding to the teacher model and the student model respectively; Based on the classification results and output logit corresponding to the teacher model, the probability distribution of the output logit corrected by the teacher model and the probability distribution of non-target categories are obtained; Based on the classification result and output logit corresponding to the student model, obtain the probability distribution of the output logit of the student model; Based on the probability distribution of the corrected output logit of the teacher model and the probability distribution of non-target categories, and the probability distribution of the output logit of the student model, knowledge distillation is performed to train the student model, thereby obtaining the desired student model.

[0005] Preferably, the data set is preprocessed to obtain a preprocessed data set that can be directly input into the teacher model, specifically: After converting all image samples of the dataset into tensor form, normalization preprocessing, random cropping preprocessing, and random flipping preprocessing are performed on all image samples of the dataset to obtain a preprocessed dataset that can be directly input into the teacher model.

[0006] Preferably, after preprocessing the data set to obtain a preprocessed data set that can be directly input into the teacher model, the method further includes: Based on the scale difference and architecture property requirements between the teacher model and the student model, neural network models with the same architecture but different scales or different architectures but different scales are selected as the teacher model and the student model respectively.

[0007] Preferably, the preprocessed data set is input into the teacher model and the student model respectively, and the classification results and output logits corresponding to the teacher model and the student model are obtained respectively, specifically: Input the preprocessed data set into the trained teacher model to obtain the classification result of the trained teacher model and the penultimate layer output logit; The preprocessed data set is input into the student model for training, and the stochastic gradient descent method is used as the optimizer during the training process, and the learning rate decay method is used when the preset training number conditions are met; after each training of the student model is completed, the classification result and output logit of the student model are obtained.

[0008] Preferably, the probability distribution of the output logit of the teacher model after correction is obtained based on the classification result and output logit corresponding to the teacher model, specifically: Based on the output logit corresponding to the teacher model, determine the output probability distribution of the teacher model; based on the output probability distribution and the true label of the data set, determine the classification error samples of the classification result corresponding to the teacher model; Based on the misclassified samples, the output logit is corrected; the corrected output logit is processed using a softmax function to obtain a probability distribution of the corrected output logit.

[0009] Preferably, the output logit is corrected based on the misclassified sample, specifically: The output logit corresponding to the misclassified sample is obtained, and a corresponding gain is added to the output logit corresponding to the misclassified sample, so as to correct the output logit.

[0010] Preferably, the non-target category probability distribution of the teacher model is obtained based on the classification result and output logit corresponding to the teacher model, specifically: According to the real label of the data set, the value corresponding to the output logit is determined; the value corresponding to the output logit is reduced and adjusted so that the classification probability corresponding to the output logit after the adjustment is reduced by the softmax function is zero, so that the probability distribution of the classification result only retains the non-target category, thereby obtaining the non-target category probability distribution of the teacher model.

[0011] Preferably, the probability distribution of the output logit of the student model is obtained based on the classification result and output logit corresponding to the student model, specifically: Based on the true label of the data set, the classification result corresponding to the student model is marked; Based on the output logit corresponding to the student model and the true label of the data set, the classification result corresponding to the student model is processed to obtain the probability distribution of the output logit of the student model.

[0012] Preferably, the probability distribution of the output logit and the probability distribution of non-target categories after correction based on the teacher model and the probability distribution of the output logit of the student model are subjected to knowledge distillation, specifically: The probability distribution of the corrected output logit of the teacher model and the probability distribution of the output logit of the student model, as well as the probability distribution of the non-target category and the probability distribution of the output logit of the student model are respectively calculated by KL divergence to obtain the cross entropy loss and the knowledge distillation loss; Based on the cross entropy loss and the knowledge distillation loss, the total training loss of the student model is obtained; and based on the total training loss, knowledge distillation is performed on the data set.

[0013] Preferably, the student model is trained to obtain the required student model, specifically: Based on the knowledge distillation result of the data set, the student model is trained again until the student model meets the preset learning performance conditions, thereby obtaining the student model.

[0014] Compared with the prior art, the present invention has the following beneficial effects: Preprocess the data set to obtain a preprocessed data set that can be directly input into the teacher model; input the preprocessed data set into the teacher model and the student model respectively, and obtain the classification results and output logit corresponding to the teacher model and the student model respectively. Preprocess the data set and convert the data set into a preprocessed data set that matches the teacher model, so that it is convenient to directly use the preprocessed data set to train the teacher model and the student model, and improve the model applicability of the preprocessed data set. Input the preprocessed data set into the teacher model and the student model respectively, and obtain the classification results and output logit corresponding to the teacher model and the student model respectively. In this way, the knowledge learning of the teacher model and the student model can be effectively and comprehensively compared, providing a reliable basis for the teacher model to transfer knowledge to the student model.

[0015] Based on the classification results and output logit corresponding to the teacher model, the probability distribution of the corrected output logit of the teacher model and the probability distribution of non-target categories are obtained; based on the classification results and output logit corresponding to the student model, the probability distribution of the output logit of the student model is obtained. Obtaining the probability distribution of the corrected output logit of the teacher model and the probability distribution of non-target categories facilitates the use of probability reconstruction technology to correct the classification errors of the teacher model, so that the knowledge transferred from the teacher model to the student model has higher quality and ensures that the student model is not limited to the errors of the teacher model. Obtaining the probability distribution of the output logit of the student model can serve as a basis for subsequently retaining the non-target category relationship of the teacher model outside the correctly classified category. The non-target category relationship represents the learning of the teacher model on the overall distribution of the data set. By judging the non-target category, the teacher model can have a deeper understanding of the data set, which is conducive to enhancing the generalization ability of the student model.

[0016] Based on the probability distribution of the corrected output logit of the teacher model and the probability distribution of non-target categories, and the probability distribution of the output logit of the student model, knowledge distillation is performed to train the student model to obtain the required student model. By inserting the teacher model probability reconstruction knowledge distillation between the teacher model and the student model, the output probability distribution of the teacher model is reconstructed to improve the knowledge quality, improve the knowledge type, make the student model learn more accurately, and optimize the model compression performance.

[0017] In general, the knowledge distillation method based on teacher model output reconstruction of the present invention has the following advantages: first, the quality of knowledge distillation is improved by correcting the incorrectly classified parts of the teacher model, and the confidence is improved by using the gain in the correction process, so as to construct a new loss function and add it to the overall loss until the training process of the student model; second, the teacher model's understanding of non-target categories is retained in the correction process, and the student model learns the relationship between non-target categories in the probability distribution of the teacher model output, thereby enhancing the generalization ability of the student model and improving the performance of the overall knowledge distillation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 It is a flow chart of the knowledge distillation method based on teacher model output reconstruction provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention rather than all structures are shown in the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0020] The terms "include" and "have" and any variations thereof in the present invention are intended to cover non-exclusive inclusions. For example, a process, method, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0021] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0022] See also Figure 1 As shown, the present invention provides a knowledge distillation method based on teacher model output reconstruction, the method comprising the following steps: S100, preprocessing the data set to obtain a preprocessed data set that can be directly input into the teacher model; inputting the preprocessed data set into the teacher model and the student model respectively, and obtaining the classification results and output logit corresponding to the teacher model and the student model respectively.

[0023] Furthermore, the data set is preprocessed to obtain a preprocessed data set that can be directly input into the teacher model, specifically: After converting all image samples of the dataset into tensor form, normalization preprocessing, random cropping preprocessing, and random flipping preprocessing are performed on all image samples of the dataset to obtain a preprocessed dataset that can be directly input into the teacher model.

[0024] Furthermore, after preprocessing the data set to obtain a preprocessed data set that can be directly input into the teacher model, it also includes: Based on the scale difference and architecture property requirements between the teacher model and the student model, neural network models with the same architecture but different scales or different architectures but different scales are selected as the teacher model and the student model respectively.

[0025] Furthermore, the preprocessed data set is input into the teacher model and the student model respectively, and the classification results and output logit corresponding to the teacher model and the student model are obtained respectively, specifically: Input the preprocessed data set into the trained teacher model to obtain the classification results of the trained teacher model and the penultimate layer output logit; The preprocessed data set is input into the student model for training, and the stochastic gradient descent method is used as the optimizer during the training process, and the learning rate decay method is used when the preset training times are met; after each training of the student model, the classification result and output logit of the student model are obtained.

[0026] In actual operation, all image samples in the data set are first converted into tensor form, and then all image samples in tensor form are normalized, randomly cropped, and randomly flipped. This can achieve the diversification of image samples and provide sufficient and reliable data samples for subsequent teacher model and student model training. Among them, the data set can be selected from any existing CIFAR10, CIFAR100, ImageNet and other data sets; the above-mentioned normalization preprocessing, random cropping preprocessing, and random flipping preprocessing of image samples can use PyTorch's own preprocessing functions to perform corresponding preprocessing on image samples.

[0027] In knowledge distillation, the selection of teacher model and student model is very important. The teacher model is usually a large-scale deep neural network model, and the student model is usually a small-scale deep neural network model. The performance of the teacher model and the performance of the student model do not always maintain a nearly linear relationship. When the scale of the teacher model is much larger than that of the student model, the capacity of the student model cannot accept the knowledge from the teacher model, and the performance of the trained student model may even be worse than that of the student model trained by a teacher model with lower performance. Therefore, when selecting the teacher model and the student model, the scale difference between the two should be considered to avoid poor knowledge distillation effect due to the large scale difference between the two. In addition, the architecture structure of the teacher model and the student model is also a very important influencing factor. Common model architectures mainly include ResNet, WideResNet, VGG, ShuffleNet, MobileNet, etc. The different architecture structures of the teacher model and the student model will also affect the effect of knowledge distillation. Therefore, based on the scale difference and architecture structure attribute requirements between the teacher model and the student model, neural network models with the same architecture structure but different scales or different architecture structures but different scales are adaptively selected as the teacher model and the student model respectively to meet the needs of different knowledge distillation occasions. The architecture structure attribute requirements can be but are not limited to architecture structure type requirements.

[0028] The preprocessed data set is input into the trained teacher model to obtain the classification results of the trained teacher model and the penultimate output logit, which can comprehensively and accurately characterize the knowledge classification of the teacher model. The preprocessed data set is also input into the student model for training, and the stochastic gradient descent method is used as the optimizer and the learning rate decay method is used during the training process. The learning rate is reduced to 10% of the original when the training cycle is 150, 180 and 210 times, respectively. At the same time, momentum optimization is used for the learning rate to accelerate the training process, prevent overfitting, and make the training process proceed in the direction of minimizing the loss value. After each training of the student model, the classification results and output logit of the student model are obtained, which can improve the training effect of the student model.

[0029] S200, based on the classification result and output logit corresponding to the teacher model, obtain the probability distribution of the output logit corrected by the teacher model and the probability distribution of the non-target category; Based on the classification results and output logit corresponding to the student model, the probability distribution of the output logit of the student model is obtained.

[0030] Furthermore, based on the classification results and output logit corresponding to the teacher model, the probability distribution of the output logit after correction of the teacher model is obtained, specifically: Based on the output logit corresponding to the teacher model, determine the output probability distribution of the teacher model; based on the output probability distribution and the true label of the data set, determine the classification error samples of the classification results corresponding to the teacher model; Based on the misclassified samples, the output logit is corrected; the corrected output logit is processed using the softmax function to obtain the probability distribution of the corrected output logit.

[0031] Furthermore, based on the misclassified samples, the output logit is corrected as follows: Get the output logit corresponding to the misclassified sample, and add the corresponding gain to the output logit corresponding to the misclassified sample, so as to correct the output logit.

[0032] Furthermore, based on the classification results and output logit corresponding to the teacher model, the probability distribution of non-target categories of the teacher model is obtained, specifically: According to the true label of the data set, the value corresponding to the output logit is determined; the value corresponding to the output logit is adjusted to be reduced so that the classification probability corresponding to the output logit after the adjustment is reduced by the softmax function is zero, so that the probability distribution of the classification result only retains the non-target category, thereby obtaining the probability distribution of the non-target category of the teacher model.

[0033] Furthermore, based on the classification results and output logit corresponding to the student model, the probability distribution of the output logit of the student model is obtained, specifically: Based on the true labels of the data set, the classification results corresponding to the student model are identified; Based on the output logit corresponding to the student model and the true label of the data set, the classification result corresponding to the student model is processed to obtain the probability distribution of the output logit of the student model.

[0034] In actual operation, knowledge distillation is embedded in the training process of the teacher model and the student model, the output probability distribution of the teacher model and the student model is obtained and the corresponding distillation loss is calculated until the training is carried out. Specifically, the output logit and probability distribution of the teacher model and the student model are obtained in each round of training. Given a dataset D, x is a sample in the dataset D, and y is the one-hot true label of sample x. The teacher model is represented as , the student model is expressed as At the same time, the penultimate layer output logit of the deep learning neural network is defined as ,in represents the output logit of the mth category, and C represents the total number of categories; then and Represents the teacher model and student model for the i-th sample The output logit is In this way, based on the output logit corresponding to the teacher model, the output probability distribution of the teacher model is determined; based on the output probability distribution and the true label of the data set, the misclassified samples of the classification results corresponding to the teacher model are determined.

[0035] Then, based on the misclassified samples, the output logit is corrected. Set the probability distribution of the model output ,in Represents the probability of the mth category, and uses the softmax function to soften the modified output logit to obtain the probability distribution of the modified output logit, the formula of which is as follows:

[0036] In the above formula, T represents the hyperparameter temperature used to control the degree of label softening. When T is greater than 1, the softmax function softens the label, making the entire probability distribution more uniform and reflecting the relationship between categories.

[0037] In addition, the error correction of the teacher model classification results can improve the quality of knowledge transfer from the teacher model to the student model and improve the performance of the student model, thereby achieving better knowledge distillation effect. Specifically, according to the real labels provided by the dataset and the categories judged by the teacher model, find the samples that the teacher model misclassified and their corresponding output logit. According to the gain G, the output logit corresponding to the misclassified samples is corrected, which is specifically ;in, Indicates the output logit corresponding to the misclassified sample, Represents the corrected output logit. By adding the gain G, the confidence of the correct category can be increased to achieve the purpose of correcting the error.

[0038] For the retention of the relationship between non-target categories, the relationship between the remaining categories can enhance the generalization ability of the model and improve the problem of overconfidence and mismatch between accuracy caused by the use of hard labels. Specifically, in the output probability distribution of the model, the category with the highest probability accounts for a large proportion. In order to highlight the relationship between the remaining categories more, the probability of the target category can be suppressed and "set to zero", and only the probability of the remaining categories is retained, thereby providing more information for the student model. To this end, the value corresponding to the output logit is determined according to the true label of the data set; the value corresponding to the output logit is reduced and adjusted so that the classification probability corresponding to the output logit after the adjustment is reduced by the softmax function is zero, so that the probability distribution of the classification result only retains the non-target category, thereby obtaining the probability distribution of the non-target category of the teacher model.

[0039] S300, based on the probability distribution of the corrected output logit of the teacher model and the probability distribution of the non-target category, and the probability distribution of the output logit of the student model, knowledge distillation is performed to train the student model, thereby obtaining the desired student model.

[0040] Furthermore, knowledge distillation is performed based on the probability distribution of the corrected output logit of the teacher model and the probability distribution of the non-target category, and the probability distribution of the output logit of the student model, as follows: The probability distribution of the corrected output logit of the teacher model and the probability distribution of the output logit of the student model, as well as the probability distribution of the non-target category and the probability distribution of the output logit of the student model are respectively calculated by KL divergence to obtain the cross entropy loss and knowledge distillation loss; Based on the cross entropy loss and knowledge distillation loss, the total training loss of the student model is obtained; then based on the total training loss, knowledge distillation is performed on the dataset.

[0041] Furthermore, the student model is trained to obtain the desired student model, specifically: Based on the knowledge distillation results of the data set, the student model is trained again until the student model meets the preset learning performance conditions, thereby obtaining the student model.

[0042] The probability distribution of the corrected output logit of the teacher model and the probability distribution of the output logit of the student model, as well as the probability distribution of the non-target category and the probability distribution of the output logit of the student model are respectively calculated by KL divergence to obtain the cross entropy loss LCE and the knowledge distillation loss Lours. The total training loss Ltotal of the student model is obtained using the following formula:

[0043] In the above formula, represents the equilibrium parameter.

[0044] Based on the total training loss, knowledge distillation is performed on the data set; based on the knowledge distillation results of the data set, the student model is trained again until the student model meets the preset learning performance conditions, thereby obtaining the student model; wherein the preset learning performance conditions may be but are not limited to that the actual learning rate of the student model exceeds the preset learning rate threshold.

[0045] Through the description of the above implementation methods, technicians in this field can clearly understand that each implementation method can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it, and other embodiments may also be used. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. The knowledge distillation method based on teacher model output reconstruction is characterized by: The method comprises the following steps: Preprocessing the data set to obtain a preprocessed data set that can be directly input into the teacher model; inputting the preprocessed data set into the teacher model and the student model respectively, and obtaining the classification results and output logits corresponding to the teacher model and the student model respectively; Based on the classification results and output logit corresponding to the teacher model, the probability distribution of the output logit corrected by the teacher model and the probability distribution of non-target categories are obtained; Based on the classification result and output logit corresponding to the student model, obtain the probability distribution of the output logit of the student model; Based on the probability distribution of the corrected output logit of the teacher model and the probability distribution of non-target categories, and the probability distribution of the output logit of the student model, knowledge distillation is performed to train the student model, thereby obtaining the desired student model.

2. The method according to claim 1, characterized in that The data set is preprocessed to obtain a preprocessed data set that can be directly input into the teacher model, specifically: After converting all image samples of the dataset into tensor form, normalization preprocessing, random cropping preprocessing, and random flipping preprocessing are performed on all image samples of the dataset to obtain a preprocessed dataset that can be directly input into the teacher model.

3. The method according to claim 2, characterized in that After preprocessing the data set to obtain a preprocessed data set that can be directly input into the teacher model, it also includes: Based on the scale difference and architecture property requirements between the teacher model and the student model, neural network models with the same architecture but different scales or different architectures but different scales are selected as the teacher model and the student model respectively.

4. The method according to claim 3, characterized in that The preprocessed data set is input into the teacher model and the student model respectively, and the classification results and output logits corresponding to the teacher model and the student model are obtained respectively, specifically: Input the preprocessed data set into the trained teacher model to obtain the classification result of the trained teacher model and the penultimate layer output logit; Inputting the preprocessed data set into the student model for training, and using the stochastic gradient descent method as an optimizer during the training process and using the learning rate decay method when the preset training times are met; After each training of the student model is completed, the classification result and output logit of the student model are obtained.

5. The method according to claim 1, characterized in that The probability distribution of the output logit after the teacher model is corrected is obtained based on the classification result and output logit corresponding to the teacher model, specifically: Based on the output logit corresponding to the teacher model, determine the output probability distribution of the teacher model; based on the output probability distribution and the true label of the data set, determine the classification error samples of the classification result corresponding to the teacher model; Based on the misclassified samples, the output logit is corrected; the corrected output logit is processed using a softmax function to obtain a probability distribution of the corrected output logit.

6. The method according to claim 5, characterized in that The output logit is corrected based on the misclassified sample, specifically: The output logit corresponding to the misclassified sample is obtained, and a corresponding gain is added to the output logit corresponding to the misclassified sample, so as to correct the output logit.

7. The method according to claim 1, characterized in that The classification result and output logit corresponding to the teacher model are used to obtain the probability distribution of the non-target category of the teacher model, specifically: According to the real label of the data set, the value corresponding to the output logit is determined; the value corresponding to the output logit is reduced and adjusted so that the classification probability corresponding to the output logit after the adjustment is reduced by the softmax function is zero, so that the probability distribution of the classification result only retains the non-target category, thereby obtaining the non-target category probability distribution of the teacher model.

8. The method according to claim 1, characterized in that: The probability distribution of the output logit of the student model is obtained based on the classification result and output logit corresponding to the student model, specifically: Based on the true label of the data set, the classification result corresponding to the student model is marked; Based on the output logit corresponding to the student model and the true label of the data set, the classification result corresponding to the student model is processed to obtain the probability distribution of the output logit of the student model.

9. The method according to claim 1, characterized in that: The probability distribution of the output logit and the probability distribution of non-target categories based on the modified output logit of the teacher model and the probability distribution of the output logit of the student model are subjected to knowledge distillation, specifically: The probability distribution of the corrected output logit of the teacher model and the probability distribution of the output logit of the student model, as well as the probability distribution of the non-target category and the probability distribution of the output logit of the student model are respectively calculated by KL divergence to obtain the cross entropy loss and the knowledge distillation loss; Based on the cross entropy loss and the knowledge distillation loss, the total training loss of the student model is obtained; and based on the total training loss, knowledge distillation is performed on the data set.

10. The method according to claim 9, characterized in that The student model is trained to obtain the required student model, specifically: Based on the knowledge distillation result of the data set, the student model is trained again until the student model meets the preset learning performance conditions, thereby obtaining the student model.

Images