Pretreatment method for improving Logit knowledge distillation

Abstract

The invention discloses a preprocessing method for improving Logit knowledge distillation, and relates to the technical field of computer vision and deep learning. The invention aims to solve key problems existing in traditional knowledge distillation. In traditional knowledge distillation, when a soft label is migrated from a teacher model to a student model at a fixed temperature, learning weights of students for each sample are the same, which greatly restricts further improvement of student model performance. Aiming at the situation, firstly, the output of the teacher model and the output of the student model are processed, the value of a high-score part is increased, the value of a low-score part is reduced, and a two-way stretching effect is formed. And then converting the data into standard normal distribution by utilizing a Z-score standardization mode according to the mean value and the standard deviation of the data, thereby realizing accurate matching of output of the teacher model and the student model so as to promote optimization of performance of the student model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and specifically to a preprocessing method for improving Logit knowledge distillation. Background Art

[0002] In the past, the booming development of deep neural networks (DNNs) has brought about earth-shaking changes in the field of computer vision. As the performance and capacity of deep neural networks continue to improve, their model size and computational costs have also risen. Although large models often mean larger capacity, researchers have always been committed to compressing models without losing too much accuracy. In addition to building lightweight models, knowledge distillation (KD) has become an emerging way to achieve this goal. Specifically, it is to transfer the knowledge of a pre-trained large model (i.e., the teacher network) to a small target model (i.e., the student network).

[0003] Traditional knowledge distillation first proposed to distill the knowledge of the teacher network into the student network by minimizing the Kullback-Leibler (KL) divergence between the prediction results of the two. In this process, the temperature T of the softmax function is introduced to soften the prediction probability. Usually, the temperature is a hyperparameter set globally in advance and remains constant throughout the training process. CTKD uses an adversarial learning module to predict the sample temperature to adapt to the difficulty differences of different samples. However, existing Logit-based KD methods generally assume that the teacher and the student use the same temperature, ignoring the possibility of different temperature values ​​in the KL divergence.

[0004] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a preprocessing method for improving Logit knowledge distillation to solve the technical problems raised in the background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a preprocessing method for improving Logit knowledge distillation, comprising at least the following steps:

[0007] S1: Input the RGB image into the teacher model and the student model respectively to extract features;

[0008] S2: Input the features extracted by the teacher model and the student model into the corresponding classifiers to obtain the output results;

[0009] S3: Based on the output of S2, i.e. the sample score logit, for the part with higher scores in the teacher-student model, increase its score according to the 10-percentile range of the sample score, and for the part with lower scores in the teacher-student model, reduce its score according to the 10-percentile range of the sample score;

[0010] S4: The results of S3 are processed using the Z-score standardization method, converting the mean of the data distribution to 0 and the standard deviation to 1;

[0011] S5: The standardized score Logit distribution is softened using oftmax-T, and the loss is calculated using KL divergence;

[0012] S6: Get the true label and use the mean square error formula to calculate the loss between the output of the student model in S2 and the true label.

[0013] Furthermore, the teacher model and the student model in S1 both use convolutional neural networks as the backbone networks. The teacher model and the student model respectively extract features of RGB images, and the feature maps are reduced layer by layer with the depth of the network.

[0014] Furthermore, the S3 at least includes the following steps:

[0015] First, sort the sample score Logit data from small to large, and get the sorted sample score x (1) ≤x (2) ≤…≤x (n) , calculate the i-th quantile P i and the j-th quantile P j , where 0≤i≤j≤n, the formula is as follows:

[0016] P i =x (i) (1)

[0017] P j =x (j) (2)

[0018] IDR=P j -P i (3)

[0019] Among them, x (i) represents the score of the i-th sample, x (j) represents the score of the jth sample, and IDR represents the quantile range;

[0020] Then, let the original score be x origin , the improved score is y i , for scores higher than P j The partial improvement score, x origin >Pj When , as shown in calculation formula (4);

[0021] y i =x origin +k×IDR (4)

[0022] For scores below P i The part of the score reduction, x origin <P j When , as shown in formula (5);

[0023] y i =x origin -k×IDR (5)

[0024] For scores in P i With P j For the parts in between, the score remains the same;

[0025] Where k represents the improvement factor.

[0026] Furthermore, S4 processes the result of S3 using a Z-score standardization method, and the calculation formula is shown in formula (6):

[0027]

[0028] Among them, μ represents the mean and σ represents the standard deviation;

[0029] The mean value and standard deviation are calculated as shown in formula (7) and formula (8):

[0030]

[0031]

[0032] Here, n represents the number of samples.

[0033] Furthermore, the S5 at least includes the following steps:

[0034] The result of the teacher model after Z-score standardization is recorded as Z(t), and the result of the student model after Z-score standardization is recorded as Z(s). Z(t) and z(s) are softened using the softmax-T mathematical formula respectively. The calculation process is shown in formula (9):

[0035]

[0036] Among them, z j (0≤j≤n) represents the jth sample score, z k (0≤k≤n) represents the kth sample score, t represents the temperature, and e represents the constant;

[0037] The softened teacher is denoted as P T , the student is denoted by Q S , and use KL divergence to calculate the loss Loss of the teacher and student output dis , as shown in formula (10):

[0038]

[0039] Among them, j (0≤j≤n) represents the jth sample.

[0040] Further, the S6 at least includes the following steps:

[0041] Let x i is the predicted value of the student network model, t i is the true label value, as shown in formula (11):

[0042]

[0043] Where n represents the number of samples;

[0044] The total loss is shown in formula (12):

[0045] Loss all =α×Loss dis +β×Loss stu (12)

[0046] Among them, α and β are weight coefficients.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention aims to solve the key problems existing in traditional knowledge distillation. In traditional knowledge distillation, when a fixed temperature is used to migrate soft labels from the teacher model to the student model, the student's learning weights for each sample are the same, which greatly limits the further improvement of the student model's performance. In response to this situation, the present invention first processes the outputs of the teacher model and the student model, increases the value of the higher score part and reduces the value of the lower score part, forming a two-way stretching effect. Then, using the Z-score normalization method, the data is converted into a standard normal distribution based on the mean and standard deviation of the data, thereby achieving an accurate match between the outputs of the teacher model and the student model to promote the optimization of the student model's performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0052] See also Figure 1 , a preprocessing method for improving Logit knowledge distillation, comprising at least the following steps:

[0053] S1: Input the RGB image into the teacher model and the student model respectively to extract features, that is, in the feature extraction stage, input the RGB image into the teacher model and the student model to obtain the output result Logit of the model. Then, the Logit is bidirectionally stretched according to the 10-quantile distance, so that the model can focus more on high-scoring samples;

[0054] In S1, both the teacher model and the student model use convolutional neural networks as the backbone network. The teacher model and the student model extract the features of RGB images respectively, and the feature maps shrink layer by layer with the depth of the network.

[0055] S2: Input the features extracted by the teacher model and the student model into the corresponding classifiers to obtain the output results;

[0056] S3: Based on the output of S2, i.e. the sample score logit, for the part with higher scores in the teacher-student model, increase its score according to the 10-percentile range of the sample score, and for the part with lower scores in the teacher-student model, reduce its score according to the 10-percentile range of the sample score;

[0057] S3 includes at least the following steps:

[0058] First, sort the sample score Logit data from small to large, and get the sorted sample score x (1) ≤x (2) ≤…≤x (n) , calculate the i-th quantile P i and the j-th quantile P j , where 0≤i≤j≤n, the formula is as follows:

[0059] Pi =x (i) (1)

[0060] P j =x (j) (2)

[0061] IDR=P j -P i (3)

[0062] Among them, x (i) represents the score of the i-th sample, x (j) represents the score of the jth sample, and IDR represents the quantile range;

[0063] Then, let the original score be x origin , the improved score is y i , for scores higher than P j The partial improvement score, x origin >P j When , as shown in calculation formula (4);

[0064] y i =x origin +k×IDR (4)

[0065] For scores below P i The part of the score reduction, x origin <P j When , as shown in formula (5);

[0066] y i =x origin -k×IDR (5)

[0067] For scores in P i With P j For the parts in between, the score remains the same;

[0068] Where k represents the improvement factor.

[0069] S4: The results of S3 are processed using the Z-score standardization method, converting the mean of the data distribution to 0 and the standard deviation to 1;

[0070] S4 uses the Z-score standardization method to process the results of S3. The calculation formula is shown in formula (6):

[0071]

[0072] Among them, μ represents the mean and σ represents the standard deviation;

[0073] The mean value and standard deviation are calculated as shown in formula (7) and formula (8):

[0074]

[0075]

[0076] Here, n represents the number of samples.

[0077] S5: The standardized score Logit distribution is softened using oftmax-T, and the loss is calculated using KL divergence;

[0078] S5 at least includes the following steps:

[0079] The result of the teacher model after Z-score standardization is recorded as Z(t), and the result of the student model after Z-score standardization is recorded as Z(s). Z(t) and z(s) are softened using the softmax-T mathematical formula respectively. The calculation process is shown in formula (9):

[0080]

[0081] Among them, z j (0≤j≤n) represents the jth sample score, z k (0≤k≤n) represents the kth sample score, t represents the temperature, and e represents the constant;

[0082] The softened teacher is denoted as P T , the student is denoted by Q S , and use KL divergence to calculate the loss Loss of the teacher and student output dis , as shown in formula (10):

[0083]

[0084] Among them, j (0≤j≤n) represents the jth sample.

[0085] S6: Get the true label and use the mean square error formula to calculate the loss between the output of the student model in S2 and the true label.

[0086] S6 at least comprises the following steps:

[0087] Let x i is the predicted value of the student network model, t i is the true label value, as shown in formula (11):

[0088]

[0089] Where n represents the number of samples;

[0090] The total loss is shown in formula (12):

[0091] Loss all =α×Loss dis +β×Loss stu (12)

[0092] Among them, α and β are weight coefficients.

[0093] In summary:

[0094] The present invention first bidirectionally stretches the Logit according to the 10-quantile range, and then uses Z-score standardization to map the Logit to the same space, and weights this operation on the temperature coefficient, so that the student model pays more attention to high-scoring samples.

[0095] During the research, the inventors found that in traditional knowledge distillation, when a fixed temperature is used to transfer soft labels from the teacher model to the student model, the student's learning weights for each sample are the same, which greatly limits the further improvement of the student model performance. Therefore, the following concept is adopted: First, based on the traditional knowledge distillation KD method, in order to address the problem of large differences in learning abilities between the teacher network model and the student network model, the outputs of the teacher and student models are bidirectionally stretched, so that the student model pays more attention to high-scoring samples during training. Subsequently, Z-score normalization is used to map the outputs to the same space for easy comparison and learning, so as to improve the generalization of the student model.

[0096] The references to this case are as follows:

[0097] References (such as patents / papers / standards) are listed below:

[0098] [1]Hinton EG, Vinyals O, Dean J. Distilling the Knowledge in aNeuralNetwork.[J].CoRR,2015,abs / 1503.02531.

[0099] [2]Hu, C., Li, X., Liu, D., Chen, X., Wang, J., Liu,

[0100] [3]Gou,J.,Yu,B.,Maybank,S.J.,Tao,D.:Knowledge distillation:Asurvey.International Journal ofComputerVision 129(6),1789–1819(2021).

[0101] [4]Chen,P.,Liu,S.,Zhao,H.,Jia,J.:Distilling knowledge via knowledgereview.In:Proceedings ofthe IEEE / CVF Conference on ComputerVision andPatternRecognition,pp.5008–5017(2021).

[0102] [5]Hu,C.,Li,X.,Liu,D.,Chen,X.,Wang,J.,Liu,X.:Teacher-studentarchitecturefor knowledge learning:A survey.arXiv preprint arXiv:2210.17332(2022)

[0103] [6]Ahn,S.,Hu,S.X.,Damianou,A.,Lawrence,N.D.,Dai,Z.:Variationalinformationdistillation for knowledge transfer.In:Proceedings ofthe IEEE / CVFConferenceon ComputerVision and Pattern Recognition,pp.9163–9171(2019).

[0104] [7]Chen,D.,Mei,J.-P.,Zhang,H.,Wang,C.,Feng,Y.,Chen,C.:Knowledge dis-tillation with the reused teacher classifier.In:Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,pp.11933–11942(2022).

[0105] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A preprocessing method for improving Logit knowledge distillation, characterized by: At least the following steps are included: S1: Input the RGB image into the teacher model and the student model respectively to extract features; S2: Input the features extracted by the teacher model and the student model into the corresponding classifiers to obtain the output results; S3: Based on the output of S2, i.e. the sample score logit, for the part with higher scores in the teacher-student model, increase its score according to the 10-percentile range of the sample score, and for the part with lower scores in the teacher-student model, reduce its score according to the 10-percentile range of the sample score; S4: The results of S3 are processed using the Z-score standardization method, converting the mean of the data distribution to 0 and the standard deviation to 1; S5: The standardized score Logit distribution is softened using oftmax-T, and the loss is calculated using KL divergence; S6: Get the true label and use the mean square error formula to calculate the loss between the output of the student model in S2 and the true label.

2. A preprocessing method for improving Logit knowledge distillation according to claim 1, characterized in that: The teacher model and the student model in S1 both use convolutional neural networks as the backbone networks. The teacher model and the student model extract features of RGB images respectively, and the feature maps are reduced layer by layer as the depth of the network increases.

3. A preprocessing method for improving Logit knowledge distillation according to claim 1, characterized in that: The S3 at least includes the following steps: First, sort the sample score Logit data from small to large, and get the sorted sample score x (1) ≤x (2) ≤···≤x (n) , calculate the i-th quantile P i and the j-th quantile P j , where 0≤i≤j≤n, the formula is as follows: P i =x (i) (1) P j =x (j) (2) IDR=P j -P i (3) Among them, x (i) represents the score of the i-th sample, x (j) represents the score of the jth sample, and IDR represents the quantile range; Then, let the original score be x origin , the improved score is y i , for scores higher than P j The partial improvement score, x origin >P j When , as shown in the calculation formula (4); and i =x origin +k×IDR (4) For scores below P i The part of the score reduction, x origin <P j When , as shown in formula (5); and i =x origin -k×IDR (5) For scores in P i With P j For the parts in between, the score remains the same; Where k represents the improvement factor.

4. A preprocessing method for improving Logit knowledge distillation according to claim 2, characterized in that: S4 uses the Z-score standardization method to process the results of S3, and the calculation formula is shown in formula (6): Among them, μ represents the mean and σ represents the standard deviation; The mean value and standard deviation are calculated as shown in formula (7) and formula (8): Here, n represents the number of samples.

5. A preprocessing method for improving Logit knowledge distillation according to claim 3, characterized in that: The S5 at least comprises the following steps: The result of the teacher model after Z-score standardization is recorded as Z(t), and the result of the student model after Z-score standardization is recorded as Z(s). Z(t) and Z(s) are softened using the softmax-T mathematical formula respectively. The calculation process is shown in formula (9): Among them, z j (0≤j≤n) represents the jth sample score, z k (0≤k≤n) represents the kth sample score, t represents the temperature, and e represents the constant; The softened teacher is denoted as P T , the student is denoted by Q S , and use KL divergence to calculate the loss Loss between the teacher and the student output dis , as shown in formula (10): Among them, j (0≤j≤n) represents the jth sample.

6. A preprocessing method for improving Logit knowledge distillation according to claim 5, characterized in that: The S6 at least comprises the following steps: set up x i is the predicted value of the student network model, t i is the true label value, as shown in formula (11): Where n represents the number of samples; The total loss is shown in formula (12): Loss all =α×Loss dis +β×Loss stu (12) Among them, α and β are weight coefficients.

Images