An image classification method based on information entropy dynamic adjustment of temperature knowledge distillation

Abstract

The present invention discloses an image classification method based on information entropy to dynamically adjust temperature knowledge distillation, comprising the following steps: obtaining a labeled image and constructing a target temperature fitting function; a residual convolution temperature prediction network outputs a dynamic distillation temperature while receiving supervised training of the target temperature fitting function; and utilizing the distillation temperature to soften the classification results of the teacher network and the classification results of the student network after logits fusion, thereby realizing knowledge distillation training and optimization. The present invention introduces information entropy as the basis for dynamically adjusting the temperature, and adaptively adjusts the distillation temperature according to the complexity of the sample. During the training process, the classification results and information entropy of the teacher network and the student network are used to generate the target temperature. The target temperature is input into the residual convolution temperature prediction network as a supervisory signal to optimize the image classification task, better cope with the complexity differences of image samples, and improve the classification accuracy and robustness of the model in image classification tasks.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically is an image classification method based on information entropy dynamic regulation of temperature knowledge distillation. Background Art

[0002] In recent years, with the rapid development of deep learning technology, image classification tasks have made significant progress. Convolutional neural networks (CNNs), as the leading method for image classification, have become a widely used foundational architecture in computer vision due to their powerful ability to capture spatial features. However, their rapidly growing demand for computing resources and storage space has severely impacted the deployment of these models in resource-constrained environments. For example, in smartphones, embedded devices, and edge computing scenarios, there is a significant conflict between the demand for real-time inference and the size and computational complexity of the models.

[0003] To address this issue, knowledge distillation has gained widespread attention as an effective method for model compression and acceleration. The basic idea of ​​knowledge distillation is to use a complex teacher model as a reference and transfer its rich knowledge to a simpler student model. In this way, the student model can significantly reduce computational and storage requirements while maintaining high accuracy. Although knowledge distillation technology has shown great potential in the field of model compression, most existing methods rely on static temperature parameters to control the degree of softening of the teacher model's prediction results. However, the static temperature setting ignores the dynamic changes in the samples themselves and the data distribution during training. This fixed temperature processing method is often difficult to adapt to the needs of different training stages, which in turn affects the distillation effect.

[0004] The main limitations of current knowledge distillation in image classification tasks are reflected in the following aspects:

[0005] 1. Static temperature settings ignore the complexity of different samples and training stages, which may cause the student model to miss key information during the learning process.

[0006] 2. Fixed temperature parameters may make it difficult to help the model converge effectively in the early stages of training, and may make the model too smooth in the later stages of training, unable to fully capture the subtle differences in the training data.

[0007] 3. The existing knowledge transfer process lacks sensitivity to the relative differences between target categories, which limits the model's ability to learn high-dimensional features.

[0008] Traditional knowledge distillation methods typically use a fixed temperature parameter to control the softening of the teacher network's output. However, this fixed temperature ignores variations in sample complexity and model uncertainty. For simple samples, a higher temperature may lead to unnecessary smoothing, compromising the guiding value of the student network. For complex samples, however, a fixed temperature may not be sufficient to effectively extract dark knowledge (i.e., information about low-confidence categories in the teacher network's output). Therefore, a fixed temperature setting cannot adapt to the dynamic demands of the training process across different samples, limiting the effectiveness of knowledge distillation.

[0009] To address the above issues, the present invention proposes an image classification method that utilizes a dynamic distillation temperature calculation method based on sample information entropy to address the static temperature setting problem in the knowledge distillation process, thereby improving image classification accuracy. Information entropy, as a measure of sample uncertainty, can be used to guide the calculation of dynamic temperature. By introducing sample information entropy, the model can adjust the distillation temperature based on the uncertainty of the student model output during training. Specifically, when the sample uncertainty is high, the model increases the temperature so that the student model can better learn details from the teacher model's soft targets; when the sample uncertainty is low, the temperature is correspondingly reduced, allowing the student model to focus more on learning clear classification boundaries.

[0010] The core concept of this invention is to enhance the learning ability of the student model during the knowledge distillation process by utilizing a dynamically adjusted temperature parameter. Unlike traditional methods, this invention automatically adjusts the distillation temperature based on sample information entropy, ensuring that the model can adaptively adjust its learning strategy at different stages of training. This not only improves the efficiency of knowledge transfer but also enables the student model to better capture subtle differences in the training data, thereby enhancing the model's classification performance.

[0011] In summary, the dynamic temperature adjustment method based on information entropy proposed in the present invention solves the problem of inflexible temperature setting in existing methods by dynamically optimizing the knowledge transfer in the distillation process, and significantly improves the knowledge distillation effect in image classification tasks. Summary of the Invention

[0012] The purpose of the present invention is to provide an image classification method based on information entropy dynamic adjustment temperature knowledge distillation to address the deficiencies of the existing technology.

[0013] In a first aspect, the present invention provides an image classification method based on information entropy dynamic temperature knowledge distillation, the method comprising the following steps:

[0014] Step S1: Obtain labeled images and divide them into training set and test set.

[0015] Step S2: Construct the target temperature fitting function:

[0016] 2.1 The training set is used as input to the pre-trained teacher network to obtain the first prediction classification result; the information entropy of the first prediction classification result is calculated from the first prediction classification result; according to the information entropy of the first prediction classification result, the proportion of correctly classified samples in different information entropy intervals is counted, and the above proportion data is fitted to obtain a global temperature fitting function , represents information entropy;

[0017] 2.2 The training set is used as input and input into the student network to obtain the second prediction classification result; the information entropy of the second prediction classification result is calculated from the second prediction classification result; according to the information entropy of the second prediction classification result, the data is divided according to the category and the proportion of correctly classified samples in different information entropy intervals is counted, and the above proportion data is fitted to obtain the local temperature fitting function , i represents the image category;

[0018] 2.3 Construct the target temperature fitting function based on the global temperature fitting function and the local temperature fitting function :

[0019]

[0020] in is the accuracy of the student network on the test set in the previous iteration, is the hyperparameter scaling factor.

[0021] Step S3: The residual convolution temperature prediction network outputs the dynamic distillation temperature and receives supervised training of the target temperature fitting function.

[0022] Step S4: Perform logits fusion on the top k categories with the largest difference in ability between the teacher network and the student network, where k ≥ 1.

[0023] Step S5: Use the distillation temperature to soften the classification results of the teacher network and the classification results of the student network after logits fusion to achieve knowledge distillation training and optimization.

[0024] Preferably, step S3 is specifically:

[0025] The classification results output by the teacher network and the student network and the corresponding information entropy are spliced ​​and input into the residual convolution temperature prediction network to generate the dynamic distillation temperature of each sample.

[0026] Preferably, the residual convolution temperature prediction network in step S3 includes a two-dimensional convolution layer connected in series, three residual blocks connected in series, a global average pooling layer, and a fully connected layer.

[0027] Preferably, step S4 is specifically:

[0028] 4.1 For each image category, compare the classification accuracy of the teacher network and the student network on that category, and select the top k1 categories with the largest gap; k1>k;

[0029] 4.2 Evaluate the information entropy gap: For k1 categories, calculate the information entropy difference between the corresponding teacher network and the student network, and select the top k categories with the largest information entropy gap;

[0030] 4.3 Logits Fusion:

[0031] For the selected k image categories, the classification results of the corresponding teacher network and student network are weighted fused and used as the classification result of the student network after logits fusion.

[0032] Preferably, step S4.2 is specifically:

[0033] For k1 categories, the information entropy of the teacher network is:

[0034] Entropy_T = [ E T_1 ,... , E T_k1 ] ;

[0035] in E T_i ,i∈[1,k1] represents the sum of information entropy of all samples of the i-th category in the k1 categories by the teacher network;

[0036] For k1 categories, student network information entropy:

[0037] Entropy_S = [ E S_1 ,... , E S_k1 ] ;

[0038] in E S_i ,i∈[1,k1] represents the sum of information entropy of all samples of the i-th category in the k1 categories by the student network;

[0039] Calculate the information entropy difference:

[0040] Entro py − GAP= [| E T_1 − E S_1 | ,...,| E T_k1 − E S_k1 | ] ;

[0041] Selected The image categories corresponding to the first k largest elements in the set are used as the fusion target categories.

[0042] In a second aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the method.

[0043] In a third aspect, the present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the method.

[0044] The beneficial effects of the present invention are:

[0045] The present invention introduces information entropy as the basis for dynamically adjusting the temperature, and can adaptively adjust the distillation temperature according to the complexity of the sample. During the training process, the output probability and information entropy of the teacher network and the student network are saved and input into the target temperature fitting function to generate the target temperature for each sample. This temperature can not only effectively smooth the output of the teacher network, but also improve the generalization ability of the student network on complex samples. The target temperature is input as a supervisory signal into the residual convolution temperature prediction network to optimize the training process of the image classification task. Compared with the fixed temperature method, the present invention can better cope with the complexity differences of image samples and improve the classification accuracy and robustness of the model in image classification tasks.

[0046] Preferably, in the present invention, the fitting function can be polynomial regression or neural network regression. By dynamically adjusting the distillation temperature, the student network can converge more quickly in the early stage of training and effectively learn the deep knowledge of the teacher network in the later stage of training.

[0047] Preferably, in addition to the above steps, the method of the present invention may also include performing logits fusion on samples of a specific category, and guiding the knowledge sharing between the teacher network and the student network on key categories through the CKA score to further improve the accuracy and robustness of the model.

[0048] According to the method of the present invention, during the knowledge distillation process, the student network can fully utilize the knowledge of the teacher network and significantly improve its classification performance on complex samples through an adaptive temperature adjustment strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Flowchart of the present invention.

[0050] Figure 2 This is the overall process architecture diagram of the present invention.

[0051] Figure 3 This is a visualization of the information entropy of the pre-trained vgg13 on the CIFAR100 training set. The horizontal axis represents the information entropy, the vertical axis represents the 50,000 samples in the training set, the red points represent classification errors, and the green points represent correct classifications.

[0052] Figure 4 It is the ratio of the number of correctly classified samples corresponding to each 0.1 interval of information entropy of the pre-trained vgg13 on the cifar100 training set to the total number of samples in that interval.

[0053] Figure 5 This is a display of the classification of the categories with the most correct classifications (Figure a) and the most classification errors (Figure b) of the pre-trained vgg13 on the cifar100 test set in different information entropy intervals.

[0054] Figure 6 for the reason and The target temperature fitting function composed of .

[0055] Figure 7 Calculation process of the supervision signal (target temperature).

[0056] Figure 8 Workflow for the residual convolutional temperature prediction network.

[0057] Figure 9 It is the ratio of the number of samples correctly classified for each category of MobileNet-V2 on the cifar100 test set to the number of samples correctly classified for the teacher network, where the teacher network is vgg13 and the student network is MobileNet-V2. The vertical axis is class, and the horizontal axis is the ratio of the number of samples correctly classified for the corresponding category by the student network to the number of samples correctly classified for the corresponding category by the teacher network. DETAILED DESCRIPTION

[0058] The present invention will be further analyzed below with reference to specific embodiments and accompanying drawings.

[0059] The embodiments of the present invention provide an image classification method, which solves the irrationality of static temperature commonly used in traditional knowledge distillation methods by introducing a method of dynamically regulating temperature using information entropy. The static temperature remains unchanged throughout the training process, ignoring the complexity differences between samples, resulting in poor results at different training stages and on different samples. In contrast, our method uses information entropy as a guide to dynamically adjust the temperature, enabling it to adaptively adapt to the complexity of the sample and the needs of the training stage, thereby significantly improving the performance of the student model. Therefore, Figure 1-2 As shown, the method of the present invention is specifically:

[0060] Step S1: Obtain labeled images and divide them into training set and test set;

[0061] Taking the Cifar100 dataset as an example, the dataset has 100 categories, its training set contains 50,000 pictures, and the test set contains 10,000 pictures. Each picture has its corresponding true label.

[0062] Step S2: constructing a target temperature fitting function, where the target temperature fitting function is composed of a global temperature fitting function and a local temperature fitting function.

[0063] like Figure 3 Figure 2 shows a visualization of the information entropy of the pre-trained VGG13 model on the CIFAR100 training set. By analyzing the information entropy output by the pre-trained teacher network on the training set, we found a significant correlation between the magnitude of information entropy and sample density and classification accuracy. Specifically, smaller information entropy tends to correspond to regions with high sample density, where classification accuracy is higher. Conversely, as information entropy increases, classification accuracy decreases, and the number of samples in high entropy regions is relatively small. Figure 4 It is the proportion of correct classification corresponding to each 0.1 interval of information entropy of the pre-trained vgg13 on the cifar100 training set.

[0064] 2.1 Use the training set as input to the pre-trained teacher network to obtain the corresponding predicted classification results. Calculate the information entropy data A based on the classification results predicted by the teacher network:

[0065] Assume that the training set samples are ,N is the total number of training set samples, the pre-trained teacher network is The classification probability is , then the information entropy corresponding to the sample is:

[0066]

[0067] Where C is the total number of categories in the training set, and y is the number of samples The corresponding categories, T represents the teacher network.

[0068] The entropy value set of all samples is represented as the information entropy dataset A, which contains the information entropy of all training set samples:

[0069]

[0070] From data A, we can get the ratio of the number of correctly classified samples in different information entropy intervals to the total number of samples in the interval, which is calculated by Figure 4 The information entropy interval of 0.1 precision listed in the figure is reduced to 0.01, which can obtain more detailed distribution information. Then, a global temperature fitting function can be preliminarily obtained by fitting the polynomial function. , which receives the information entropy x of the sample as input and returns the temperature corresponding to the sample.

[0071] 2.2 Obtained in step 2.1 is a relatively global temperature. For different categories, the classification ability of the teacher network may also have a large gap, such as Figure 5 As shown, the same global temperature fitting function is used for samples of category 35 and category 94. This may be inappropriate, so we introduce a class-based local temperature fitting function , x represents the sample information entropy, i represents the category corresponding to the sample, for the i-th category The fitting process is consistent with the process mentioned in 2.1. The data used in the fitting process is the ratio of the number of correctly classified samples in different information entropy intervals corresponding to the i-th category to the total number of samples in the interval. , the steps are as follows:

[0072] Statistics of the information entropy data of the student network for each sample in the training set after each iteration

[0073] ;

[0074] S represents the student network. The calculation steps are the same as those described in 2.1. That is, the training set is used as input and fed into the student network to obtain the corresponding predicted classification results. The information entropy data B is calculated based on the classification results predicted by the student network.

[0075] The information entropy of samples of all categories in B is obtained by dividing the set B according to the image category:

[0076]

[0077] Where C is the total number of categories, is the information entropy data of each sample of the student network in the i-th category, where represents the information entropy data of the kth sample in the i-th class in the training set, j = N / C represents the total number of samples in the i-th class of the training set, so The data relied on is The calculation process is consistent with that described in 2.1, that is, for each classification data, by narrowing the information entropy interval and combining the student network to classify each sample correctly, the proportion of the number of correctly classified samples in each 0.01 information entropy interval to the total number of samples in the interval can be statistically obtained. By fitting the polynomial function, a local temperature fitting function is initially obtained. , i represents the image category;

[0078] 2.3 Constructing target temperature fitting function Deep learning training is a progressive process. It is obviously not reasonable to use the same temperature at different stages of training. A higher temperature produces a smoother probability curve. In the early stages of training, fitting the student network to a smooth probability curve may cause confusion in the student network. Therefore, we believe that in the early stages of training, the optimization direction of the student network should be indicated, and there is no need to pay too much attention to dark knowledge. In the later stages of training, the ability of the student network has reached a certain stage. At this time, the temperature can be gradually increased, and the proportion of non-target class probabilities can be appropriately increased to guide the student network to learn dark knowledge. Therefore During the training process, it will change dynamically as the student network's capabilities improve, such as Figure 6 As shown, the target temperature fitting function is:

[0079]

[0080] in is the accuracy of the student network on the test set in the previous iteration, is the hyperparameter scaling factor used to adjust the temperature range;

[0081] The final target temperature is calculated as Figure 7 shown.

[0082] Step S3: The residual convolution temperature prediction network outputs the dynamic distillation temperature and simultaneously receives supervised training using the target temperature fitting function. Specifically, the classification results and corresponding information entropy output by the teacher network and the student network are concatenated and input into the residual convolution temperature prediction network to generate the dynamic distillation temperature for each sample.

[0083] In step S2, we generate adaptive temperatures for different samples by fitting the temperature function to the target temperature, but we need an effective mechanism to predict these temperatures and apply them to the training process of knowledge distillation. Therefore, we propose to use the residual convolution temperature prediction network (ResTempNet) to solve this problem, as shown in Figure 8As shown in Figure 1, the output probabilities of the teacher and student networks and their corresponding information entropy are concatenated and input into the residual convolutional temperature prediction network to generate the dynamic distillation temperature for each sample. The network structure consists of multiple residual blocks to enhance its ability to learn complex patterns.

[0084] The residual convolution temperature prediction network includes a two-dimensional convolutional layer connected in series, three residual blocks connected in series, a global average pooling layer, and a fully connected layer:

[0085] 1) Convolutional processing: Process input through convolutional networks to extract complex features.

[0086] 2) Residual Block: Multiple residual blocks are used to extract features in depth and maintain gradient stability.

[0087] 3) Global average pooling: Reduce the dimension of features and extract global features.

[0088] 4) Fully connected layer: converts the extracted global features into the final decision output.

[0089] At the same time, the output of the target temperature fitting function obtained by fitting in step S2 is used as a supervision signal, input into the convolutional temperature prediction network, and applied to the loss function of the residual convolutional temperature prediction network. In order to ensure that the distillation temperature generated by the network is Fitting temperature to target To remain consistent, we use the mean squared error (MSE) as the loss function.

[0090] The loss function is defined as:

[0091]

[0092] Where N is the number of training set samples, represents the i-th sample in the training set, is the output temperature obtained by inputting the i-th sample into the residual convolution temperature prediction network, is the target temperature obtained by inputting the i-th sample into the target temperature fitting function.

[0093] Ultimately, the target temperature will serve as the guiding target for the output of the residual convolutional temperature prediction network, which is used to train the network to optimize the accuracy and adaptability of temperature prediction.

[0094] Step S4, the teacher network and the student network may be homogeneous networks or heterogeneous networks. The differences in capabilities and architectures may cause the student network to show different capability gaps in different categories, such as Figure 9As shown in the figure, this ability gap becomes more pronounced at different stages during student network training. Therefore, before each loss calculation, we evaluate the classification ability gap between the teacher and student networks for each category and perform logits fusion on the categories with the most significant gaps, i.e., the top k categories with the largest difference in ability between the teacher and student networks, where k ≥ 1. Specifically,

[0095] 4.1 For each image category, compare the classification accuracy of the teacher network and the student network on that category, and select the top k1 categories with the largest gap, k1>k.

[0096] 4.2 Evaluating the information entropy gap: For k1 categories, calculate the information entropy difference between the corresponding teacher network and the student network:

[0097] For these k1 categories, the information entropy of the teacher network is:

[0098] Entropy_T = [ E T_1 ,... , E T_k1 ] ;

[0099] in E T_i ,i∈[1,k1] represents the sum of information entropy of all samples of the i-th category in the k1 categories by the teacher network;

[0100] For these k1 categories, the student network information entropy is:

[0101] Entropy_S = [ E S_1 ,... , E S_k1 ] ;

[0102] in E S_i ,i∈[1,k1] represents the sum of information entropy of all samples of the i-th category in the k1 categories by the student network;

[0103] Calculate the information entropy difference:

[0104] Entro py − GAP= [| E T_1 − E S_1 | ,...,| E T_k1 − E S_k1 | ] ;

[0105] Selected The image categories corresponding to the first k largest elements in the set are used as the fusion target categories;

[0106] 4.3 Logits Fusion:

[0107] For the selected k image categories, the classification results of the corresponding teacher network and student network are weightedly fused and used as the classification result of the student network after logits fusion; the fusion ratio is guided by the CKA scores output by the teacher network and the student network. The specific calculation process is as follows: for any image sample, let the output of the student network be S and the output of the teacher network be T.

[0108] a) Calculate the centralized kernel matrix:

[0109]

[0110]

[0111] These kernel matrices are then centered:

[0112]

[0113]

[0114] in, is the kernel matrix output by the student network, is the kernel matrix output by the teacher network, is the student network kernel matrix The result after centralization is is the kernel matrix of the teacher network The result after centering, 1 is a matrix of all 1s, and n is the number of samples.

[0115] b) Normalization:

[0116]

[0117]

[0118] represents the normalized student network kernel matrix; Represents the normalized teacher network kernel matrix

[0119] c) Calculate the CKA value:

[0120]

[0121] in, Represents the Frobenius norm of the matrix. Represents the calculation matrix The trace of , which is the sum of all elements on the main diagonal of the matrix.

[0122] Here, and are the centralized kernel matrices of the student network and the teacher network, respectively, and are calculated by inner product. By multiplying their transposed matrices and calculating their traces, we can obtain the similarity between them. This similarity measures the degree of similarity between the outputs of the student network and the teacher network in the feature space.

[0123] After calculating the CKA score, we fuse the logits of the teacher and student networks for the selected k categories according to the CKA score. For the i-th training sample, assuming that t is one of the categories with the largest fit gap in the set k, then for the i-th training sample, the logits of the teacher network and the student network can be expressed as:

[0124] S logits i =[ w 1 , w 2 ,…, w t ,…, w C ]∈ ℝ 1×C

[0125] T logits i =[ q 1 , q 2 ,…, q t ,…, q C ]∈ ℝ 1×C

[0126] in is the predicted probability of the teacher network for the i-th sample, is the predicted probability of the student network for the i-th sample, and C is the number of classes.

[0127] Then after the fusion for:

[0128] S logits i =[ w 1 , w 2 ,…, q t ⋅(1− CKA )+ w t ⋅ CKA ,…, w C ]∈ ℝ 1×C

[0129] By fusing logits, the fusion ratio is dynamically adjusted to better guide students' online learning.

[0130] Loss calculation:

[0131] Step S5: using distillation temperature The classification results of the teacher network and the classification results of the student network after logits fusion are softened to achieve knowledge distillation training and optimization.

[0132] Based on the fused student network logits and the distillation temperature, the final distillation loss is calculated:

[0133] ;

[0134] in, is the output of the teacher network, It is the output of the student network after logits fusion.

[0135] The fused logits incorporate both the student network's output and some of the teacher network's knowledge, further optimizing the student network's learning process. This strategy allows for better utilization of the teacher network's knowledge, helping the student network achieve better results on difficult-to-classify categories.

[0136] Experimental results analysis:

[0137] To verify the effectiveness of our proposed method for dynamically controlling temperature using information entropy, we will conduct a comparative experiment on an image classification task with the baseline distillation method. The distillation temperature of the baseline KD is fixed at 4:

[0138] 1. Verification results on CIFAR-100. The teacher and student networks use the same architecture. The results show that our method improves the performance of the baseline KD. All results are the average of 5 experiments:

[0139] Table 1

[0140]

[0141] 2. Verification results on CIFAR-100. The teacher and student networks use different architectures. The results show that our method improves performance over the baseline KD. All results are the average of 5 trials.

[0142] Table 2

[0143]

[0144] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An image classification method based on information entropy dynamic temperature adjustment knowledge distillation, characterized by The method comprises the following steps: Step S1: Obtain labeled images and divide them into training set and test set; Step S2: constructing a target temperature fitting function, wherein the target temperature fitting function includes a global temperature fitting function and a local temperature fitting function; specifically: 2.1 The training set is used as input to the pre-trained teacher network to obtain the first prediction classification result; the information entropy of the first prediction classification result is calculated from the first prediction classification result; according to the information entropy of the first prediction classification result, the proportion of correctly classified samples in different information entropy intervals is counted, and the above proportion data is fitted to obtain a global temperature fitting function , represents information entropy; 2.2 The training set is used as input and input into the student network to obtain the second prediction classification result; the information entropy of the second prediction classification result is calculated from the second prediction classification result; according to the information entropy of the second prediction classification result, the data is divided according to the category and the proportion of correctly classified samples in different information entropy intervals is counted, and the above proportion data is fitted to obtain the local temperature fitting function , i represents the image category; 2.3 Construct the target temperature fitting function based on the global temperature fitting function and the local temperature fitting function : ； in The accuracy of the student network on the test set for the previous iteration; is the hyperparameter scaling factor; Step S3: The residual convolution temperature prediction network outputs the dynamic distillation temperature and simultaneously receives supervised training of the target temperature fitting function; Step S4: Perform logits fusion on the top k categories with the largest difference in ability between the teacher network and the student network, where k ≥ 1. Specifically: 4.1 For each image category, compare the classification accuracy of the teacher network and the student network on that category, and select the top k1 categories with the largest gap; k1>k; 4.2 For k1 categories, calculate the absolute value of the difference between the information entropy of the corresponding teacher network and the student network, and select the top k categories with the largest information entropy gap; specifically: For k1 categories, the information entropy of the teacher network is: ； in represents the sum of information entropy of all samples of the i-th category in the k1 categories by the teacher network; For k1 categories, student network information entropy: Entropy_S = ; in represents the sum of information entropy of all samples of the i-th category in the k1 categories by the student network; Calculate the information entropy difference: ； Selected The image categories corresponding to the first k largest elements in the set are used as the fusion target categories; 4.3 Logits Fusion; For the selected k image categories, the classification results of the corresponding teacher network and student network are weighted fused and used as the classification result of the student network after logits fusion; Step S5: Use the distillation temperature to soften the classification results of the teacher network and the classification results of the student network after logits fusion to achieve knowledge distillation training and optimization.

2. The method according to claim 1, characterized in that Step S3 specifically includes: The classification results output by the teacher network and the student network and the corresponding information entropy are spliced ​​and input into the residual convolution temperature prediction network to generate the dynamic distillation temperature of each sample.

3. The method according to claim 1 or 2, characterized in that The residual convolution temperature prediction network in step S3 includes a two-dimensional convolution layer connected in series, three residual blocks connected in series, a global average pooling layer, and a fully connected layer.

4. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the method according to any one of claims 1 to 3.

5. A machine-readable storage medium, characterized in that The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the method described in any one of claims 1 to 3.

Images