Image classification method for dynamically adjusting temperature knowledge distillation based on information entropy

Abstract

The invention discloses an image classification method for dynamically adjusting temperature knowledge distillation based on information entropy, and the method comprises the steps: obtaining an image with a label, and constructing a target temperature fitting function; the residual error convolution temperature prediction network outputs a dynamic distillation temperature, and accepts supervised training of a target temperature fitting function at the same time; and softening a classification result of the teacher network and a classification result of the student network after logits fusion by using the distillation temperature to realize knowledge distillation training and optimization. According to the method, the information entropy is introduced as a basis for dynamically adjusting the temperature, and the distillation temperature is adaptively adjusted according to the sample complexity. In the training process, the classification results of the teacher network and the student network and the information entropy are used for generating the target temperature. The target temperature is used as a supervision signal to be input into the residual convolution temperature prediction network, the image classification task is optimized, the image sample complexity difference is better dealt with, and the classification accuracy and robustness of the model in the image classification task are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and specifically is an image classification method based on information entropy to dynamically adjust temperature knowledge distillation. Background Art

[0002] In recent years, with the rapid development of deep learning technology, image classification tasks have made significant progress. As the main method for image classification, convolutional neural networks (CNNs) have become a widely used infrastructure in the field of computer vision due to their powerful ability to capture spatial features. However, their demand for computing resources and storage space has grown rapidly, seriously affecting the deployment of these models in resource-constrained environments. For example, in smartphones, embedded devices, and edge computing scenarios, there is a significant contradiction between the demand for real-time reasoning and the size and computational complexity of the model.

[0003] To solve this problem, knowledge distillation technology has received widespread attention as an effective model compression and acceleration method. The basic idea of ​​knowledge distillation is to use a complex teacher model as a reference and transfer its rich knowledge to a relatively simplified student model. In this way, the student model can significantly reduce the amount of computation and storage requirements while maintaining high accuracy. Although knowledge distillation technology has shown great potential in the field of model compression, most existing methods rely on static temperature parameters to control the degree of softening of the teacher model's prediction results. However, the setting of static temperature ignores the dynamic changes of the sample itself and the data distribution during training. This fixed temperature processing method is often difficult to adapt to the needs of different training stages, which in turn affects the distillation effect.

[0004] The main limitations of current knowledge distillation in image classification tasks are reflected in the following aspects: 1. Static temperature setting ignores the complexity of different samples and training stages, which may cause the student model to miss key information during the learning process.

[0005] 2. Fixed temperature parameters may not help the model converge effectively in the early stages of training, and may make the model too smooth in the later stages of training and unable to fully capture the subtle differences in the training data.

[0006] 3. The existing knowledge transfer process lacks sensitivity to the relative differences between target categories, which limits the model's ability to learn high-dimensional features.

[0007] Traditional knowledge distillation methods usually use a fixed temperature parameter to control the softening of the teacher network's output, but the fixed temperature ignores changes in sample complexity and model uncertainty. For simple samples, a higher temperature may lead to unnecessary smoothing and loss of guidance value for the student network; while for complex samples, a fixed temperature may not be sufficient to effectively extract dark knowledge (i.e., low-confidence category information in the teacher network output). Therefore, the fixed temperature setting on different samples cannot adapt to the dynamic needs of the training process, limiting the effect of knowledge distillation.

[0008] Based on the above problems, the present invention proposes an image classification method, which uses a dynamic distillation temperature calculation method based on sample information entropy to solve the static temperature setting problem in the knowledge distillation process, thereby improving the image classification accuracy. Information entropy, as an indicator to measure sample uncertainty, can be used to guide the calculation of dynamic temperature. By introducing sample information entropy, the model can adjust the distillation temperature according to the uncertainty of the student model output during training. Specifically, when the uncertainty of the sample is high, the model will increase the temperature so that the student model can better learn details from the soft target of the teacher model; when the uncertainty of the sample is low, the temperature will be reduced accordingly, so that the student model can focus more on learning clear classification boundaries.

[0009] The core idea of ​​the present invention is to use dynamically adjusted temperature parameters to enhance the learning ability of the student model during the knowledge distillation process. Unlike traditional methods, the present invention automatically adjusts the distillation temperature according to the sample information entropy to ensure that the model can adaptively adjust the learning strategy at different stages of training. This not only improves the efficiency of knowledge transfer, but also enables the student model to better capture the subtle differences in the training data, thereby improving the classification performance of the model.

[0010] In summary, the dynamic temperature adjustment method based on information entropy proposed in the present invention solves the problem of inflexible temperature setting in the existing method by dynamically optimizing the knowledge transfer in the distillation process, and significantly improves the knowledge distillation effect in image classification tasks. Summary of the invention

[0011] The purpose of the present invention is to provide an image classification method based on information entropy to dynamically adjust temperature knowledge distillation in view of the deficiencies in the prior art.

[0012] In a first aspect, the present invention provides an image classification method based on information entropy dynamic adjustment temperature knowledge distillation, the method comprising the following steps: Step S1: Obtain labeled images and divide them into training set and test set.

[0013] Step S2: construct the target temperature fitting function: 2.1 The training set is used as input to the pre-trained teacher network to obtain the first prediction classification result; the information entropy of the first prediction classification result is calculated from the first prediction classification result; according to the information entropy of the first prediction classification result, the proportion of correctly classified samples in different information entropy intervals is counted, and the above proportion data is fitted to obtain a global temperature fitting function , represents information entropy; 2.2 The training set is used as input and input into the student network to obtain the second prediction classification result; the information entropy of the second prediction classification result is calculated from the second prediction classification result; according to the information entropy of the second prediction classification result, the data is divided according to the category and the proportion of correctly classified samples in different information entropy intervals is counted, and the above proportion data is fitted to obtain the local temperature fitting function , i represents the image category; 2.3 Construct the target temperature fitting function based on the global temperature fitting function and the local temperature fitting function : in is the accuracy of the student network on the test set in the previous iteration, is the hyperparameter scaling factor.

[0014] Step S3: The residual convolution temperature prediction network outputs the dynamic distillation temperature and receives supervised training of the target temperature fitting function.

[0015] Step S4: perform logits fusion on the first k categories with the largest ability difference between the teacher network and the student network, k ≥ 1.

[0016] Step S5: Use the distillation temperature to soften the classification results of the teacher network and the classification results of the student network after logits fusion to achieve knowledge distillation training and optimization.

[0017] Preferably, step S3 specifically comprises: The classification results output by the teacher network and the student network and the corresponding information entropy are concatenated and input into the residual convolution temperature prediction network to generate the dynamic distillation temperature of each sample.

[0018] Preferably, the residual convolution temperature prediction network in step S3 includes a two-dimensional convolutional layer connected in series, three residual blocks connected in series, a global average pooling layer, and a fully connected layer.

[0019] Preferably, step S4 is specifically: 4.1 For each image category, compare the classification accuracy of the teacher network and the student network on that category, and select the first k1 categories with the largest gap; k1>k; 4.2 Evaluate the information entropy gap: For k1 categories, calculate the information entropy difference between the corresponding teacher network and the student network, and select the top k categories with the largest information entropy gap; 4.3 Logits Fusion: For the selected k image categories, the classification results of the corresponding teacher network and student network are weighted fused and used as the classification result of the student network after logits fusion.

[0020] Preferably, step S4.2 is specifically: For k1 categories, the information entropy of the teacher network is: Entropy_T = [ E T_1 ,... , E T_k1 ] ; in E T_i ,i∈[1,k1] represents the sum of information entropy of all samples of the i-th category in the k1 categories by the teacher network; For k1 categories, student network information entropy: Entropy_S = [ E S_1 ,... , E S_k1 ] ; in E S_i ,i∈[1,k1] represents the sum of information entropy of all samples of the i-th category in the k1 categories by the student network; Calculate the information entropy difference: Entro py − GAP= [| E T_1 − E S_1 | ,...,| E T_k1 − E S_k1 | ] ; Select The image categories corresponding to the first k largest elements in the set are used as the fusion target categories.

[0021] In a second aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the method.

[0022] In a third aspect, the present invention provides a machine-readable storage medium storing machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the method.

[0023] The beneficial effects of the present invention are: The present invention can adaptively adjust the distillation temperature according to the complexity of the sample by introducing information entropy as the basis for dynamically adjusting the temperature. During the training process, the output probability and information entropy of the teacher network and the student network are saved and input into the target temperature fitting function to generate the target temperature of each sample. This temperature can not only effectively smooth the output of the teacher network, but also improve the generalization ability of the student network on complex samples. The target temperature is input into the residual convolution temperature prediction network as a supervisory signal to optimize the training process of the image classification task. Compared with the fixed temperature method, the present invention can better cope with the complexity differences of image samples and improve the classification accuracy and robustness of the model in image classification tasks.

[0024] Preferably, in the present invention, the fitting function can be a polynomial regression or a neural network regression, and by dynamically adjusting the distillation temperature, the student network can converge more quickly in the early stage of training and effectively learn the deep knowledge of the teacher network in the later stage of training.

[0025] Preferably, in addition to the above steps, the method of the present invention may also include performing logits fusion on samples of a specific category, and guiding the knowledge sharing between the teacher network and the student network on key categories through the CKA score to further improve the accuracy and robustness of the model.

[0026] According to the method of the present invention, during the knowledge distillation process, the student network can fully utilize the knowledge of the teacher network and significantly improve its classification performance on complex samples through an adaptive temperature adjustment strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the present invention.

[0028] Figure 2 It is the overall process architecture diagram of the present invention.

[0029] Figure 3 This is the information entropy visualization of the pre-trained vgg13 on the CIFAR100 training set. The horizontal axis represents the information entropy, the vertical axis represents the 50,000 samples in the training set, the red points represent classification errors, and the green points represent correct classifications.

[0030] Figure 4It is the ratio of the number of correctly classified samples corresponding to each 0.1 interval of information entropy of the pre-trained vgg13 on the cifar100 training set to the total number of samples in that interval.

[0031] Figure 5 The classification of the categories with the most correct classifications (Figure a) and the most misclassifications (Figure b) of the pre-trained vgg13 on the cifar100 test set in different information entropy intervals.

[0032] Figure 6 Reason and The target temperature fitting function composed of.

[0033] Figure 7 Calculation process of the supervisory signal (target temperature).

[0034] Figure 8 Workflow for the residual convolutional temperature prediction network.

[0035] Fig. 9 It is the ratio of the number of samples correctly classified in each category of MobileNet-V2 on the cifar100 test set to the number of samples correctly classified in the teacher network, where the teacher network is vgg13, the student network is MobileNet-V2, the vertical axis is class, and the horizontal axis is the ratio of the number of samples correctly classified in the corresponding category by the student network to the number of samples correctly classified in the corresponding category by the teacher network. DETAILED DESCRIPTION

[0036] The present invention is further analyzed below in conjunction with specific embodiments and drawings.

[0037] The embodiments of the present invention provide an image classification method, which solves the irrationality of static temperature commonly used in traditional knowledge distillation methods by introducing a method of dynamically regulating temperature using information entropy. The static temperature remains unchanged throughout the training process, ignoring the complexity differences between samples, resulting in poor results at different training stages and on different samples. In contrast, our method uses information entropy as a guide to dynamically adjust the temperature, so that it can adaptively adapt to the complexity of the sample and the needs of the training stage, thereby significantly improving the performance of the student model. Therefore, Figure 1-2 As shown, the method of the present invention is specifically:

[0038] Step S1, obtain labeled images and divide them into training set and test set; Taking the Cifar100 dataset as an example, the dataset has 100 categories. Its training set contains 50,000 pictures and the test set contains 10,000 pictures. Each picture has its corresponding true label.

[0039] Step S2: construct a target temperature fitting function, where the target temperature fitting function is composed of a global temperature fitting function and a local temperature fitting function.

[0040] like Figure 3 As shown in the figure, the information entropy visualization of the pre-trained vgg13 on the CIFAR100 training set is shown. By analyzing the information entropy output by the pre-trained teacher network on the training set, we found that the size of the information entropy is significantly correlated with the sample density and classification accuracy. Specifically, smaller information entropy often corresponds to high-density areas of samples, in which the classification accuracy is higher. On the contrary, as the information entropy increases, the classification accuracy of the samples gradually decreases, and the number of samples in the high information entropy area is small. Figure 4 It is the proportion of correct classification corresponding to each 0.1 interval of information entropy of the pre-trained vgg13 on the cifar100 training set.

[0041] 2.1 Take the training set as input and input it into the pre-trained teacher network to obtain the corresponding predicted classification results. Calculate the information entropy data A based on the classification results predicted by the teacher network: Assume that the training set samples are , N is the total number of samples in the training set, and the pre-trained teacher network is trained on the samples The classification probability is , then the information entropy corresponding to the sample is: Where C is the total number of categories in the training set, and y is the sample Corresponding categories, T represents the teacher network.

[0042] The entropy value set of all samples is represented as the information entropy data set A, which contains the information entropy of all training set samples: From data A, we can statistically obtain the ratio of the number of correctly classified samples in different information entropy intervals to the total number of samples in the interval, which is given by Figure 4 The information entropy interval of 0.1 precision listed in the figure is reduced to 0.01, which can obtain more detailed distribution information. Then, a global temperature fitting function can be preliminarily obtained by fitting the polynomial function. , which receives the information entropy x of the sample as input and returns the temperature corresponding to the sample.

[0043] 2.2 Step 2.1 Obtained is a relatively global temperature. For different categories, the classification ability of the teacher network may also vary greatly, such as Figure 5 As shown, the same global temperature fitting function is used for samples of category 35 and category 94. may be inappropriate, so we introduce a class-based local temperature fitting function , x represents the sample information entropy, i represents the category corresponding to the sample, and for the i-th category The fitting process is consistent with the process mentioned in 2.1. The data used in the fitting process is the ratio of the number of correctly classified samples in different information entropy intervals corresponding to the i-th category to the total number of samples in the interval. , the steps are as follows: Statistics of the information entropy data of the student network for each sample in the training set after each iteration ; S represents the student network. The calculation steps are the same as those described in 2.1, that is, the training set is used as input and input into the student network to obtain the corresponding predicted classification results. The information entropy data B is calculated based on the classification results predicted by the student network. B contains the information entropy of samples of all categories, and the set B is divided according to the image category to obtain: Where C is the total number of categories, is the information entropy data of each sample in the student network in the i-th category, where represents the information entropy data of the kth sample in the i-th class in the training set, j = N / C represents the total number of samples in the i-th class of the training set, so The data relied on are The calculation process is consistent with that described in 2.1, that is, for each classified data, by narrowing the information entropy interval and combining the student network to classify each sample correctly, the proportion of the number of correctly classified samples in each 0.01 information entropy interval to the total number of samples in the interval can be statistically obtained. By fitting the polynomial function, a local temperature fitting function is preliminarily obtained. , i represents the image category; 2.3 Constructing the target temperature fitting function ; Deep learning training is a progressive process. It is obviously not reasonable to use the same temperature at different stages of training. A larger temperature produces a smoother probability curve. In the early stage of training, fitting the student network to a smooth probability curve may cause confusion in the student network. Therefore, we believe that the optimization direction of the student network should be indicated in the early stage of training, and there is no need to pay too much attention to dark knowledge. In the later stage of training, the ability of the student network has reached a certain stage. At this time, the temperature can be gradually increased, and the proportion of non-target class probabilities can be appropriately increased to guide the student network to learn dark knowledge. Therefore During the training process, it will change dynamically as the capabilities of the student network improve, such as Figure 6 As shown, the target temperature fitting function is:

[0044] in is the accuracy of the student network on the test set in the previous iteration, is the hyperparameter scaling factor used to adjust the temperature range; The final target temperature is calculated as Figure 7 shown.

[0045] Step S3: The residual convolution temperature prediction network outputs the dynamic distillation temperature and receives supervised training of the target temperature fitting function. Specifically, the classification results output by the teacher network and the student network and the corresponding information entropy are spliced ​​and input into the residual convolution temperature prediction network to generate the dynamic distillation temperature of each sample. In step S2, we generate adaptive temperatures for different samples by fitting the temperature function through the target temperature, but an effective mechanism is needed to predict these temperatures and apply them to the training process of knowledge distillation. Therefore, we propose to use the residual convolution temperature prediction network (ResTempNet) to solve this problem, as shown in Figure 8 As shown in Figure 1, the output probabilities of the teacher network and the student network and their corresponding information entropy are concatenated and input into the residual convolution temperature prediction network to generate the dynamic distillation temperature of each sample. The network structure consists of multiple residual blocks to enhance its ability to learn complex patterns.

[0046] The residual convolution temperature prediction network includes a two-dimensional convolutional layer connected in series, three residual blocks connected in series, a global average pooling layer, and a fully connected layer: 1) Convolutional processing: Process input through convolutional networks to extract complex features.

[0047] 2) Residual blocks: Multiple residual blocks are used to extract features in depth and maintain gradient stability.

[0048] 3) Global average pooling: reduce the dimension of features and extract global features.

[0049] 4) Fully connected layer: converts the extracted global features into the final decision output.

[0050] At the same time, the output of the target temperature fitting function obtained by fitting in step S2 is used as a supervision signal, input into the convolutional temperature prediction network, and applied to the loss function of the residual convolutional temperature prediction network. In order to ensure that the distillation temperature generated by the network is Fitting temperature to target To remain consistent, we use mean squared error (MSE) as the loss function.

[0051] The loss function is defined as: Where N is the number of training set samples, represents the i-th sample in the training set, is the output temperature obtained by inputting the i-th sample into the residual convolution temperature prediction network, is the target temperature obtained by inputting the ith sample into the target temperature fitting function.

[0052] Ultimately, the target temperature will serve as the guiding target for the output of the residual convolutional temperature prediction network, which is used to train the network to optimize the accuracy and adaptability of temperature prediction.

[0053] Step S4, the teacher network and the student network may be homogeneous networks or heterogeneous networks. The differences in capabilities and architectures may cause the student network to show different capability gaps in different categories, such as Fig. 9 As shown in the figure, this ability gap becomes more significant at different stages of training during the training of the student network. Therefore, before calculating the loss each time, we evaluate the classification ability gap between the teacher network and the student network in each category, and perform logits fusion for the categories with the most significant gap, that is, the first k categories with the largest ability difference between the teacher network and the student network, k ≥ 1; specifically:

[0054] 4.1 For each image category, compare the classification accuracy of the teacher network and the student network on that category, and select the top k1 categories with the largest gap, k1>k.

[0055] 4.2 Evaluate the information entropy gap: For k1 categories, calculate the information entropy difference between the corresponding teacher network and the student network: For these k1 categories, the information entropy of the teacher network is: Entropy_T = [ E T_1 ,... , E T_k1 ] ; in E T_i ,i∈[1,k1] represents the sum of information entropy of all samples of the i-th category in the k1 categories by the teacher network; For these k1 categories, the student network information entropy is: Entropy_S = [ E S_1 ,... , E S_k1 ] ; in E S_i ,i∈[1,k1] represents the sum of information entropy of all samples of the i-th category in the k1 categories by the student network; Calculate the information entropy difference: Entro py − GAP= [| E T_1 − E S_1 | ,...,| E T_k1 − E S_k1 | ] ; Select The image categories corresponding to the first k largest elements in the set are taken as the fusion target categories; 4.3 Logits Fusion: For the selected k image categories, the classification results of the corresponding teacher network and student network are weighted fused and used as the classification result of the student network after logits fusion; the fusion ratio is guided by the CKA scores output by the teacher network and the student network. The specific calculation process is as follows: for any image sample, let the output of the student network be S and the output of the teacher network be T.

[0056] a) Calculate the centralized kernel matrix: These kernel matrices are then centered: in, is the kernel matrix output by the student network, is the kernel matrix output by the teacher network, is the student network kernel matrix The result after centralization is is the teacher network kernel matrix The result after centering, 1 is a matrix of all 1s, and n is the number of samples.

[0057] b) Normalization: represents the normalized student network kernel matrix; Represents the normalized teacher network kernel matrix c) Calculate the CKA value: in, Represents the Frobenius norm of a matrix. Represents the calculation matrix The trace of , which is the sum of all elements on the main diagonal of the matrix.

[0058] Here, and are the centralized kernel matrices of the student network and the teacher network, respectively, which are calculated by inner product. After multiplying their transposed matrices and calculating their traces, we can get the similarity between them, which measures the similarity between the outputs of the student network and the teacher network in the feature space.

[0059] After calculating the CKA score, we fuse the logits of the teacher and student networks for the selected k categories according to the CKA score. For the i-th training sample, assuming that t is one of the categories with the largest fit gap in the set k, then for the i-th training sample, the logits of the teacher network and the student network can be expressed as: S logits i =[ w 1 , w 2 ,…, w t ,…, w C ]∈ ℝ 1×C T logits i =[ q 1 , q 2 ,…, q t ,…, q C ]∈ ℝ 1×C in is the predicted probability of the teacher network for the i-th sample, is the predicted probability of the student network for the i-th sample, and C is the number of classes.

[0060] Then after the fusion for: S logits i =[ w 1 , w 2 ,…, q t ⋅(1− CKA )+ w t ⋅ CKA ,…, w C ]∈ ℝ 1×C By fusing logits, the fusion ratio is dynamically adjusted to better guide students' online learning.

[0061] Loss calculation: Step S5: using distillation temperature The classification results of the teacher network and the student network after logits fusion are softened to achieve knowledge distillation training and optimization.

[0062] Based on the fused student network logits and combined with the distillation temperature, the final distillation loss is calculated: ; in, is the output of the teacher network, It is the output of the student network after logits fusion.

[0063] The fused logits contain both the output of the student network and part of the knowledge of the teacher network, which further optimizes the learning process of the student network. Through this strategy, the knowledge of the teacher network can be better utilized to help the student network achieve better learning results on categories that are more difficult to classify.

[0064] Experimental results analysis: In order to verify the effectiveness of the proposed information entropy dynamic temperature control method, a comparative experiment is conducted on the image classification task with the baseline distillation method. The distillation temperature of the baseline KD is fixed at 4: 1. Verification results on CIFAR-100. The teacher network and the student network use the same architecture. The results show that our method has improved the performance of the baseline KD. All results are the average of 5 experiments:

[0065] Table 1

[0066] 2. Verification results on CIFAR-100. The teacher network and the student network use different architectures. The results show that our method has improved the performance of the baseline KD. All results are the average of 5 experiments.

[0067] Table 2

[0068] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An image classification method based on information entropy dynamic temperature adjustment knowledge distillation, characterized in that The method comprises the following steps: Step S1, obtain labeled images and divide them into training set and test set; Step S2, constructing a target temperature fitting function, wherein the target temperature fitting function includes a global temperature fitting function and a local temperature fitting function; Step S3, the residual convolution temperature prediction network outputs the dynamic distillation temperature and simultaneously receives supervised training of the target temperature fitting function; Step S4: perform logits fusion on the top k categories with the largest ability difference between the teacher network and the student network, k ≥ 1; Step S5: Use the distillation temperature to soften the classification results of the teacher network and the classification results of the student network after logits fusion to achieve knowledge distillation training and optimization.

2. The method according to claim 1, characterized in that: Step S2 specifically includes: S2.1 takes the training set as input and inputs it into the pre-trained teacher network to obtain the first prediction classification result; calculates the information entropy of the first prediction classification result from the first prediction classification result; based on the information entropy of the first prediction classification result, counts the proportion of correctly classified samples in different information entropy intervals, and fits the above proportion data to obtain a global temperature fitting function , represents information entropy; S2.2 takes the training set as input and inputs it into the student network to obtain the second prediction classification result; calculates the information entropy of the second prediction classification result from the second prediction classification result; according to the information entropy of the second prediction classification result, divides the data according to the category and counts the proportion of correctly classified samples in different information entropy intervals, and fits the above proportion data to obtain the local temperature fitting function , i represents the image category; S2.3 Construct the target temperature fitting function based on the global temperature fitting function and the local temperature fitting function :

3. Among them The accuracy of the student network on the test set for the previous iteration; is the hyperparameter scaling factor.

4. The method according to claim 1, characterized in that: Step S3 specifically includes: The classification results output by the teacher network and the student network and the corresponding information entropy are concatenated and input into the residual convolution temperature prediction network to generate the dynamic distillation temperature of each sample.

5. The method according to claim 1 or 3, characterized in that: The residual convolution temperature prediction network described in step S3 includes a two-dimensional convolutional layer connected in series, three residual blocks connected in series, a global average pooling layer, and a fully connected layer.

6. The method according to claim 1, characterized in that: Step S4 specifically includes: S4.1 For each image category, compare the classification accuracy of the teacher network and the student network on that category, and select the first k1 categories with the largest gap; k1>k; S4.2 For k1 categories, calculate the absolute value of the difference between the information entropy of the corresponding teacher network and the student network, and select the first k categories with the largest information entropy gap; S4.3 Logits Fusion: For the selected k image categories, the classification results of the corresponding teacher network and student network are weighted fused and used as the classification result of the student network after logits fusion.

7. The method according to claim 5, characterized in that: Step S4.2 specifically includes: For k1 categories, the information entropy of the teacher network is as follows: ； in represents the sum of information entropy of all samples of the i-th category in the k1 categories by the teacher network; For k1 categories, the student network information entropy is as follows: Entropy_S = ; in represents the sum of information entropy of all samples of the i-th category in the k1 categories by the student network; The information entropy difference is calculated as follows: ； Select The image categories corresponding to the first k largest elements in the set are used as the fusion target categories.

8. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the method according to any one of claims 1 to 6.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the method described in any one of claims 1-6.

Images