An image classification method based on responsive knowledge distillation and related equipment

Abstract

The application relates to the technical field of deep learning, and provides an image classification method based on responsive knowledge distillation and related equipment, which comprises the following steps: classifying a plurality of sample images by using a teacher model to obtain a teacher classification label of each sample image, and classifying the plurality of sample images by using a student model to obtain a student classification label of each sample image; calculating a sharpness index of the teacher model according to all the teacher classification labels, and calculating a sharpness index of the student model according to all the student classification labels; calculating a target sharpness index of the student model based on the sharpness index of the teacher model and the sharpness index of the student model; calculating an optimal distillation temperature of knowledge distillation according to the target sharpness index, and constructing a loss function based on the optimal distillation temperature; training the student model based on the loss function, and classifying a to-be-classified image by using the trained student model. The method can improve the image classification accuracy of the student model.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to an image classification method and related equipment based on responsive knowledge distillation. Background Technology

[0002] With the rapid development of deep learning in the field of artificial intelligence, large-scale pre-trained models have become the mainstream technology, and large-scale models have shown significant effects in image classification. However, due to the huge demand for computing and memory resources on large-scale models, they are difficult to deploy directly on resource-constrained edge devices, thus giving rise to the need for model compression techniques such as pruning, quantization, and knowledge distillation.

[0003] As an efficient model compression technique, knowledge distillation transfers knowledge from a teacher model (a large pre-trained model) to a student model (a smaller model), thereby reducing computational resource consumption. Existing techniques mainly explore knowledge extraction and transfer from the following aspects: responsive knowledge distillation, feature-level knowledge distillation, and relational knowledge distillation. Responsive knowledge distillation directly focuses on the model output, using the soft-label information of the teacher model to guide the training of the student model. By adjusting the softening temperature parameter to smooth the teacher output and combining cross-entropy and Kullback-Leibler (KL) divergence to construct the total loss, the student model can capture the teacher's fine-grained knowledge while also taking into account the real label information. This method is intuitive and has been validated in numerous applications. Feature-level knowledge distillation focuses on the expression of intermediate layer features in the teacher model, requiring the student model to imitate the teacher's representation process at each layer. Early methods achieved knowledge transfer by constructing intermediate layer mappings, while subsequent techniques introduced attention mechanisms and activation transfer losses to enhance feature learning. However, due to mismatches between teachers and students in terms of intermediate feature dimensions, number of layers, and expression methods, complex mapping modules are often required, increasing computational and memory burdens and potentially introducing redundancy or noise. Relational knowledge distillation focuses on extracting information about relationships between samples or features at different levels. By calculating similarities, inner products, or angles in the teacher's model output or intermediate feature space, it enables students to better capture the implicit structural relationships between data while maintaining their own expressive abilities. However, it often faces problems of high computational complexity, noise interference, and difficulty in parameter tuning when capturing complex relationships.

[0004] In contrast, responsive knowledge distillation offers greater compatibility. It directly utilizes the soft labels output by the teacher model, smooths the output by adjusting the temperature, and constructs the total loss using cross-entropy and KL divergence. This avoids the tediousness of intermediate layer alignment and complex relationship modeling while efficiently capturing the fine-grained knowledge transmitted by the teacher, achieving simple and efficient model training. However, the "teacher-student gap" problem still exists in practical applications; when the teacher model is too large, it may inhibit the learning effect of the student model. To address this, researchers have proposed Teacher Assistant Knowledge Distillation (TAKD), which introduces a teacher assistant network of intermediate size to achieve multi-step knowledge transfer from a large teacher to a small student. However, this method increases the complexity of network training and parameter tuning and relies on the design and size selection of the teacher assistant network, making it difficult to always find the optimal balance. Researchers have proposed a knowledge distillation method incorporating an annealing mechanism (Annealing-KD), which calculates the softening coefficient by simulating the annealing process. However, because the output layer probability is normalized, this method can only perform distillation in the layer preceding the model output: the logits layer. Efficient Student Knowledge Distillation (ESKD), proposed by some researchers, uses a flexible architecture and flexible learning to temporarily expand the student model during training to fully absorb teacher knowledge; however, dynamically adjusting the model structure and advanced learning strategies increases implementation complexity and parameter tuning difficulty. Attention Transfer Knowledge Distillation (ATKD), proposed by other researchers, narrows the sharpness gap between teacher and student outputs through adaptive temperature adjustment, making them more similar; however, it fails to fully consider the student model's current learning capacity, potentially leading to insufficient student absorption of the knowledge transmitted by the teacher.

[0005] Overall, existing technologies have explored knowledge distillation schemes from three perspectives: response, features, and relationships, and attempted to address the "teacher-student gap" problem. However, the gap between student and teacher models remains significant, resulting in low image classification accuracy for student models. Summary of the Invention

[0006] This application provides an image classification method and related equipment based on responsive knowledge distillation, which can solve the problem of low image classification accuracy of student models.

[0007] In a first aspect, embodiments of this application provide an image classification method based on responsive knowledge distillation, the image classification method comprising:

[0008] The teacher model is used to classify multiple sample images to obtain a teacher classification label for each sample image, and the student model is used to classify multiple sample images to obtain a student classification label for each sample image; the student model is obtained by knowledge distillation learning from the teacher model.

[0009] The sharpness index of the teacher model is calculated based on all teacher classification labels, and the sharpness index of the student model is calculated based on all student classification labels; the sharpness index is used to describe the degree of concentration of the model output probability distribution.

[0010] The target sharpness index of the student model is calculated based on the sharpness index of the teacher model and the sharpness index of the student model.

[0011] The optimal distillation temperature for knowledge distillation is calculated based on the target sharpness index, and a loss function is constructed based on the optimal distillation temperature; the loss function is used to describe the output gap between the teacher model and the student model.

[0012] The student model is trained based on the loss function to obtain the trained student model, and the trained student model is used to classify the image to be classified.

[0013] Optionally, a sharpness index for the teacher model can be calculated based on all teacher classification labels, including:

[0014] Calculate the local Riemann curvature based on all teacher classification labels, and calculate the adjustment factor based on all teacher classification labels;

[0015] The sharpness index of the teacher model is calculated based on local Riemann curvature and moderating factors.

[0016] Optionally, calculate the local Riemann curvature based on all teacher classification labels, including:

[0017] Through the formula:

[0018]

[0019] F = diag(y1, y2, ..., y n )-yy T

[0020] Calculate the local Riemann curvature K(y);

[0021] Where F represents the information content matrix, y represents all teacher category labels, and y = [y1, y2, ..., y]. n ] T y1 represents the teacher classification label of the first sample image, y2 represents the teacher classification label of the second sample image, and y nLet represent the teacher classification label of the nth sample image, and Tr(·) represent the trace operation of the matrix, i.e., the sum of the elements on the main diagonal. This represents the second-order gradient operator with respect to the input vector, and det(·) represents the determinant of the matrix.

[0022] The moderating factor was calculated based on all teacher classification labels, including:

[0023] Through the formula:

[0024]

[0025] Calculate the adjustment factor Γ(y);

[0026] Among them, y i Let ε represent the teacher classification label of the i-th sample image, where ε is a small positive number.

[0027] Optionally, the sharpness index of the teacher model can be calculated based on local Riemann curvature and moderating factors, including:

[0028] Through the formula:

[0029]

[0030] Calculate the sharpness index of the teacher model

[0031] Where α represents the smoothing hyperparameter.

[0032] Optionally, the target sharpness index for the student model is calculated based on the sharpness index of the teacher model and the sharpness index of the student model, including:

[0033] Through the formula:

[0034]

[0035] Calculate the target sharpness index

[0036] in, The sharpness index representing the student model. The sharpness index of the teacher model is represented by φ(τ), which represents the asymptotic function.

[0037]

[0038] Where k represents the span of the sliding window, σ represents the Sigmoid function, β represents the smoothing control factor, and L KD L represents the distillation loss, τ represents the training progress, and L represents the training progress. KD (k) represents the distillation loss during the t-th training iteration, L KD (tk) represents the distillation loss of the tkth training iteration, τt τ represents the training progress in the t-th training iteration. t-k Let t represent the training progress of the tkth training iteration, where t = 1, 2, ..., T, and T represents the total number of training iterations.

[0039] Optionally, the optimal distillation temperature for knowledge distillation can be calculated based on the target sharpness index, including:

[0040] Construct an objective function based on the target sharpness index;

[0041] Determine whether the value of the objective function is less than a preset threshold;

[0042] If so, the temperature parameter in the student model will be used as the optimal distillation temperature for knowledge distillation.

[0043] Otherwise, update the temperature parameters in the student model, and return to the steps of classifying each sample image using the teacher model to obtain the teacher classification label for each sample image, and classifying each sample image using the student model to obtain the student classification label for each sample image.

[0044] Optionally, the objective function is:

[0045]

[0046] Where J(T) represents the value of the objective function. An index representing the sharpness of the teacher model. This indicates the sharpness index of the target.

[0047] Optionally, the loss function is:

[0048]

[0049] in, Let λ be the value of the loss function, and λ represent the balancing parameter. Represents cross-entropy loss, Represents the loss from asymptotic knowledge distillation:

[0050]

[0051] in, Indicates the optimal distillation temperature. The temperature parameter represents the teacher model, and KL() represents the KL divergence. and represents the original logits output of the teacher model and the student model, respectively, and softmax represents the activation function.

[0052] Optionally, the student model is trained based on the loss function to obtain the trained student model, including:

[0053] Determine whether the value of the loss function is less than or equal to the preset value of the loss function;

[0054] If so, the student model will be used as the trained student model;

[0055] Otherwise, update the parameters in the student model, increment the training count by 1, and return to the steps of classifying multiple sample images using the teacher model to obtain the teacher classification label for each sample image, and classifying multiple sample images using the student model to obtain the student classification label for each sample image.

[0056] Secondly, embodiments of this application provide an image classification apparatus based on responsive knowledge distillation, comprising:

[0057] The classification module is used to classify multiple sample images using the teacher model to obtain a teacher classification label for each sample image, and to classify multiple sample images using the student model to obtain a student classification label for each sample image; the student model is obtained by knowledge distillation learning from the teacher model.

[0058] The first calculation module is used to calculate the sharpness index of the teacher model based on all teacher classification labels and the sharpness index of the student model based on all student classification labels; the sharpness index is used to describe the degree of concentration of the model output probability distribution.

[0059] The second calculation module is used to calculate the target sharpness index of the student model based on the sharpness index of the teacher model and the sharpness index of the student model.

[0060] The module is used to calculate the optimal distillation temperature for knowledge distillation based on the target sharpness index, and to construct a loss function based on the optimal distillation temperature; the loss function is used to describe the output gap between the teacher model and the student model.

[0061] The training module is used to train the student model based on the loss function to obtain the trained student model, and then use the trained student model to classify the image to be classified.

[0062] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image classification method based on responsive knowledge distillation.

[0063] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image classification method based on responsive knowledge distillation.

[0064] The above-mentioned solution in this application has the following beneficial effects:

[0065] In the embodiments of this application, multiple sample images are classified using a teacher model to obtain a teacher classification label for each sample image, and multiple sample images are classified using a student model to obtain a student classification label for each sample image. Then, the sharpness index of the teacher model is calculated based on all teacher classification labels, and the sharpness index of the student model is calculated based on all student classification labels. Then, the target sharpness index of the student model is calculated based on the sharpness index of the teacher model and the sharpness index of the student model. Then, the optimal distillation temperature of knowledge distillation is calculated based on the target sharpness index, and a loss function is constructed based on the optimal distillation temperature. Finally, the student model is trained based on the loss function to obtain a trained student model, and the trained student model is used to classify the images to be classified. The method of calculating the sharpness index of the teacher model and the student model can evaluate the output data of the two models and reflect the gap between them. By introducing a target sharpness index and calculating the optimal distillation temperature based on the target sharpness index, the student model is trained based on the optimal distillation temperature. This effectively alleviates the mismatch in structure and knowledge quality between the teacher model and the student model and the transition difficulties of the student model during training, thereby improving the stability of the student model training, reducing the gap between the teacher model and the student model, improving the performance of the student model, and ultimately improving the accuracy of the student model in image classification.

[0066] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 A flowchart of an image classification method based on responsive knowledge distillation provided in an embodiment of this application;

[0069] Figure 2 This is a schematic diagram of the output value trend curve provided in an embodiment of this application;

[0070] Figure 3 A schematic diagram of the structure of an image classification device based on responsive knowledge distillation provided in an embodiment of this application;

[0071] Figure 4This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0072] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0073] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0074] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0075] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0076] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0077] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0078] To address the significant gap between existing student and teacher models, this application provides an image classification method based on responsive knowledge distillation. This method classifies multiple sample images using a teacher model to obtain a teacher classification label for each image, and then classifies multiple sample images using a student model to obtain a student classification label for each image. The method then calculates the sharpness index of the teacher model based on all teacher classification labels, and the same applies to the student model based on all student classification labels. Finally, a target sharpness index for the student model is calculated based on both the teacher and student sharpness indices. The optimal distillation temperature for knowledge distillation is then calculated based on the target sharpness index, and a loss function is constructed based on this optimal temperature. Finally, the student model is trained using this loss function to obtain a trained student model, which is then used to classify the images to be classified. The method of calculating the sharpness index of the teacher model and the student model can evaluate the output data of the two models and reflect the gap between them. By introducing a target sharpness index and calculating the optimal distillation temperature based on the target sharpness index, the student model is trained based on the optimal distillation temperature. This effectively alleviates the mismatch in structure and knowledge quality between the teacher model and the student model and the transition difficulties of the student model during training, thereby improving the stability of the student model training, reducing the gap between the teacher model and the student model, improving the performance of the student model, and ultimately improving the accuracy of the student model in image classification.

[0079] The image classification method based on responsive knowledge distillation provided in this application will be illustrated by example below.

[0080] like Figure 1 As shown, the image classification method based on responsive knowledge distillation provided in this application includes the following steps:

[0081] Step 11: Use the teacher model to classify multiple sample images to obtain the teacher classification label for each sample image, and use the student model to classify multiple sample images to obtain the student classification label for each sample image.

[0082] The aforementioned teacher and student models are both models used for image classification, such as Convolutional Neural Networks (CNNs). The student model is learned through knowledge distillation from the teacher model, and has a larger parameter scale compared to the student model. The teacher classification label is the category label predicted by the teacher model for the sample image, and the student classification label is the category label predicted by the student model for the sample image. The sample images mentioned above are sample data from actual image classification tasks. For example, if it is necessary to classify traffic participants and identify the category of traffic participants at an intersection, the sample images can be pictures with real category labels (which can be captured by traffic cameras, dashcams, etc. at the intersection), with category labels such as pedestrians, motor vehicles, and non-motor vehicles.

[0083] Specifically, each sample image is input into the teacher model to obtain the teacher classification label for each sample image, and each sample image is input into the student model to obtain the student classification label for each sample image.

[0084] Step 12: Calculate the sharpness index of the teacher model based on all teacher classification labels, and calculate the sharpness index of the student model based on all student classification labels.

[0085] The sharpness metric describes the concentration of the model's output probability distribution. It not only describes the concentration of the output probability distribution but also reflects its response characteristics near the classification decision boundary from an information geometry perspective: higher sharpness indicates a more concentrated output distribution and greater sensitivity to interference, typically corresponding to higher confidence and a more definitive classification; lower sharpness indicates a flatter output, weaker model discrimination tendency, and higher uncertainty. Similarly, the sharpness metric can be derived for student models and can be used for subsequent dynamic adjustment of distillation targets and optimization of training strategies.

[0086] In some embodiments of this application, the step of calculating the sharpness index of the teacher model based on all teacher classification labels includes:

[0087] The first step is to calculate the local Riemann curvature based on all teacher classification labels, and then calculate the adjustment factor based on all teacher classification labels.

[0088] Local Riemann curvature describes the “bending” of the model output distribution on the information manifold: when the output distribution is highly concentrated, the curvature is large, which means that the model’s predictions on some classes are more certain and sharp; conversely, the curvature is small, which means that the distribution is flatter and the uncertainty is higher. The adjustment factor reflects the model’s sensitivity to small perturbations and its response to low-probability classes.

[0089] Specifically, through the formula:

[0090]

[0091] F = diag(y1, y2, ..., y n )-yy T

[0092] Calculate the local Riemann curvature K(y).

[0093] Where F represents the information content matrix, y represents all teacher category labels, and y = [y1, y2, ..., y]. n ] T y1 represents the teacher classification label of the first sample image, y2 represents the teacher classification label of the second sample image, and y n Let represent the teacher classification label of the nth sample image, and Tr(·) represent the trace operation of the matrix, i.e., the sum of the elements on the main diagonal. This represents the second-order gradient operator with respect to the input vector, and det(·) represents the determinant of the matrix.

[0094] Through the formula:

[0095]

[0096] Calculate the adjustment factor Γ(y).

[0097] Among them, y i Let ε represent the teacher classification label of the i-th sample image, where ε>0 is a tiny positive number introduced to prevent the denominator from being zero.

[0098] The second step is to calculate the sharpness index of the teacher model based on the local Riemann curvature and the moderating factor.

[0099] Specifically, through the formula:

[0100]

[0101] Calculate the sharpness index of the teacher model

[0102] Where α represents the smoothing hyperparameter.

[0103] It should be noted that the process of calculating the sharpness index of the student model based on all student classification labels is the same as the process of calculating the sharpness index of the teacher model. That is, the local Riemann curvature is calculated based on all student classification labels, and the moderation factor is calculated based on all student classification labels. Then, the sharpness index of the student model is calculated based on the local Riemann curvature and the moderation factor.

[0104] Step 13: Calculate the target sharpness index of the student model based on the sharpness index of the teacher model and the sharpness index of the student model.

[0105] Specifically, through the formula:

[0106]

[0107] Calculate the target sharpness index

[0108] in, The sharpness index representing the student model. The sharpness index of the teacher model is represented by φ(τ), which represents the asymptotic function.

[0109]

[0110] Where k represents the span of the sliding window, σ represents the Sigmoid function, β represents the smoothing control factor, and L KD L represents the distillation loss, τ represents the training progress, and L represents the training progress. KD (t) represents the distillation loss during the t-th training iteration, L KD (tk) represents the distillation loss of the tkth training iteration, τ t τ represents the training progress in the t-th training iteration. t-k Let t represent the training progress of the tkth training iteration, where t = 1, 2, ..., T, and T represents the total number of training iterations.

[0111] The sharpness index described above will be illustrated with a specific example below.

[0112] The trend curve of the model output value is as follows Figure 2 As shown, the horizontal axis represents categories, and the vertical axis represents probabilities. The teacher model represents the trend curve corresponding to the teacher model, and the teacher sharpness index describes the sharpness of this trend curve. The student model represents the trend curve corresponding to the student model, and the student sharpness index describes the sharpness of this trend curve. The curve between the two is the curve corresponding to the target sharpness index. The hollow arrows indicate the difference between the teacher model's trend curve and the curve corresponding to the target sharpness index. Figure 2 'a' represents the trend curve in the early stages of training. Figure 2 b represents the trend curve at the end of the training period.

[0113] It is worth mentioning that the target sharpness index calculated by the above formula has an upper bound as the sharpness index of the teacher model and a lower bound as the sharpness index of the student model, ensuring that the bachelor's degree model gradually accepts more difficult knowledge.

[0114] Step 14: Calculate the optimal distillation temperature for knowledge distillation based on the target sharpness index, and construct a loss function based on the optimal distillation temperature.

[0115] The loss function described above is used to describe the output gap between the teacher model and the student model.

[0116] In some embodiments of this application, the steps of calculating the optimal distillation temperature for knowledge distillation based on the target sharpness index and constructing a loss function based on the optimal distillation temperature include:

[0117] The first step is to construct an objective function based on the target sharpness index.

[0118] Specifically, the objective function is:

[0119]

[0120] Where J(T) represents the value of the objective function. An index representing the sharpness of the teacher model. This indicates the sharpness index of the target.

[0121] The second step is to determine whether the value of the objective function is less than a preset threshold.

[0122] If so, the temperature parameter in the student model will be used as the optimal distillation temperature for knowledge distillation.

[0123] Otherwise, update the temperature parameters in the student model, and return to the steps of classifying each sample image using the teacher model to obtain the teacher classification label for each sample image, and classifying each sample image using the student model to obtain the student classification label for each sample image.

[0124] Specifically, the expression for updating the temperature parameter is:

[0125]

[0126] Where T represents the temperature parameter in the student model, and α represents the learning rate.

[0127] For example, the aforementioned preset threshold is a pre-set value.

[0128] The third step is to construct a loss function based on the optimal distillation temperature.

[0129] Specifically, the loss function is:

[0130]

[0131] in, Let λ be the value of the loss function, and λ represent the balancing parameter. This represents the cross-entropy loss (calculated based on the classification results output by the student model and the true classification labels). Represents the loss from asymptotic knowledge distillation:

[0132]

[0133] in, Indicates the optimal distillation temperature. The temperature parameter represents the teacher model, and KL() represents the KL divergence. and represents the original logits output of the teacher model and the student model, respectively, and softmax represents the activation function.

[0134] It should be noted that the temperature parameters in the teacher and student models are hyperparameters of the softmax function in the models.

[0135] Step 15: Train the student model based on the loss function to obtain the trained student model, and use the trained student model to classify the image to be classified.

[0136] The images to be classified mentioned above are images that need to be classified, such as images of traffic participants that do not have actual classification labels.

[0137] Specifically, it determines whether the value of the loss function is less than or equal to the preset value of the loss function.

[0138] If so, the student model will be used as the trained student model.

[0139] Otherwise, update the parameters in the student model, increment the training count by 1, and return to the steps of classifying multiple sample images using the teacher model to obtain the teacher classification label for each sample image, and classifying multiple sample images using the student model to obtain the student classification label for each sample image.

[0140] It should be noted that by inputting the image to be classified into the trained student model, the class label of the image to be classified is output. Using the trained student model to classify the image can effectively improve the accuracy of traffic participant classification in the task of traffic participant classification.

[0141] For example, the default value of the loss function is a pre-set value.

[0142] It is worth mentioning that calculating the sharpness index of the teacher model and the student model can evaluate the output data of the two models, reflecting the gap between their output data. By introducing a target sharpness index and calculating the optimal distillation temperature based on the target sharpness index, and training the student model based on the optimal distillation temperature, the mismatch in structure and knowledge quality between the teacher model and the student model and the transition difficulties of the student model during training can be effectively alleviated. This improves the stability of student model training, thereby reducing the gap between the teacher model and the student model, improving the performance of the student model, and ultimately improving the accuracy of the student model in image classification.

[0143] Furthermore, this application achieves effective matching of the teacher model and student model outputs during the knowledge distillation process by introducing progressive learning, adaptive temperature control, and a clear sharpness metric, thereby improving the training effect and overall performance of the student model.

[0144] The method of this application will be illustrated below with a specific example.

[0145] The experimental environment was configured as follows: On the hardware side, a high-performance computer was used as the experimental platform, equipped with an E5-2680V4 CPU, 64GB of RAM, and a Tesla V100 professional computing card. On the software side, Ubuntu 22.04 operating system, Python 3.9 programming language, and PyTorch 1.13 deep learning framework were used.

[0146] In the specific implementation, the publicly available Canadian Institute for Advanced Research (CIFAR) dataset was used as the experimental data. For model selection, the mainstream ResNet (Residual Network) was chosen as the backbone network structure, and the depth of the teacher model was controlled to simulate teacher models of different sizes. Table 1 shows the accuracy results of using ResNet8 as the student model paired with four teacher models of different sizes on the CIFAR10 dataset.

[0147] Table 1

[0148]

[0149]

[0150] It is evident that the traditional KD method suffers from performance degradation in student models when dealing with large teacher models, but the method proposed in this application effectively solves this problem.

[0151] The image classification apparatus based on responsive knowledge distillation provided in this application is described below by way of example.

[0152] like Figure 3 As shown, this application provides an image classification device based on responsive knowledge distillation. The image classification device 300 based on responsive knowledge distillation includes:

[0153] The classification module 301 is used to classify multiple sample images using the teacher model to obtain the teacher classification label for each sample image, and to classify multiple sample images using the student model to obtain the student classification label for each sample image; the student model is obtained by knowledge distillation learning from the teacher model.

[0154] The first calculation module 302 is used to calculate the sharpness index of the teacher model based on all teacher classification labels and to calculate the sharpness index of the student model based on all student classification labels; the sharpness index is used to describe the degree of concentration of the model output probability distribution.

[0155] The second calculation module 303 is used to calculate the target sharpness index of the student model based on the sharpness index of the teacher model and the sharpness index of the student model.

[0156] Module 304 is used to calculate the optimal distillation temperature for knowledge distillation based on the target sharpness index, and to construct a loss function based on the optimal distillation temperature; the loss function is used to describe the output gap between the teacher model and the student model.

[0157] Training module 305 is used to train the student model based on the loss function to obtain the trained student model, and then use the trained student model to classify the image to be classified.

[0158] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0160] like Figure 4 As shown, an embodiment of this application provides a terminal device, wherein the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 4The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 executes the computer program D102 to implement the steps in any of the above method embodiments.

[0161] Specifically, when the processor D100 executes the computer program D102, it classifies multiple sample images using a teacher model to obtain a teacher classification label for each sample image, and classifies multiple sample images using a student model to obtain a student classification label for each sample image. Then, it calculates the sharpness index of the teacher model based on all teacher classification labels, and calculates the sharpness index of the student model based on all student classification labels. Then, it calculates the target sharpness index of the student model based on the sharpness index of the teacher model and the sharpness index of the student model. Then, it calculates the optimal distillation temperature of knowledge distillation based on the target sharpness index, and constructs a loss function based on the optimal distillation temperature. Finally, it trains the student model based on the loss function to obtain a trained student model, and uses the trained student model to classify the images to be classified. The method of calculating the sharpness index of the teacher model and the student model can evaluate the output data of the two models and reflect the gap between them. By introducing a target sharpness index and calculating the optimal distillation temperature based on the target sharpness index, the student model is trained based on the optimal distillation temperature. This effectively alleviates the mismatch in structure and knowledge quality between the teacher model and the student model and the transition difficulties of the student model during training, thereby improving the stability of the student model training, reducing the gap between the teacher model and the student model, improving the performance of the student model, and ultimately improving the accuracy of the student model in image classification.

[0162] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0163] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0164] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0165] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0166] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to an image classification method device / terminal device based on responsive knowledge distillation, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0167] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0168] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image classification method based on responsive knowledge distillation, characterized in that, include: A teacher model is used to classify multiple sample images to obtain a teacher classification label for each sample image, and a student model is used to classify the multiple sample images to obtain a student classification label for each sample image. The student model is obtained by knowledge distillation learning from the teacher model. The sharpness index of the teacher model is calculated based on all teacher classification labels, and the sharpness index of the student model is calculated based on all student classification labels; the sharpness index is used to describe the degree of concentration of the model output probability distribution. The target sharpness index of the student model is calculated based on the sharpness index of the teacher model and the sharpness index of the student model. The optimal distillation temperature for knowledge distillation is calculated based on the target sharpness index, and a loss function is constructed based on the optimal distillation temperature; the loss function is used to describe the output gap between the teacher model and the student model. The student model is trained based on the loss function to obtain the trained student model, and the trained student model is used to classify the image to be classified. The step of calculating the sharpness index of the teacher model based on all teacher classification labels includes: Calculate the local Riemann curvature based on all teacher classification labels, and calculate the adjustment factor based on all teacher classification labels; The sharpness index of the teacher model is calculated based on the local Riemann curvature and the adjustment factor.

2. The image classification method according to claim 1, characterized in that, The calculation of local Riemann curvature based on all teacher classification labels includes: Through the formula: Calculate local Riemann curvature ; in, Representing the information content matrix, This indicates all teacher category tags. , This represents the teacher classification label for the first sample image. This represents the teacher classification label for the second sample image. Indicates the first Teacher classification labels for each sample image. This represents the trace operation of a matrix, which is the sum of the elements on the main diagonal. This represents the second-order gradient operator with respect to the input vector. Represents the determinant of a matrix; The calculation of the adjustment factor based on all teacher classification labels includes: Through the formula: Calculate the adjustment factor ; in, Indicates the first Teacher classification labels for each sample image. It is a small positive number.

3. The image classification method according to claim 2, characterized in that, The calculation of the sharpness index of the teacher model based on the local Riemann curvature and the adjustment factor includes: Through the formula: Calculate the sharpness index of the teacher model ; in, This represents the smoothing hyperparameter.

4. The image classification method according to claim 1, characterized in that, The calculation of the target sharpness index of the student model based on the sharpness index of the teacher model and the sharpness index of the student model includes: Through the formula: Calculate the target sharpness index ; in, A sharpness index representing the student model. An index representing the sharpness of the teacher model. Representing an asymptotic function: in, Indicates the span of the sliding window. This represents the Sigmoid function. Indicates the smoothing regulation factor. Indicates distillation loss, Indicates training progress. Indicates the first Distillation loss during training Indicates the first Distillation loss during training Indicates the first The training progress of this training session. Indicates the first The training progress of this training session. , This indicates the total number of training sessions.

5. The image classification method according to claim 1, characterized in that, The calculation of the optimal distillation temperature for knowledge distillation based on the target sharpness index includes: Construct an objective function based on the target sharpness index; Determine whether the value of the objective function is less than a preset threshold; If so, the temperature parameter in the student model will be used as the optimal distillation temperature for knowledge distillation. Otherwise, update the temperature parameters in the student model and return to the steps of classifying each sample image using the teacher model to obtain the teacher classification label for each sample image, and classifying each sample image using the student model to obtain the student classification label for each sample image.

6. The image classification method according to claim 5, characterized in that, The objective function is: in, This represents the value of the objective function. An index representing the sharpness of the teacher model. This indicates the sharpness index of the target.

7. The image classification method according to claim 1, characterized in that, The loss function is: in, The value of the loss function. Represents the balance parameters. Represents cross-entropy loss, Represents the loss from asymptotic knowledge distillation: in, Indicates the optimal distillation temperature. The temperature parameter represents the teacher model. Denotes KL divergence, and These represent the original logits outputs of the teacher model and the student model, respectively. This represents the activation function.

8. The image classification method according to claim 1, characterized in that, The step of training the student model based on the loss function to obtain the trained student model includes: Determine whether the value of the loss function is less than or equal to a preset value for the loss function; If so, then the student model described above will be used as the trained student model; Otherwise, update the parameters in the student model, increment the training count by 1, and return to the steps of classifying multiple sample images using the teacher model to obtain the teacher classification label for each sample image, and classifying the multiple sample images using the student model to obtain the student classification label for each sample image.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image classification method based on responsive knowledge distillation as described in any one of claims 1 to 8.

Images