A temperature adaptive knowledge distillation method, system, device and storage medium

Abstract

The application discloses a temperature adaptive knowledge distillation method, system, device and storage medium. The method comprises the following steps: constructing a first temperature adjustment mechanism corresponding to each batch; constructing a second temperature adjustment mechanism corresponding to each round of training; calculating a target temperature through the first temperature adjustment mechanism, and calculating a distillation loss according to the target temperature; calculating a focal attention loss according to a teacher attention heat map and a student attention heat map; performing weighted summation on a standard classification loss, the distillation loss and the focal attention loss to obtain a total loss function; updating weight parameters of a student model according to the total loss function until all batches are trained, and entering a next round of training; adjusting a temperature range of the current round of training by using the second temperature adjustment mechanism; and training each batch based on the temperature range until a training termination condition is reached, so that a trained temperature adaptive knowledge distillation model is obtained. The application can improve the accuracy and efficiency of image analysis.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a temperature-adaptive knowledge distillation method, system, device, and storage medium. Background Technology

[0002] The training process of knowledge distillation consists of two stages. The first stage is the pre-training of the teacher model, typically using supervised learning on a large dataset to optimize its parameters and achieve high performance. The second stage is the training of the student model, which, on the same training set, utilizes both the soft labels generated by the teacher model and the hard labels provided by the dataset to guide the student model in optimizing the total loss. Through backpropagation, the student model progressively approximates the output distribution of the teacher model while maintaining its predictive ability for hard labels. In image recognition tasks, this process can effectively improve the performance of the student model under lightweight conditions, such as achieving fast and accurate image object detection on edge devices.

[0003] However, in classic knowledge distillation, student models are typically initialized with random weights, meaning they learn "from scratch." This "cold start" approach suffers from severe efficiency problems, and the learning trajectory is prone to suboptimal results. Furthermore, the distillation temperature is a static hyperparameter, unable to adapt to dynamic training processes, leading to a rigid knowledge transfer strategy. Classic knowledge distillation only allows the student model to fit the final logits output of the teacher model, resulting in a single source of knowledge. Relying solely on the logits of the teacher model's final output layer leads to a one-sided knowledge transfer, neglecting the teacher model's deeper "process knowledge."

[0004] Therefore, existing knowledge distillation methods are relatively inefficient and inaccurate in image content analysis. Summary of the Invention

[0005] This application aims to propose a temperature-adaptive knowledge distillation method, system, device, and storage medium, which can not only improve the accuracy of image analysis results but also increase the efficiency of image analysis in image content analysis.

[0006] In a first aspect, embodiments of this application provide a temperature-adaptive knowledge distillation method, the method comprising:

[0007] Obtain the training image dataset and the images to be analyzed;

[0008] Construct a temperature-adaptive knowledge distillation model that includes a teacher model with multiple network structure layers and a student model with multiple network structure layers;

[0009] The temperature adaptive knowledge distillation model is trained to obtain a trained temperature adaptive knowledge distillation model, specifically as follows:

[0010] The weight parameters of the network structure layers in the trained teacher model are transferred to the network structure layers of the student model.

[0011] The training image dataset used in each round of training is divided into multiple batches;

[0012] Calculate the standard classification loss of the student model, and based on the standard classification loss, construct a first temperature regulation mechanism for each batch; and construct a second temperature regulation mechanism for each round of training.

[0013] For each batch, the target temperature is calculated using the first temperature regulation mechanism, and the distillation loss is calculated based on the target temperature;

[0014] For each image in each batch, a teacher attention heatmap and a student attention heatmap are generated, and the focus attention loss is calculated based on the teacher attention heatmap and the student attention heatmap;

[0015] The total loss function is obtained by weighted summing of the standard classification loss, the distillation loss, and the focus attention loss.

[0016] The weight parameters of the student model are updated by backpropagation according to the total loss function until all batches are trained. Then, the next round of training is started. The temperature range of the current round of training is adjusted by the second temperature adjustment mechanism. Each batch is trained based on the temperature range until the training termination condition is reached, and a well-trained temperature adaptive knowledge distillation model is obtained.

[0017] The image to be analyzed is input into the trained temperature adaptive knowledge distillation model to obtain the image analysis results.

[0018] Compared with the prior art, the first aspect of this application has the following beneficial effects:

[0019] This method transfers the weight parameters of the network structure layers in the trained teacher model to the network structure layers in the student model; divides the training image dataset used in each round of training into multiple batches; calculates the standard classification loss of the student model, and constructs a first temperature adjustment mechanism for each batch based on the standard classification loss; and constructs a second temperature adjustment mechanism for each round of training; for each batch, the target temperature is calculated through the first temperature adjustment mechanism, and the distillation loss is calculated based on the target temperature; teacher attention heatmaps and student attention heatmaps are generated for each image in each batch, and the focus attention loss is calculated based on the teacher attention heatmaps and student attention heatmaps; the standard classification loss, distillation loss, and focus attention loss are weighted and summed to obtain the total loss function; the weight parameters of the student model are updated by backpropagation based on the total loss function until all batches are trained, and then the next round of training begins, using the second temperature adjustment mechanism to adjust the temperature range of the current round of training, and training is performed for each batch based on the temperature range until the training termination condition is met, resulting in a trained temperature-adaptive knowledge distillation model. Thus, by transferring the weight parameters of the network structure layers in the trained teacher model to the network structure layers of the student model, the problem of low efficiency in cold start is solved, thereby improving the efficiency of image analysis. Adaptive adjustment of the knowledge distillation temperature through the first and second temperature regulation mechanisms addresses the inability of existing knowledge distillation techniques to adapt to dynamic training processes, leading to rigidity in the distillation process and improving the overall efficiency and final effect of knowledge transfer. By calculating the focus attention loss based on the teacher's and student's attention heatmaps, the student model can concentrate its attention on the visual evidence deemed most important by the teacher model, solving the problems of single knowledge sources and one-sided content in existing knowledge distillation techniques, thereby improving the accuracy and robustness of the student model. Therefore, by inputting the image to be analyzed into a trained temperature-adaptive knowledge distillation model, not only can the accuracy of image analysis results be improved, but the efficiency of image analysis can also be enhanced.

[0020] Secondly, embodiments of this application also provide a temperature-adaptive knowledge distillation system, the system comprising:

[0021] The data acquisition unit is used to acquire the training image dataset and the images to be analyzed.

[0022] The model building unit is used to build a temperature-adaptive knowledge distillation model that includes a teacher model with multiple network structure layers and a student model with multiple network structure layers.

[0023] The model training unit is used to train the temperature adaptive knowledge distillation model to obtain a trained temperature adaptive knowledge distillation model, specifically as follows:

[0024] The weight parameters of the network structure layers in the trained teacher model are transferred to the network structure layers of the student model.

[0025] The training image dataset used in each round of training is divided into multiple batches;

[0026] Calculate the standard classification loss of the student model, and based on the standard classification loss, construct a first temperature regulation mechanism for each batch; and construct a second temperature regulation mechanism for each round of training.

[0027] For each batch, the target temperature is calculated using the first temperature regulation mechanism, and the distillation loss is calculated based on the target temperature;

[0028] For each image in each batch, a teacher attention heatmap and a student attention heatmap are generated, and the focus attention loss is calculated based on the teacher attention heatmap and the student attention heatmap;

[0029] The total loss function is obtained by weighted summing of the standard classification loss, the distillation loss, and the focus attention loss.

[0030] The weight parameters of the student model are updated by backpropagation according to the total loss function until all batches are trained. Then, the next round of training is started. The temperature range of the current round of training is adjusted by the second temperature adjustment mechanism. Each batch is trained based on the temperature range until the training termination condition is reached, and a well-trained temperature adaptive knowledge distillation model is obtained.

[0031] The image analysis unit is used to input the image to be analyzed into the trained temperature adaptive knowledge distillation model to obtain the image analysis results.

[0032] Thirdly, embodiments of this application also provide an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform a temperature-adaptive knowledge distillation method as described above.

[0033] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a temperature-adaptive knowledge distillation method as described above.

[0034] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0035] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0036] Figure 1 This is a schematic flowchart of an embodiment of the temperature-adaptive knowledge distillation method provided in this application;

[0037] Figure 2 yes Figure 1 A flowchart illustrating step S103 in the process;

[0038] Figure 3 This is a schematic diagram of an embodiment of the temperature-adaptive knowledge distillation system provided in this application;

[0039] Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0040] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0041] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0042] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0043] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0044] First, let's analyze some of the terms used in this application:

[0045] Soft Label: The probability output of the teacher model after being smoothed by temperature T. Unlike "hard labels," it contains information about the relative relationships between categories and is key to transmitting "dark knowledge" in knowledge distillation.

[0046] Hard Label: The true, definitive label provided in the dataset, usually in one-hot encoding form, where the correct category is 1 and the rest are 0.

[0047] Cold start: refers to the student model being trained starting from randomly initialized weights, that is, learning "from zero".

[0048] Warm Start: This refers to a situation where the student model acquires some prior knowledge (such as inheriting the weights of the teacher model) through some means (such as the fast start scheme in this embodiment) before training begins, instead of starting from a completely random state.

[0049] Focus Mask: In the teacher attention guidance module of this embodiment, it refers to a binary mask generated based on the teacher model attention heatmap and used to delineate the "learning focus area".

[0050] In classic knowledge distillation, student models are typically initialized with random weights, essentially learning "from scratch." This "cold start" approach suffers from severe efficiency issues, easily leading to suboptimal learning trajectories. Furthermore, the distillation temperature is a static hyperparameter, unable to adapt to dynamic training processes, resulting in a rigid knowledge transfer strategy. Classic knowledge distillation merely allows the student model to fit the final logits output of the teacher model, resulting in a singular knowledge source. Relying solely on the logits of the teacher model's final output layer leads to a one-sided knowledge transfer, neglecting the teacher model's deeper "process knowledge."

[0051] To address the issues of low efficiency and low accuracy in image content analysis in existing knowledge distillation methods, this application proposes a temperature-adaptive knowledge distillation method, system, device, and storage medium.

[0052] Reference Figure 1 and Figure 2 This application provides a schematic flowchart of a temperature-adaptive knowledge distillation method. This method is applied to an electronic device, such as a server or a mobile terminal. Figure 1 and Figure 2 As shown, the temperature-adaptive knowledge distillation method may include the following steps:

[0053] Step S101: Obtain the training image dataset and the images to be analyzed;

[0054] Step S102: Construct a temperature-adaptive knowledge distillation model that includes a teacher model with multiple network structure layers and a student model with multiple network structure layers;

[0055] Step S103: Train the temperature adaptive knowledge distillation model to obtain the trained temperature adaptive knowledge distillation model, specifically as follows:

[0056] Step S201: Transfer the weight parameters of the network structure layer in the trained teacher model to the network structure layer in the student model;

[0057] Step S202: Divide the training image dataset used in each round of training into multiple batches;

[0058] Step S203: Calculate the standard classification loss of the student model, and based on the standard classification loss, construct the first temperature regulation mechanism for each batch; and construct the second temperature regulation mechanism for each round of training.

[0059] Step S204: For each batch, calculate the target temperature through the first temperature regulation mechanism, and calculate the distillation loss based on the target temperature;

[0060] Step S205: Generate a teacher attention heatmap and a student attention heatmap for each image in each batch, and calculate the focus attention loss based on the teacher attention heatmap and the student attention heatmap;

[0061] Step S206: The standard classification loss, distillation loss, and focus attention loss are weighted and summed to obtain the total loss function;

[0062] Step S207: Update the weight parameters of the student model by backpropagation according to the total loss function until all batches are trained and enter the next round of training. Use the second temperature adjustment mechanism to adjust the temperature range of the current round of training. Train each batch based on the temperature range until the training termination condition is reached, and obtain the trained temperature adaptive knowledge distillation model.

[0063] Step S104: Input the image to be analyzed into the trained temperature adaptive knowledge distillation model to obtain the image analysis results.

[0064] In this embodiment, the weight parameters of the network structure layers in the trained teacher model are transferred to the network structure layers of the student model; the training image dataset used in each round of training is divided into multiple batches; the standard classification loss of the student model is calculated, and a first temperature adjustment mechanism corresponding to each batch is constructed based on the standard classification loss; and a second temperature adjustment mechanism corresponding to each round of training is constructed; for each batch, the target temperature is calculated through the first temperature adjustment mechanism, and the distillation loss is calculated based on the target temperature; a teacher attention heatmap and a student attention heatmap are generated for each image in each batch, and the focus attention loss is calculated based on the teacher attention heatmap and the student attention heatmap; the standard classification loss, distillation loss, and focus attention loss are weighted and summed to obtain the total loss function; the weight parameters of the student model are updated by backpropagation based on the total loss function until all batches are trained, and the next round of training is entered. The second temperature adjustment mechanism is used to adjust the temperature range of the current round of training, and training is performed for each batch based on the temperature range until the training termination condition is reached, resulting in a trained temperature adaptive knowledge distillation model. Thus, by transferring the weight parameters of the network structure layers in the trained teacher model to the network structure layers of the student model, the problem of low efficiency in cold start is solved, thereby improving the efficiency of image analysis. Adaptive adjustment of the knowledge distillation temperature through the first and second temperature regulation mechanisms addresses the inability of existing knowledge distillation techniques to adapt to dynamic training processes, leading to rigidity in the distillation process and improving the overall efficiency and final effect of knowledge transfer. By calculating the focus attention loss based on the teacher's and student's attention heatmaps, the student model can concentrate its attention on the visual evidence deemed most important by the teacher model, solving the problems of single knowledge sources and one-sided content in existing knowledge distillation techniques, thereby improving the accuracy and robustness of the student model. Therefore, by inputting the image to be analyzed into a trained temperature-adaptive knowledge distillation model, not only can the accuracy of image analysis results be improved, but the efficiency of image analysis can also be enhanced.

[0065] The above-mentioned training of the temperature adaptive knowledge distillation model to obtain a trained temperature adaptive knowledge distillation model can be achieved by using a training image dataset to train the temperature adaptive knowledge distillation model.

[0066] The standard classification loss of the student model can be calculated using the weighted cross-entropy loss.

[0067] The aforementioned first temperature regulation mechanism can be used to adjust the temperature of knowledge distillation in order to obtain the optimal temperature corresponding to the current batch.

[0068] The aforementioned second temperature regulation mechanism can be used to regulate the temperature range of knowledge distillation.

[0069] The above calculation of focal attention loss based on teacher attention heatmaps and student attention heatmaps can be performed by using mean squared error to calculate focal attention loss based on teacher attention heatmaps and student attention heatmaps.

[0070] The temperature mentioned above can be a hyperparameter used in knowledge distillation to smooth (or sharpen) the Logits output to generate “soft labels”.

[0071] In some implementations, the weight parameters of the network structure layers in the trained teacher model are transferred to the network structure layers of the student model, including:

[0072] Obtain the first layer name and first parameter tensor shape of each network structure layer in the teacher model, and obtain the second layer name and second parameter tensor shape of each network structure layer in the student model, wherein the parameter tensor shape includes the kernel size and the number of channels;

[0073] Based on the first layer name, the first parameter tensor shape, the second layer name, and the second parameter tensor shape, perform structural matching between each network structure layer in the teacher model and each network structure layer in the student model;

[0074] If the network structure layers in the teacher model match the network structure layers in the student model, then the weight parameters of the matching network structure layers in the trained teacher model will be transferred to the matching network structure layers in the student model.

[0075] In this embodiment, by transferring the weight parameters of the matching network structure layer in the trained teacher model to the matching network structure layer in the student model, the problem of low efficiency in cold start is solved. By starting learning from a "warm start" state, the student can understand the higher-order knowledge conveyed by the teacher more quickly, significantly accelerating the convergence process of the model and thus improving the efficiency of image analysis.

[0076] The shape of the parameter tensor mentioned above includes, but is not limited to, the kernel size and the number of channels. For example, the shape of the parameter tensor of the convolutional layer can be (kernel_h, kernel_w, in_channels, out_channels), where kernel_h is the height of the convolutional kernel, kernel_w is the width of the convolutional kernel, in_channels is the number of input channels, and out_channels is the number of output channels.

[0077] The above-mentioned structural matching of each network structure layer in the teacher model with each network structure layer in the student model based on the first layer name, the first parameter tensor shape, the second layer name, and the second parameter tensor shape can be achieved by matching the first layer name with the second layer name and the first parameter tensor shape with the second parameter tensor shape.

[0078] In some implementations, a first temperature regulation mechanism is constructed for each batch based on the standard classification loss, including:

[0079] ;

[0080] in, For the first Round Target temperature for the batch For the first Maximum temperature of the wheel , For the first Minimum temperature of the wheel , For the first Round Standard classification loss of the batch This is the preset loss threshold.

[0081] In this embodiment, the temperature of knowledge distillation is adjusted through a first temperature adjustment mechanism to obtain the optimal temperature (i.e., the target temperature), ensuring that local learning is efficient and targeted. Combined with the subsequent second temperature adjustment mechanism, it can solve the problem that existing knowledge distillation technology cannot adapt to dynamic training processes, leading to the rigidity of the distillation process, and improve the overall efficiency and final effect of knowledge transfer.

[0082] In some implementations, a second temperature regulation mechanism is constructed for each round of training, including:

[0083] After each round of training is completed, the accuracy of the student model corresponding to the current round on the validation image dataset is calculated.

[0084] The accuracy rate of the current round is compared with that of the previous round. If the accuracy rate of the current round is greater than that of the previous round, the temperature range is narrowed; if the accuracy rate of the current round is less than or equal to that of the previous round, the temperature range is widened.

[0085] In this embodiment, the constructed second temperature regulation mechanism adjusts the temperature range, which can optimize the overall rhythm of knowledge transfer according to the training progress. Combined with the first temperature regulation mechanism, it can solve the problem that the existing knowledge distillation technology cannot adapt to the dynamic training process, resulting in the rigidity of the distillation process, and improve the overall efficiency and final effect of knowledge transfer.

[0086] In some implementations, focusing attention loss is calculated based on teacher attention heatmaps and student attention heatmaps, including:

[0087] Determine the location index of key regions in the teacher attention heatmap, and generate a binarized focus mask based on the location index of the key regions;

[0088] Within the key regions of the binarized focus mask, focus attention loss is calculated based on the teacher's attention heatmap and the student's attention heatmap.

[0089] In this embodiment, by determining the location index of key areas in the teacher's attention heatmap and then calculating the focus attention loss, the student model can focus its attention on the visual evidence that the teacher model considers most important. This solves the problem of single knowledge sources and one-sided content, thereby improving the accuracy and robustness of the student model.

[0090] In some implementations, the location index of key regions in the teacher's attention heatmap is determined, and a binarized focus mask is generated based on the location index of the key regions, including:

[0091] Based on the preset mapping rules, determine the focus selection ratio corresponding to the current input image;

[0092] Calculate the number of key pixels to be selected based on the total number of pixels in the teacher's attention heatmap and the focus selection ratio;

[0093] Flatten the teacher attention heatmap into a one-dimensional vector;

[0094] Find the pixel with the highest number of key pixels in the one-dimensional vector as the key region, and obtain the position index of the key region;

[0095] Assign a value of 1 to the location index of the key area and a value of 0 to the location index of other areas to generate a binarized focus mask.

[0096] In this embodiment, by determining the location indices of key regions in the teacher's attention heatmap, and generating a binarized focus mask based on these indices, the student model delineates the key regions it needs to learn. This lays a solid data foundation for later calculations of focus attention loss, enabling the student model to concentrate its attention on the visual evidence deemed most important by the teacher model.

[0097] The above-mentioned method determines the focus selection ratio corresponding to the current input image according to a preset mapping rule. This can be achieved by fixing the focus selection ratio to a hyperparameter, for example, selecting the most important 15% region for all images. Alternatively, it can be done by looking up the corresponding focus selection ratio from a preset ratio lookup table based on the category of the input image.

[0098] In some implementations, generating teacher attention heatmaps and student attention heatmaps for each image in each batch includes:

[0099] In the teacher model, the first intermediate layer is selected, and the first intermediate layer feature map output by the first intermediate layer is obtained;

[0100] In the student model, a second intermediate layer is selected, and the feature map of the second intermediate layer output is obtained;

[0101] Based on the feature map of the first intermediate layer, a gradient-weighted activation map technique is used to generate a teacher attention heatmap corresponding to the teacher model for each image in each batch;

[0102] Based on the feature map of the second intermediate layer, a gradient-weighted activation map technique is used to generate a student attention heatmap corresponding to the student model for each image in each batch.

[0103] In this embodiment, by generating teacher attention heatmaps and student attention heatmaps for each image in each batch, a good data foundation can be laid for the subsequent calculation of focus attention loss.

[0104] The first and second intermediate layers mentioned above can be layers of the same level as the teacher and student models, and this embodiment does not impose specific limitations. For example, layer 3 of the teacher model or layer 3 of the student model can be selected.

[0105] The gradient-weighted activation map technique described above can be a visualization technique used to generate "heatmaps" that highlight the most important regions in the input image when a deep learning model makes a specific prediction.

[0106] The attention heatmap mentioned above can be a visualization image generated by techniques such as Grad-CAM, using color (e.g., from blue to red) or brightness to represent the level of attention the model pays to each part of the input.

[0107] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below:

[0108] Existing knowledge distillation methods have the following drawbacks:

[0109] Disadvantage 1: The student model uses a "cold start" initialization method, which leads to low efficiency at the beginning of training and the learning trajectory is prone to suboptimal.

[0110] In classic knowledge distillation, student models are typically initialized with random weights, meaning they learn "from scratch." This "cold start" approach suffers from severe efficiency problems. In the early stages of training, the student model not only has to learn basic features from the dataset (such as image edges and textures) from scratch, but also struggles to understand and fit the higher-order and abstract knowledge distributions (i.e., "hidden knowledge") from the mature teacher model. This dual burden of "learning basic representations" and "digesting higher-order knowledge" makes the student model converge extremely slowly in the early stages of training. More seriously, due to the randomness and disorder of the initial state, the student model struggles to effectively utilize the teacher's guidance signals immediately, easily falling into a suboptimal learning trajectory. Even if it eventually converges, its performance may fall far short of the theoretical upper limit. As some improvement studies have attempted, "warming up" the student model by directly copying some parameters from the teacher model to the student model precisely demonstrates that the standard "cold start" approach has an inherent flaw that urgently needs to be addressed.

[0111] Disadvantage 2: Distillation temperature is a static hyperparameter, which cannot adapt to the dynamic training process, resulting in a rigid knowledge transfer strategy.

[0112] In classical methods, the temperature coefficient T is a global hyperparameter that needs to be manually preset and remains constant throughout training. This static setting cannot adapt to the dynamic nature of student model learning. In the early stages of training, the student model lacks sufficient knowledge and requires a higher temperature T to "soften" the teacher model's output, enabling it to learn more generalized inter-category relationships. In the later stages of training, the student model has acquired certain capabilities, requiring a lower temperature T to "sharpen" the teacher model's output, forcing it to learn more precise discriminative knowledge. A fixed temperature T cannot simultaneously meet the needs of different stages, becoming a suboptimal choice that compromises effectiveness. Furthermore, the setting of this hyperparameter heavily relies on expensive grid search experiments, increasing the application cost and difficulty of the algorithm.

[0113] Disadvantage 3: The knowledge source is singular, relying solely on the final output layer's logits, resulting in a one-sided knowledge transfer that ignores deeper "process knowledge".

[0114] Classical knowledge distillation only allows student models to fit the final logits output of the teacher model. This means that knowledge transfer is "result-oriented" rather than "process-oriented." The power of a deep neural network lies not only in its final decision but also in the rich hierarchical structure of features extracted and abstracted step-by-step from input to output in each intermediate layer. These intermediate feature maps contain crucial information about "how the teacher model thinks," such as where its "focus of attention" is. Classical methods completely discard this valuable intermediate process information, only teaching student models "what" they are, but not "why" or "how to see." This one-dimensional knowledge transfer means that student models can only perform superficial imitation, making it difficult to learn the teacher model's deeper "problem-solving strategies," thus limiting their generalization ability and robustness.

[0115] In summary, the most significant problem with existing knowledge distillation techniques lies in their adoption of a knowledge transfer paradigm characterized by "inefficient starting point, rigid process, and one-sided content." Specifically, it begins training in an inefficient "cold start" manner, employs static and unchanging guidance strategies (fixed temperature T) during training, and the transmitted guidance content is one-sided, limited to the final result (logits), failing to form an efficient, adaptive, and comprehensive teaching system.

[0116] This triple simplification in design results in the student model becoming an inefficient "imitator" rather than a profound "inheritor," making it difficult to truly learn the essence of the teacher model. Consequently, it encounters a bottleneck in preserving performance after model compression. The purpose of the technical solution in this embodiment is precisely to systematically solve this core problem.

[0117] To address the three core shortcomings of existing knowledge distillation techniques—"inefficient starting point (cold start)," "rigid process (static temperature)," and "one-sided content (only logits)"—this embodiment aims to provide a temperature-adaptive knowledge distillation method based on teacher model attention guidance. This method treats the student model training process as a carefully guided, dynamic, and multi-dimensional learning process, rather than a simple endpoint imitation. The purpose of this embodiment is:

[0118] (1) Solve the problem of “inefficient starting point”: overcome the problem of slow initial convergence and low learning efficiency caused by the student model starting training with random weights, and realize the rapid start of the student model training process.

[0119] (2) Solve the problem of "process rigidity": Break the limitation of fixed, global static distillation temperature in traditional knowledge distillation, realize dynamic self-adaptation of distillation process, and make the "temperature" of knowledge transfer match the learning status of student model in real time.

[0120] (3) Solve the problem of “one-sided content”: Go beyond the surface knowledge of merely conveying the final prediction results (logits), introduce the “attention focus” of the teacher model in the reasoning process as a deeper level of “process knowledge” to achieve multi-dimensional and deep knowledge transfer.

[0121] The technical solution to the above problems in this embodiment includes the following:

[0122] (1) The problem of “inefficient start-up” is solved by using the “Fast Startup Scheme (FSS)”.

[0123] Current technologies treat model initialization and knowledge distillation as separate stages, resulting in student models starting training from a random state with no prior knowledge (cold start). To solve the "cold start" problem, the separation between initialization and distillation must be broken, allowing student models to possess certain "prior knowledge" before training begins. The most direct and efficient source of prior knowledge is the already well-trained teacher model itself. The weight parameters of the teacher model are themselves a condensed representation of its knowledge.

[0124] Therefore, the first step in this embodiment is to perform a weight transfer initialization operation before formal training begins. Specifically, the system iterates through all network structure layers of the student model and the teacher model. For any layer that is structurally identical (i.e., the layer name and parameter dimensions are the same), the weight parameters that have already been trained in the teacher model are directly copied to the network structure layer that matches the student model.

[0125] Through this step, the student model is no longer a "blank slate," but inherits a large number of basic feature extraction capabilities from the teacher model (such as edge and texture recognition). It starts learning from a "warm start" state, enabling it to understand the higher-order knowledge conveyed by the teacher more quickly, thus fundamentally solving the problem of low efficiency in cold starts and significantly accelerating the convergence process of the student model.

[0126] The implementation principle of the Fast Startup Scheme (FSS) module:

[0127] From a broader perspective, this module's principle is a general parameterized knowledge initialization paradigm designed to solve the "cold start" problem. This paradigm is not limited to specific neural network architectures (such as ResNet), but is applicable to any neural network with a hierarchical structure where the teacher and student models share some structural similarity or mappability. Its core idea is to inject the parameter knowledge already learned by the teacher model into the student model before formal training begins, using a pre-defined transfer mechanism, as "prior knowledge" for its learning. This module aims to address the inefficiency of student model "cold start." Its implementation principle is based on weight-based prior knowledge transfer.

[0128] Operational Procedure: Before training officially begins, the system performs a one-time initialization operation. In a preferred embodiment, the specific operation of this transfer mechanism is as follows:

[0129] Matching and copying: By comparing the layer name and the shape of the parameter tensor, the system identifies the layers (i.e., network structure layers) in the teacher model and student model that have the same structure (i.e., the layer type, convolution kernel size, and number of channels are all the same).

[0130] Weight parameter injection: For each successfully matched layer, the system directly copies and loads the pre-trained weight parameters (param_t) from the teacher model into the corresponding layer of the student model. For layers with structural mismatches, such as when the student model has fewer channels than the teacher model, this transfer mechanism can also include using a pre-defined linear transformation (such as a 1x1 convolution) or parameter selection strategy to adapt the weights of the teacher model before injection. For layers that cannot be transferred, the student model retains its randomly initialized weights.

[0131] Through the above operations, the student model inherits a large number of basic and intermediate feature extraction capabilities from the teacher model before training begins, and starts learning from a "warm start" state in which it is prepared for knowledge. This enables it to understand the higher-order knowledge passed in subsequent stages more quickly, greatly accelerating the convergence process.

[0132] (2) Solve the problem of "process rigidity" by "Adaptive Temperature Knowledge Distillation (AT-KD)".

[0133] Current techniques using a fixed distillation temperature T cannot adapt to the dynamic changes in student models at different learning stages (from requiring macro-level guidance to requiring fine-grained correction). To address the problem of "process rigidity," the distillation temperature T must be transformed from a static hyperparameter into a dynamic variable that reflects the student's learning state. The classification loss_cls of the student model is an excellent real-time indicator of its current learning state (or "confusion"). High loss indicates that the student model is "confused" and needs more macro-level and vague guidance (high T); low loss indicates that the student model is "confident" and needs more precise guidance that is closer to the true labels (low T).

[0134] Therefore, this embodiment designs a two-stage adaptive temperature mechanism, including:

[0135] At the micro level (batch-level adaptation): In each training iteration, the system first calculates the standard classification loss `loss_cls` for the student model. Then, this loss value is mapped to a dynamic `current_temp` via a function (`get_temperature`). This `current_temp` is immediately used to calculate the distillation loss `loss_div` for the current batch.

[0136] Standard classification loss can use weighted cross-entropy loss, with higher weights for the target class:

[0137] ;

[0138] in, Indicates the number of categories. For the standard classification loss, loss_cls For category The corresponding logits, For category The weights, through the weights Adjusting the loss to give the target category a greater guiding role in the calculation of the total loss guides the student model to focus on the accuracy of target category classification.

[0139] Distillation loss is calculated as follows:

[0140] ;

[0141] in, For distillation loss_div, and These are the logits of the student model and the teacher model, respectively. It is the softmax function. It is a function of temperature. It is the KL divergence.

[0142] At the macro level (periodic adaptive): After each training epoch, the system adjusts the dynamic range of temperature T based on the overall performance (test_acc) of the student model on the validation set. If the accuracy of the student model on the validation set improves, the range is narrowed for "refinement"; if the accuracy of the student model on the validation set stagnates, the range is widened for "breakthrough".

[0143] This mechanism enables the "teaching method" of knowledge distillation to respond in real time to the "learning status" of the student's model, providing directional guidance when the student's model is confused and refining details after the student's model has been mastered. This solves the problem of rigidity in the distillation process, realizing truly personalized and dynamic tutoring, thereby improving the overall efficiency and final effect of knowledge transfer.

[0144] The implementation principle of the Adaptive Temperature Knowledge Distillation (AT-KD) module:

[0145] This module aims to address the problem of a fixed distillation temperature T, which cannot adapt to the dynamic learning of the student model. Its implementation principle is to establish a two-layer dynamic temperature regulation mechanism that is linked in real-time to the student model's learning status.

[0146] Micro-batch-level adaptation: In each training batch, the system performs the following steps:

[0147] First, the standard classification loss (e.g., cross-entropy loss) of the student model on this batch of data is calculated. This loss value is considered a direct quantification of the student's current "confusion level." Then, this value is input into a pre-defined temperature mapping function, get_temperature(). The core logic is: higher loss results in higher temperature, and lower loss results in lower temperature. The specific implementation includes adaptive temperature adjustment at both the Epoch and Batch levels, with the following steps:

[0148] Epoch-level adjustments (i.e., the second temperature regulation mechanism) dynamically adjust the temperature range based on the student model's performance at the end of each epoch (such as validation accuracy). Its goal is to optimize the overall pace of knowledge transfer based on training progress. The specific implementation is as follows:

[0149] 1) After each epoch, calculate the accuracy of the student model on the validation set. (in (This refers to the Epoch number).

[0150] 2) According to Accuracy compared to the previous epoch Compare and adjust the temperature range:

[0151] like This indicates an improvement in student model performance, narrowing the temperature range to focus on accurate predictions:

[0152] ;

[0153] ;

[0154] in, To adjust the factor, This refers to the Epoch number.

[0155] like This indicates that the student model's performance has stagnated or declined; therefore, the temperature range should be expanded to explore a wider range of knowledge.

[0156] ;

[0157] ;

[0158] in, To adjust the factor, This refers to the Epoch number.

[0159] To prevent excessive temperature fluctuations, upper and lower limits are preset:

[0160] ;

[0161] ;

[0162] in, This refers to the Epoch number. They are respectively Preset minimum and maximum values; They are respectively Preset minimum and maximum values.

[0163] Batch-level adaptive temperature adjustment (i.e., the first temperature regulation mechanism) focuses on the fact that different batches of data contain samples of varying difficulty, and the learning difficulty of the student model when processing these different batches of data also varies. The standard classification loss value calculated by the student model on the current batch is used to roughly measure the degree of this learning difficulty.

[0164] When the standard classification loss is high (i.e. learning difficulty), a higher temperature is used to make the soft labels of the teacher model smoother, helping the student model explore a wide range of relationships between categories.

[0165] Conversely, if the standard classification loss is relatively low, it indicates that the student has a good grasp of this batch of data. In this case, the temperature can be appropriately lowered to provide a more "sharp" target, encouraging the student to make more accurate fits. Fine-grained batch-level adjustments obtain guidance signals that match the current learning state by establishing a dynamic mapping relationship between loss and temperature. High-loss batches can alleviate learning pressure and avoid drastic gradient fluctuations by using a smooth distribution with a higher T; low-loss batches can improve fitting accuracy and consolidate learning results by using a sharp distribution with a lower T. This adaptive mechanism ensures that the supervision signal for each batch is as close as possible to the immediate needs of the student model. The formula for calculating batch-level temperature is:

[0166] ;

[0167] in, For the first Epoch The target temperature for a batch. For the first Epoch of , For the first Epoch of , For the first Epoch Standard classification loss for batch processing. This is the preset loss threshold.

[0168] Combining Epoch-level and Batch-level temperature adjustment strategies creates a collaborative optimization mechanism that integrates global guidance and local adaptation during the knowledge distillation process. Epoch-level adjustments set the overall pace and approximate temperature range for the training process; while batch-level adjustments fine-tune the process at each specific training step based on real-time learning feedback, i.e., batch loss. This approach ensures efficient and targeted local learning. The common goal of both Epoch-level and Batch-level temperature adjustment strategies is to find the optimal temperature point that strikes a balance between "exploring the breadth of knowledge" and "pursuing prediction accuracy" at different stages of training and for samples of varying difficulty.

[0169] This mechanism enables the "temperature" of knowledge transfer to match the "acceptance capacity" of the student model in real time, achieving true individualized instruction and solving the problem of strategy rigidity caused by static temperature in traditional knowledge distillation methods.

[0170] (3) Solve the problem of “one-sided content” by “Teacher-Guided Attention”.

[0171] Existing technologies only transmit the final logits, ignoring the valuable "thinking process" of the teacher model in reaching its conclusions—that is, which regions of the image it focuses on. To address the problem of "one-sided content," a method must be found to quantify and transmit the teacher model's "thinking process." The "attention" of neural networks can be visualized using techniques such as Gradient Weighted Class Activation Maps (Grad-CAM), generating heatmaps that visually show the key image regions the model relies on when making decisions. This is precisely the "process knowledge" that can be transmitted. Therefore, this embodiment introduces an attention-alignment-based loss term (i.e., focus attention loss):

[0172] First, during training, the system generates attention heatmaps (i.e., teacher attention heatmap_t and student attention heatmap_s) for the current input image, respectively, for the teacher model and the student model. A crucial step is that this embodiment does not directly compare the two complete heatmaps; instead, the teacher model takes the lead. The system identifies the most critical regions (i.e., the brightest pixels) from the teacher's heatmap_t and uses this to generate a "keypoints mask". Finally, the system calculates the mean squared error between the student's attention heatmap and the teacher's attention heatmap only within the region defined by this "keypoints mask," using this error as the focus attention loss_gradcam.

[0173] This step forces student models not only to imitate the teacher model's final answer, but also to learn what the teacher model "sees" and focuses on. It compels student models to concentrate their attention on the visual evidence that the teacher model deems most important, thereby learning a deeper and more generalizable visual reasoning logic. This solves the problem of limited knowledge sources and one-sided content, achieving a shift from "result imitation" to "process learning," and greatly improving the performance and robustness of student models.

[0174] The implementation principle of the Teacher-Guided Attention module:

[0175] The principle of this module is to introduce a guidance mechanism based on the teacher model's "process knowledge," which exists in the form of an "attention representation" to reflect the degree of attention the teacher model pays to different regions of the input data during reasoning. This paradigm is not limited to a specific attention representation generation technique. This module aims to solve the problems of single knowledge sources and one-sided content. Its implementation principle is to transmit the teacher's visual reasoning process (i.e., "attention") as a kind of deep knowledge, with the teacher model leading the focus of learning. Its implementation process includes the following key steps:

[0176] Step 1: Extraction of attention representations.

[0177] During training, the system selects one or more predetermined intermediate layers (including a first intermediate layer and a second intermediate layer) from both the teacher and student models, and obtains the intermediate layer feature maps (including the first and second intermediate layer feature maps) output by these layers. In a preferred embodiment, Gradient-Weighted Activation Map (Grad-CAM) is used as a method to generate attention representations, generating attention heatmaps (heatmap_t and heatmap_s) for the teacher and student models respectively for the current input image. The heatmap is a matrix with the same spatial dimension as the selected intermediate layer feature map, where the brightness value of each pixel represents the degree of attention the model pays to that location when making a decision. A high brightness value indicates high attention. It should be noted that this embodiment can use other techniques capable of generating such attention representations besides Gradient-Weighted Activation Map; this embodiment does not impose specific limitations.

[0178] Step 2: Generation of the "Focus Mask" led by the teacher model.

[0179] This is the key innovation of this embodiment. Instead of directly comparing two complete heatmaps, the system performs the following "teacher-led" steps to define the learning focus. The purpose of this step is to analyze the teacher model's attention representation (i.e., the teacher attention heatmap, heatmap_t), identify the region with the strongest response (i.e., the region the teacher model considers most crucial), and generate a binary "focus mask" (keypoints_mask) according to a preset ratio. This mask delineates the "key region" (i.e., the critical region) for this learning process. The specific generation method is as follows:

[0180] 1) Get input: Get the attention representation heatmap_t generated by the teacher model in the current batch.

[0181] 2) Determine the focus ratio (k_percent): Based on a preset mapping rule, determine a focus selection ratio k for the current input image. This ratio k can be a fixed hyperparameter (e.g., selecting the most important 15% region for all images). In a preferred embodiment, the ratio k can also be dynamic; for example, the system can look up the corresponding k value from a preset ratio lookup table based on the category of the input image. This dynamic approach allows for the use of different focus ranges for tasks of varying difficulty (such as distinguishing different species of birds, which is more difficult than distinguishing birds from cars), achieving more refined guidance.

[0182] 3) Calculate the number of key points: based on the total number of pixels in heatmap_t ( ) and the determined focus selection ratio Calculate the specific number of key pixels that need to be selected: .

[0183] 4) Keypoint Index Identification: Flatten the 2D heatmap_t into a 1D vector, and use the top-k algorithm to find the pixel with the highest value (i.e., the brightest pixel in the heatmap_t). The position indexes are pixels in number. These indexes precisely point to the key regions that the teacher model is most interested in.

[0184] 5) Generate a binary mask: Create a Boolean matrix with the same size as heatmap_t, initially set to all values ​​of False (or 0), which is the initial keypoints_mask. Then, assign True (or 1) to the corresponding positions of all keypoint indices identified in the previous step in the initial keypoints_mask, resulting in a matrix containing 0s and 1s, which is the binary "focus mask" keypoints_mask.

[0185] Through the above steps, the system generates a binary "focus mask" keypoints_mask. The regions with a value of True in this mask precisely correspond to the key regions with the strongest response in the teacher's attention heatmap.

[0186] Step 3: Calculate the loss of focus attention.

[0187] The system calculates the difference between the student model's attention representation heatmap_s and the teacher model's attention representation heatmap_t only within the "keypoints_mask" defined (i.e., on pixels where the mask value is True), for example, by calculating the mean squared error (MSE). MSE represents mean squared error. This represents a heatmap of student attention within the keypoints_mask area. This represents the teacher attention heatmap within the keypoints_mask region and is used as the attention loss_gradcam.

[0188] This mechanism forces student models not only to mimic the teacher model's answers, but also to learn the teacher model's visual reasoning logic of "where to look" and "how to look." It focuses learning resources on the most crucial visual evidence for decision-making, avoiding ineffective learning in irrelevant contexts, thereby achieving deeper and more robust knowledge transfer.

[0189] This embodiment constructs a complete optimization loop from initiation to process and then to content through three main steps: FSS weight initialization, AT-KD dynamic distillation, and teacher attention guidance. Each step is derived from the causal logic of the previous step, and they work together to ultimately achieve more efficient, in-depth, and intelligent knowledge distillation than existing technologies, thus achieving the purpose of this embodiment.

[0190] The following specific embodiment is provided to better illustrate the technical solution of this embodiment:

[0191] This embodiment provides a specific example of building and evaluating a collaborative reasoning system composed of "experts" on the CIFAR-100 image classification task. The core of this embodiment is building a system that operates in two phases: First, 20 highly specialized, lightweight "specialist" student models are trained, each proficient in one CIFAR-100 superclass; then, these specialist student models are integrated with a superclass classifier and an all-around teacher model to form an efficient and accurate joint reasoning system.

[0192] 1. Experiment setup.

[0193] On the CIFAR-100 dataset, construct a high-performance image classification system composed of multiple lightweight models. The CIFAR-100 dataset contains 100 fine-grained categories, officially divided into 20 superclasses. For example, the superclass "fish" includes 5 fine-grained categories such as "goldfish, clownfish, and sturgeon".

[0194] The Teacher Model is a high-performance ResNet56 model pre-trained on the full CIFAR-100. This Teacher Model acts as an omniscient "expert," providing knowledge distillation and attention guidance for all students and teachers.

[0195] The student model uses the ResNet20 model. This model is smaller, has fewer parameters, and is more computationally efficient than the teacher model. In this example, a dedicated ResNet20 student model will be trained for each of the 20 superclasses.

[0196] The superclass classifier (Classifier / Router) is a standalone Wide Residual Network (wrn_40_1) model that is specifically trained to accurately classify an input image into one of 20 superclasses.

[0197] The hardware environment includes a server equipped with an NVIDIA GPU (specified as a CUDA device in the code) for model training and evaluation.

[0198] 2. Implementation steps.

[0199] The implementation of this embodiment is divided into two main stages: the independent training stage of the college student model and the collaborative evaluation stage of the final system.

[0200] Phase 1: Training the model for college students (train_distill.py).

[0201] The system will train a dedicated student model for each of the 20 superclasses. The following detailed explanation uses the training of the dedicated student model for superclass SC0 (aquatic_mammals) as an example. This process will be repeated for all 20 superclasses.

[0202] Step 1: Model Initialization and Fast Startup (FSS).

[0203] Load the pre-trained ResNet56 teacher model on CIFAR-100 and create an untrained ResNet20 student model. The parameters of the teacher model are frozen (i.e., require_grad = False).

[0204] To accelerate convergence, the system employs a "fast start" scheme. The code iterates through the parameter dictionary (state_dict) of the ResNet56 teacher model, checking the layer name (i.e., the first layer name) and parameter tensor shape (i.e., the first parameter tensor shape) of each layer. For each parameter in the teacher model's parameter dictionary, the program checks the ResNet20 student model's parameter dictionary for a layer with the exact same layer name (i.e., the second layer name) and parameter tensor shape (i.e., the second parameter tensor shape). Due to the structural similarity of the ResNet family, the weight dimensions of some initial layers (such as conv1, bn1, and some residual block convolutional layers) are matched. The weight values ​​of all matching layers are directly copied from the teacher model to the student model, achieving a "warm start."

[0205] Step 2: Cyclic training with dynamic adaptation and attention guidance.

[0206] The student model enters a training loop consisting of epochs (240 epochs in total). In the i-th batch of the e-th epoch, the following operations are performed:

[0207] A batch of images and ground truth labels is retrieved from the CIFAR-100 data loader. The system creates a weighted cross-entropy loss function (code: nn.CrossEntropyLoss(weight=class_weights)). Here, class_weights is a tensor of length 100. For the five classes belonging to superclass SC0 (indices 4, 30, 55, 72, 95), their corresponding weights are set to a higher value of 10, while the weights for the other 95 classes are 1. This forces the student model to focus on the classes it is "responsible" for during training, thus cultivating its specialization.

[0208] The batch images are input into the student model and the teacher model respectively to obtain their respective logits (i.e., logit_s for the student model and logit_t for the teacher model) and intermediate layer feature maps (feat_s for the student model (i.e., the second intermediate layer feature map) and feat_t for the teacher model (i.e., the first intermediate layer feature map)). The code specifies that the output of layer3 is extracted as the source feature map (i.e., the intermediate layer feature map) for attention analysis.

[0209] The standard classification loss is calculated using the cross-entropy loss function described above, by calculating the classification loss between logit_s and the true label.

[0210] The `get_temperature()` function is called, which takes the current batch's `loss_cls` value as input and calculates an optimal distillation temperature `T` (i.e., the target temperature) in real time based on a preset loss threshold (`temp_loss_threshold=0.5`) and a dynamic temperature range (initially `[min_temp, max_temp]`). When `loss_cls` is high (the student model is uncertain), the temperature increases, making the probability distribution of the teacher model smoother and providing easier-to-learn "soft labels"; conversely, when `loss_cls` is low, the temperature decreases, allowing the student model to learn sharper, more certain knowledge.

[0211] Using T as the temperature, the KL divergence between student logits and teacher logits is calculated to obtain the dynamic distillation loss_div.

[0212] The `get_batch_gradcam()` function is called, using the layer 3 feature map of the teacher model and the final `logit_t` to generate a corresponding Grad-CAM attention heatmap `heatmap_t` for each image in the batch. The `select_keypoints()` function is then called. This function analyzes only the teacher model's heatmap `heatmap_t`. It identifies the pixels with the strongest response in the heatmap based on a dynamically calculated percentage `k_percent` and generates a binary "keypoints_mask". The value of `k_percent` is dynamically determined based on the pre-calculated classification entropy (the `student_entropies` array) of the superclass; the higher the entropy of the more difficult the superclass, the larger the selected keypoints. This allows the intensity of the attention guidance to adapt to the difficulty of the task.

[0213] Similarly, an attention heatmap (heatmap_s) is generated for the student model. Then, the mean squared error (MSE) between heatmap_s and heatmap_t is calculated only at the pixel locations where keypoints_mask is marked True, yielding the teacher-guided attention loss (loss_gradcam). This forces the student model not only to mimic the teacher model's final prediction but also its "decision-making process," i.e., to learn to focus on the same key regions in the image as the teacher model.

[0214] Through formula Calculate the total loss, where, , and These are the weight parameters. Then, based on the total loss, backpropagation is performed, executing `loss.backward()` and `optimizer.step()` to update the weight parameters of the SC0 student model.

[0215] After all batches of a given epoch have been trained, the system tests the accuracy of the SC0 student model on the validation set across its five classes. If the accuracy improves compared to the best accuracy of the previous epoch, the temperature range [min_temp, max_temp] for the next epoch is moderately tightened (multiplied by temp_adjust_factor=0.95); conversely, the range is expanded to help the model explore a wider policy space. This constitutes a two-layer dynamic adaptive mechanism.

[0216] Throughout the training process, the model state with the highest accuracy on the validation set (for the target superclass) is always saved and named best_model_SC0.pth.

[0217] Step 3: Repeat the training for all superclasses.

[0218] Repeat steps one and two to train 20 highly specialized ResNet20 student models for each of the 20 superclasses (SC0 to SC19) of CIFAR-100.

[0219] Phase Two: System Collaboration Evaluation (evaluate_joint_system.py).

[0220] After training, all models are loaded into a collaborative reasoning framework to simulate how the system would operate in actual deployment and to evaluate its performance.

[0221] Load the pre-trained ResNet_20 superclass classifier. Load all 20 college student models (e.g., best_model_SC0.pth, best_model_SC1.pth, ...). Load the generic ResNet56 teacher model as a high-confidence backup decision-maker.

[0222] Inference process (for a single test image):

[0223] Step A: Routing: The image is first fed into a superclass classifier to obtain a superclass prediction result, such as SC8: large_carnivores.

[0224] Step B: Specialist Handling: The system then calls the specialist student model trained for SC8 to perform a final fine-grained category prediction on the image, such as predicting "wolf" (category index 97).

[0225] Step C: Confidence Routing: The system performs a critical verification step. It checks whether the college student's prediction ("wolf", category 97) belongs to its designated superclass (large_carnivores, which contains categories [3, 42, 43, 88, 97]).

[0226] Scenario 1 (Confirmed Prediction): The "wolf" does indeed belong to the category of "large carnivores". In this case, the system adopts the prediction and ultimately outputs "wolf".

[0227] Scenario 2 (Inconsistent Prediction): Suppose the SC8 student model incorrectly predicts "bicycle" (category 8). "Bicycle" does not belong to "large carnivores." This indicates that the student model may have encountered a difficult sample outside its domain, resulting in a low confidence level for its prediction. In this case, the system will reject the student's prediction and return the decision-making power to the more robust, general-purpose ResNet56 teacher model, which will then provide the final prediction.

[0228] This process is repeated across the entire CIFAR-100 test set to ultimately calculate several key performance indicators for the entire collaborative system, including:

[0229] Overall System Accuracy: The degree to which the final output prediction matches the true label. Superclass Classifier Accuracy: The accuracy of routing decisions. Student Model Decision Proportion: The percentage of test samples successfully processed by the student model. Student Decision Accuracy: The accuracy of predictions made by the student model on samples where the student model made the final decision. Teacher Decision Accuracy: The accuracy of teacher predictions on samples where student decisions were rejected and the teacher model took over. Average Inference Time: Evaluates the system's operational efficiency.

[0230] This embodiment constructs a model ecosystem with clear division of labor, dynamic adaptation, and fault tolerance. It not only improves the performance of individual specialized student models through innovative training methods (dynamic temperature, adaptive attention guidance), but also integrates the advantages of these specialized models through an intelligent collaborative reasoning framework, realizing the potential for higher operating efficiency while maintaining high accuracy.

[0231] The technical solution of this embodiment can be applied to an image content analysis system for a smart terminal. Here, "smart terminal" is a broad concept, encompassing, but not limited to, edge servers, mobile phones, Internet of Things (IoT) devices, smart cameras, in-vehicle computing units, and portable medical devices—all hardware devices with limited computing resources or power consumption. The core task of this system is to efficiently and accurately perform content analysis on input digital images on these resource-constrained smart terminals and output the image analysis results. Image content analysis itself is also a broad task, and its specific forms can include image classification, object detection, semantic segmentation, or assisted diagnosis of medical images.

[0232] The following example uses "image classification," the most basic image content analysis task, to illustrate in detail how the technical solution of this embodiment can be applied to the system to improve the accuracy of image classification on resource-constrained smart terminals.

[0233] The entire application process is divided into two stages: the offline model training stage (which applies the core technology of this embodiment) and the online terminal inference stage (which demonstrates the ultimate value of this embodiment).

[0234] Phase 1: Offline training of a "small but sophisticated" student model (the core of this embodiment).

[0235] Input and Objective: The input consists of a massive image dataset (such as CIFAR-100, ImageNet) and a large, powerful but structurally complex "teacher model" (such as ResNet-56) pre-trained on a cloud or high-performance server. The objective is to train a lightweight "student model" (such as ResNet-20) suitable for deployment on smart devices.

[0236] Training can be performed using the technical solution described in this embodiment:

[0237] First, a Fast Startup Scheme (FSS) is used, where the weights of network structure layers in the teacher model that match the structure of the student model are directly copied to the student model for a "warm start" initialization. This allows the student model to have basic image feature extraction capabilities from the beginning of training, rather than starting from scratch as a "blank slate".

[0238] Next, iterative training begins. For each batch of input training images:

[0239] The teacher model not only outputs its final classification prediction results (logits), but also generates its "attention heatmap" through the Grad-Class Activation Map (Grad-CAM) technique. This heatmap highlights the key regions of the image on which the teacher model makes its judgments (for example, to identify a "bird", the teacher model will focus on its "beak" and "wings").

[0240] The system in this embodiment extracts the region with the strongest response from the attention heatmap of the teacher model and generates a "focus mask".

[0241] The student model needs to complete three tasks during learning:

[0242] (1) Learn the true labels of the images (i.e., use standard classification loss).

[0243] (2) Within the critical area defined by the “focus mask”, its own attention must be aligned with the teacher model’s attention (i.e., the teacher model’s attention guides the loss). This forces the student model to learn the teacher model’s “thinking process”, that is, to learn “where to look”.

[0244] (3) Learn the soft labels output by the teacher model (i.e., use knowledge distillation loss). The distillation temperature T used to calculate this loss is dynamically generated through the adaptive temperature (AT-KD) mechanism. When the student model has a high classification loss for the current batch of images (indicating "confusion"), the system uses a higher temperature T to provide more generalized guidance; when the loss is low (indicating "confidence"), a lower temperature T is used for fine-tuning.

[0245] Phase Two: Online Smart Terminal Image Classification.

[0246] The trained, small, and sophisticated "student model" is then deployed onto the target smart terminal (e.g., a smartphone).

[0247] When a user takes a photo with their phone or selects an image for recognition, the image is fed as input into the deployed student model.

[0248] The student model runs efficiently on the phone's local storage, analyzing image content. Because it has deeply learned the teacher model's "attention focus" and "decision logic" during training, it can more accurately identify the core content of the image. Ultimately, the system outputs a high-precision classification result, such as displaying "This is a golden retriever" on the phone screen.

[0249] Through the above application methods, the technical solution of this embodiment can bring significant beneficial effects in the practical scenario of "image content analysis of smart terminals":

[0250] (1) Improved analysis accuracy on terminal devices: Compared with models trained by traditional knowledge distillation methods, the student model trained in this embodiment has higher accuracy in tasks such as image classification because it has learned the deep process knowledge of the teacher's "attention" and adopted a dynamic adaptive teaching strategy. This means that users can obtain more reliable AI analysis results on devices such as mobile phones and cameras.

[0251] (2) Efficient deployment of complex models on resource-constrained devices: The method in this embodiment successfully condenses the "intelligence" of a high-performance teacher model that was originally too large to run on a terminal into a lightweight student model. This makes high-precision image analysis tasks that could only be completed in the cloud in the past possible to be completed directly on local devices.

[0252] (3) Reduced application costs and latency: Since the analysis task is completed on the local terminal, there is no need to upload the image data to the cloud server, which greatly reduces the consumption of network bandwidth and the computing cost of the server. At the same time, it significantly reduces the latency from uploading to obtaining the results, thus improving the user experience.

[0253] (4) Enhanced data privacy and security: Sensitive data such as images do not need to leave the user's local device, fundamentally avoiding the risk of privacy leakage that may occur during data transmission.

[0254] Compared with existing technologies, the method in this embodiment differs in the following ways: Although document CN119206429A involves adaptive temperature adjustment, its adaptive temperature mechanism introduces a cosine time table into the temperature regulation mechanism. This application, however, designs a two-level adaptive temperature mechanism, including a micro-level (batch-level adaptation) and a macro-level (periodic-level adaptation). Specifically, it constructs a first temperature regulation mechanism corresponding to each batch and a second temperature regulation mechanism corresponding to each round of training. Therefore, the temperature regulation mechanism of document CN119206429A is completely different from that of this application. Documents CN119204164B and CN118762248B do not involve a temperature regulation mechanism.

[0255] Documents CN119206429A, CN119204164B, and CN118762248B do not involve the technical solution of solving the problem of "one-sided content" in the third step of the method of this embodiment by using "Teacher-Guided Attention".

[0256] Reference Figure 3 This application also provides a temperature-adaptive knowledge distillation system, which includes a data acquisition unit 301, a model building unit 302, a model training unit 303, and an image analysis unit 304, wherein:

[0257] The data acquisition unit 301 is used to acquire the training image dataset and the images to be analyzed.

[0258] Model building unit 302 is used to build a temperature-adaptive knowledge distillation model that includes a teacher model with multiple network structure layers and a student model with multiple network structure layers;

[0259] Model training unit 303 is used to train the temperature adaptive knowledge distillation model to obtain a trained temperature adaptive knowledge distillation model, specifically as follows:

[0260] The weight parameters of the network structure layers in the trained teacher model are transferred to the network structure layers of the student model.

[0261] The training image dataset used in each round of training is divided into multiple batches;

[0262] Calculate the standard classification loss of the student model, and based on the standard classification loss, construct the first temperature regulation mechanism for each batch; and construct the second temperature regulation mechanism for each round of training.

[0263] For each batch, the target temperature is calculated using the first temperature regulation mechanism, and the distillation loss is calculated based on the target temperature;

[0264] For each image in each batch, generate a teacher attention heatmap and a student attention heatmap, and calculate the focus attention loss based on the teacher attention heatmap and the student attention heatmap;

[0265] The total loss function is obtained by weighted summing of the standard classification loss, distillation loss, and focus attention loss.

[0266] The weight parameters of the student model are updated by backpropagation based on the total loss function until all batches are trained. Then, the next round of training begins. A second temperature adjustment mechanism is used to adjust the temperature range of the current round of training. Each batch is trained based on the temperature range until the training termination condition is met, resulting in a well-trained temperature adaptive knowledge distillation model.

[0267] The image analysis unit 304 is used to input the image to be analyzed into a trained temperature adaptive knowledge distillation model to obtain the image analysis results.

[0268] It should be noted that since the temperature-adaptive knowledge distillation system in this embodiment is based on the same inventive concept as the temperature-adaptive knowledge distillation method described above, the corresponding content in the method embodiment is also applicable to this system embodiment, and will not be described in detail here.

[0269] Reference Figure 4 This application also provides an electronic device, which includes:

[0270] At least one memory;

[0271] At least one processor;

[0272] At least one program;

[0273] The program is stored in memory, and the processor executes at least one program to implement the temperature-adaptive knowledge distillation method described above in this disclosure.

[0274] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0275] The electronic devices according to embodiments of this application will now be described in detail.

[0276] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0277] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to execute the temperature adaptive knowledge distillation method of the embodiments of this disclosure.

[0278] The input / output interface 1800 is used to implement information input and output.

[0279] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0280] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0281] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0282] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the above-described temperature-adaptive knowledge distillation method.

[0283] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0284] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0285] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0286] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0287] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0288] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0289] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0290] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0291] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0292] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0293] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The embodiments of this application have been described in detail above with reference to the accompanying drawings, but this application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of this application.

[0294] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. A temperature adaptive knowledge distillation method, characterized in that, The method comprises: obtaining a training image dataset and an image to be analyzed; constructing a temperature self-adaptive knowledge distillation model comprising a teacher model and a student model having a plurality of network structure layers; training the temperature self-adaptive knowledge distillation model to obtain a trained temperature self-adaptive knowledge distillation model, specifically comprising: migrating the weight parameters of the network structure layers in the trained teacher model to the network structure layers of the student model; dividing the training image dataset used in each round of training into multiple batches; calculating the standard classification loss of the student model, and based on the standard classification loss, constructing a first temperature adjustment mechanism corresponding to each batch; and constructing a second temperature adjustment mechanism corresponding to each round of training, wherein constructing the first temperature adjustment mechanism corresponding to each batch comprises: ； wherein, is the target temperature for the batch, is the maximum temperature for the , is the minimum temperature for the , is the standard classification loss for the batch, is a preset loss threshold constructing the second temperature adjustment mechanism corresponding to each round of training comprises: after each round of training is completed, calculating the accuracy of the student model corresponding to the current round on a validation image dataset; comparing the accuracy corresponding to the current round with the accuracy of the previous round, if the accuracy corresponding to the current round is greater than the accuracy of the previous round, then the temperature range is reduced, if the accuracy corresponding to the current round is less than or equal to the accuracy of the previous round, then the temperature range is expanded; for each batch, calculating the target temperature through the first temperature adjustment mechanism, and calculating the distillation loss according to the target temperature; generating a teacher attention heat map and a student attention heat map for each image in the batch, and calculating a focal attention loss according to the teacher attention heat map and the student attention heat map; weighting and summing the standard classification loss, the distillation loss, and the focal attention loss to obtain a total loss function; updating the weight parameters of the student model according to the total loss function until all batches are trained, entering the next round of training, adjusting the temperature range of the current round of training using the second temperature adjustment mechanism, and training each batch based on the temperature range until the training termination condition is reached, obtaining the trained temperature self-adaptive knowledge distillation model; inputting the image to be analyzed into the trained temperature self-adaptive knowledge distillation model to obtain an image analysis result.

2. The temperature-adaptive knowledge distillation method of claim 1, wherein, The migration of the weight parameters of the network structure layers in the trained teacher model to the network structure layers of the student model comprises: obtaining a first layer name and a first parameter tensor shape of each network structure layer in the teacher model, and obtaining a second layer name and a second parameter tensor shape of each network structure layer in the student model, wherein the parameter tensor shape includes the size of the convolution kernel and the number of channels; performing structure matching on each network structure layer in the teacher model and each network structure layer in the student model according to the first layer name, the first parameter tensor shape, the second layer name, and the second parameter tensor shape; if the network structure layers in the teacher model match the network structure layers in the student model, then migrating the weight parameters of the matched network structure layers in the trained teacher model to the matched network structure layers of the student model.

3. The temperature-adaptive knowledge distillation method of claim 1, wherein, The step of calculating focal attention loss based on the teacher's attention heatmap and the student's attention heatmap includes: Determine the location index of the key region in the teacher attention heatmap, and generate a binarized focus mask based on the location index of the key region; Within the key regions of the binarized focus mask, focus attention loss is calculated based on the teacher attention heatmap and the student attention heatmap.

4. The temperature-adaptive knowledge distillation method of claim 3, wherein, The step of determining the location index of key regions in the teacher attention heatmap and generating a binarized focus mask based on the location index of the key regions includes: Based on the preset mapping rules, determine the focus selection ratio corresponding to the current input image; Calculate the number of key pixels to be selected based on the total number of pixels in the teacher attention heatmap and the focus selection ratio; Flatten the teacher attention heatmap into a one-dimensional vector; Find the pixel with the highest number of key pixels from the one-dimensional vector as the key region, and obtain the position index of the key region; Assign a value of 1 to the location index of the key area and a value of 0 to the location index of other areas to generate a binarized focus mask.

5. The temperature-adaptive knowledge distillation method of claim 1, wherein, The generation of teacher attention heatmaps and student attention heatmaps for each image in each batch includes: In the teacher model, a first intermediate layer is selected, and the first intermediate layer feature map output by the first intermediate layer is obtained; In the student model, a second intermediate layer is selected, and the feature map of the second intermediate layer output by the second intermediate layer is obtained; Based on the first intermediate layer feature map, a teacher attention heatmap corresponding to the teacher model is generated for each image in each batch using gradient weighted activation map technology; Based on the second intermediate layer feature map, a student attention heatmap corresponding to the student model is generated for each image in each batch using gradient-weighted activation map technology.

6. A temperature-adaptive knowledge distillation system, comprising: The system includes: The data acquisition unit is used to acquire the training image dataset and the images to be analyzed. The model building unit is used to build a temperature-adaptive knowledge distillation model that includes a teacher model with multiple network structure layers and a student model with multiple network structure layers. The model training unit is used to train the temperature adaptive knowledge distillation model to obtain a trained temperature adaptive knowledge distillation model, specifically as follows: The weight parameters of the network structure layers in the trained teacher model are transferred to the network structure layers of the student model. The training image dataset used in each round of training is divided into multiple batches; Calculate the standard classification loss of the student model, and based on the standard classification loss, construct a first temperature regulation mechanism for each batch; and construct a second temperature regulation mechanism for each round of training, wherein constructing the first temperature regulation mechanism for each batch includes: ； wherein, is the target temperature for the batch, is the maximum temperature for the , is the minimum temperature for the , is the standard classification loss for the batch, is a preset loss threshold The construction of the second temperature regulation mechanism for each round of training includes: After each round of training is completed, the accuracy of the student model corresponding to the current round on the validation image dataset is calculated. The accuracy rate of the current round is compared with the accuracy rate of the previous round. If the accuracy rate of the current round is greater than that of the previous round, the temperature range is narrowed; if the accuracy rate of the current round is less than or equal to that of the previous round, the temperature range is widened. For each batch, the target temperature is calculated using the first temperature regulation mechanism, and the distillation loss is calculated based on the target temperature; For each image in each batch, a teacher attention heatmap and a student attention heatmap are generated, and the focus attention loss is calculated based on the teacher attention heatmap and the student attention heatmap; The total loss function is obtained by weighted summing of the standard classification loss, the distillation loss, and the focus attention loss. The weight parameters of the student model are updated by backpropagation according to the total loss function until all batches are trained. Then, the next round of training is started. The temperature range of the current round of training is adjusted by the second temperature adjustment mechanism. Each batch is trained based on the temperature range until the training termination condition is reached, and a well-trained temperature adaptive knowledge distillation model is obtained. The image analysis unit is used to input the image to be analyzed into the trained temperature adaptive knowledge distillation model to obtain the image analysis results.

7. An electronic device, comprising: It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor to enable the at least one control processor to perform the temperature adaptive knowledge distillation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the temperature-adaptive knowledge distillation method as described in any one of claims 1 to 5.

Images