Knowledge distillation temperature parameter dynamic adjustment method based on PID controller

Abstract

The invention provides a knowledge distillation temperature parameter dynamic adjustment method based on a PID (Proportion Integration Differentiation) controller, which is applied to the field of deep learning. In order to solve the problem that temperature parameters are fixed and are difficult to adapt to model performance changes in the existing knowledge distillation technology, a PID controller is adopted to adjust the temperature parameters in real time, and KL divergence and cross entropy loss between student and teacher models are dynamically balanced. Through a proportional-integral-differential control strategy, temperature parameters in the distillation process are automatically adjusted to enhance the model adaptability, and the distillation efficiency and the student model generalization ability are improved. The method mainly comprises the following steps: monitoring loss difference in real time, calculating temperature adjustment amount by using a PID controller, dynamically updating temperature parameters to balance distillation loss, automatically adjusting to reduce manual parameter adjustment, and improving model performance. The method is mainly used for automatically adjusting temperature parameters of knowledge distillation, manual parameter adjustment is reduced, model performance is improved, and the method is suitable for various deep learning models and data sets and has wide application value.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, specifically to the field of model compression optimization in the optimization model knowledge distillation training process, and more specifically to a method for dynamically adjusting temperature parameters of knowledge distillation based on a PID controller. Background Art

[0002] In the field of deep learning, knowledge distillation is an important model compression and optimization technology. Knowledge distillation technology uses the rich knowledge contained in a well-trained large-scale teacher model to guide the smaller-scale student model to learn. The teacher model has a strong representation ability, can make high-precision predictions on the data and contains complex knowledge information. By defining appropriate loss functions, such as KL divergence and cross entropy loss, the "soft label" knowledge of the teacher model is transferred to the student model, so that the student model can not only fit the data label during the learning process, but also absorb the knowledge essence of the teacher model, so as to retain good performance as much as possible when the model size is reduced. However, traditional knowledge distillation usually uses fixed temperature parameters for operation. As the model training progresses, the model performance continues to change, and fixed temperature parameters are difficult to adapt to this dynamic change. Models at different stages have different requirements for knowledge distillation. Fixed temperature parameters may lead to poor distillation effects and fail to fully balance the knowledge transfer between the student model and the teacher model. For example, in the early stage of model training, a larger temperature parameter may be required to promote knowledge transfer, but as the training deepens, this parameter may no longer be appropriate. In addition, manually adjusting the temperature parameter is time-consuming and labor-intensive and relies on experience, and it is difficult to accurately grasp the optimal settings under different models and data sets. This not only limits the efficiency of knowledge distillation technology, but also has an adverse effect on the generalization ability of the model. Therefore, there is an urgent need for a method that can automatically and dynamically adjust temperature parameters to adapt to changes in model performance, improve the effectiveness and adaptability of knowledge distillation, thereby improving the overall performance of deep learning models and reducing the cost of manual parameter adjustment. Summary of the invention

[0003] The present invention focuses on the problem that temperature parameters in knowledge distillation technology in the field of deep learning are difficult to dynamically adapt to changes in model performance, and proposes a method for dynamically adjusting temperature parameters of knowledge distillation based on a PID controller.

[0004] In the process of knowledge distillation, the first task is to accurately monitor the KL divergence and cross entropy loss between the student model and the teacher model. KL divergence accurately describes the deviation of the output distribution of the student model from the output distribution of the teacher model, while cross entropy loss focuses on measuring the difference between the student model prediction result and the true label. With the real-time calculation of these two losses, we can accurately understand the status of the model in the current training stage and the effectiveness of knowledge distillation.

[0005] Then the PID controller is introduced to calculate the temperature adjustment amount. The PID controller works in coordination with three control items: proportional, integral and differential. The proportional term immediately generates an adjustment component proportional to the error based on the actual value of the current loss difference, thereby quickly responding to the loss change and prompting the temperature parameter to adjust in the direction of reducing the error; the integral term accumulates the loss error in a specific period of time in the past. Its core goal is to eliminate the steady-state error of the system, so as to ensure that the temperature adjustment fully considers the historical loss situation and effectively avoids the occurrence of continuous deviations; the differential term is adjusted according to the rate of change of the loss, which can predict the change trend of the loss in advance, and then incorporate forward-looking elements into the process of adjusting the temperature parameters to prevent the system from being unstable due to excessive adjustment. In specific operations, the current loss difference and its historical accumulation and change rate are comprehensively calculated based on the pre-set proportional coefficient, integral coefficient and differential coefficient to obtain an accurate temperature adjustment amount.

[0006] Then, based on the calculated temperature adjustment amount, the temperature parameters are updated in real time in each round of training iteration of knowledge distillation, and applied to the loss function calculation process of knowledge distillation. In this way, the student model can continue to learn in a dynamically changing temperature environment. By continuously and flexibly adjusting the temperature parameters, the KL divergence and cross entropy loss between the student model and the teacher model are promoted to a relatively balanced state, which not only ensures that the student model can fully absorb the knowledge essence contained in the teacher model, but also enables it to better fit the real data labels, achieving an organic unity of knowledge inheritance and practical application. And the whole process is automated, and there is no need for manual intervention in the adjustment of temperature parameters.

[0007] The present invention has wide applicability and can be applied to a variety of deep learning models, such as the application of convolutional neural networks (CNN) in image recognition tasks, the application of recurrent neural networks (RNN) and its variants such as long short-term memory networks (LSTM) in natural language processing tasks, and models constructed by other deep learning architectures in different fields such as speech recognition and recommendation systems. At the same time, for different types of data sets, including large-scale image data sets (such as ImageNet), text data sets (such as Wikipedia text data) and other structured or unstructured data sets, when using knowledge distillation technology to optimize the model, the present invention can effectively play a role, significantly improve the model performance and greatly reduce the cost of manual parameter adjustment.

[0008] Traditional knowledge distillation generally uses fixed temperature parameters, which is difficult to adapt to the continuously changing performance conditions during model training. The present invention innovatively introduces a dynamic temperature adjustment mechanism. The root cause is that during the training process of the deep learning model, its internal parameters are always in a dynamic process of continuous updating and optimization. Correspondingly, the model's demand for knowledge distillation will also change accordingly. For example, in the early stages of training, the model parameters have not been fully optimized. At this time, a larger temperature parameter helps the student model to quickly absorb the generalized knowledge of the teacher model and accelerate the knowledge transfer process; and as the training gradually deepens, the model gradually tends to converge. At this time, it is necessary to moderately reduce the temperature parameter to enhance the student model's fitting accuracy to the real label and prevent the student model from losing its ability to adapt to new data due to over-reliance on the teacher model. With this method of dynamically adjusting the temperature parameters, it is possible to ensure that knowledge distillation can achieve the best results at all stages of model training, effectively avoiding problems such as poor distillation effects caused by fixed temperature parameters.

[0009] Compared with the traditional simple rules or manual parameter adjustment methods, the PID controller used in the present invention demonstrates extremely outstanding advantages. Traditional parameter adjustment methods often find it difficult to accurately control the timing and amplitude of temperature parameter adjustment, and usually require a lot of experimental attempts and rich experience accumulation. The PID controller, relying on the organic integration of its three control terms of proportion, integration and differentiation, can implement comprehensive and accurate temperature adjustment based on the real-time changes in loss differences, historical accumulated data and change trends. Among them, the rapid response characteristics of the proportional term, the steady-state error elimination function of the integral term, and the forward-looking adjustment ability of the differential term work together to enable the temperature parameters to always be maintained at a more suitable level during the complex and changeable model training process, greatly improving the efficiency and stability of knowledge distillation, and successfully overcoming the time-consuming and labor-intensive manual parameter adjustment process and the problem of poor accuracy.

[0010] The method of the present invention can effectively improve the efficiency of distillation. Dynamic adjustment of temperature parameters enables the student model to acquire knowledge from the teacher model in a more appropriate manner at different training stages, effectively avoiding undesirable phenomena such as knowledge transfer obstruction or overfitting caused by fixed temperature parameters, thereby significantly accelerating the process of knowledge distillation and reducing the time and computing resource consumption required for training. Taking the image classification task as an example, the model that uses the method of the present invention for knowledge distillation can achieve a higher accuracy improvement within the same training time compared to the traditional distillation method with fixed temperature parameters; or the training time can be greatly reduced while achieving the same accuracy.

[0011] At the same time, the present invention can also enhance the generalization ability of the student model. With the help of a reasonable temperature parameter dynamic adjustment strategy, the student model can more properly balance the knowledge learned from the teacher model and the ability to fit real data. During the entire training process, it will neither lose its flexibility to adapt to new data due to over-reliance on the teacher model, nor will the generalization performance decline due to neglecting the knowledge guidance of the teacher model. Taking the text classification task in natural language processing as an example, the student model optimized by the present invention can perform classification operations more accurately when facing text data that has never been seen, and the generalization error is significantly reduced compared to traditional methods, thereby showing stronger adaptability and reliability in actual application scenarios, laying a solid foundation for the effective deployment of the model in complex and changeable real-world application environments.

[0012] Moreover, the present invention can reduce the workload and uncertainty of manual parameter adjustment. Its automated temperature parameter adjustment process completely gets rid of the tedious operation of manually trying different temperature values ​​and evaluating the effects, greatly reducing the workload of manual parameter adjustment. At the same time, because the PID controller is based on an accurate mathematical model and the real-time state of the model for intelligent adjustment, it effectively avoids the problem of unreasonable parameter settings caused by lack of experience or subjective misjudgment during manual parameter adjustment, and significantly improves the repeatability and stability of knowledge distillation. This feature enables different researchers or developers to obtain more consistent results when using this technology, which has strongly promoted the widespread application and in-depth promotion of knowledge distillation technology in the field of deep learning, and promoted the overall development and innovative breakthroughs of deep learning technology.

[0013] To sum up, the knowledge distillation temperature parameter dynamic adjustment method based on PID controller in the present invention has extremely significant innovation and practicality in the field of deep learning model optimization. It can effectively resolve many problems and shortcomings in the existing knowledge distillation technology, and provide strong technical support and guarantee for the performance improvement and diversified application expansion of deep learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A technical architecture diagram of a method for dynamically adjusting temperature parameters of knowledge distillation based on a PID controller according to the present invention; DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0016] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0017] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0018] like Figure 1 As shown, the embodiment of the present invention provides a method for dynamically adjusting distillation temperature parameters based on a PID controller, including steps S1-S7:

[0019] S1: Environment setup and preparation

[0020] Determine the appropriate hardware environment based on the actual application scenario and the size of the data set. For example, for knowledge distillation tasks of large-scale image data sets, you can choose a server equipped with multiple high-end GPUs (such as NVIDIA Tesla V100), and equipped with large-capacity memory (64GB and above) and solid-state drives to ensure data reading, writing and storage, which can meet the large number of matrix operations required in deep learning model training. In terms of software environment construction, choose a deep learning framework such as PyTorch, download and install it through the official website according to the hardware configuration and operating system prompts, and use the conda environment to install it in the Linux system. At the same time, install related auxiliary libraries, such as numpy and pandas libraries for data processing, which can be installed by entering the "pipinstall numpy pandas" command in the command line, and matplotlib libraries for visualizing training processes and results, which can be installed by entering the "pip install matplotlib" command in the command line. You may also need to install specific model libraries or toolkits, such as the OpenCV library for processing image data. Next, prepare the data set. According to the deep learning task (image recognition, natural language processing, etc.), collect or select the appropriate data set. For example, for general object recognition tasks, the ImageNet data set can be used. For image recognition in specific fields such as medical image recognition, collect the medical image data set labeled in the corresponding field. Preprocess the data set, including data cleaning, that is, removing noise and incorrectly labeled data, data normalization, such as normalizing image pixel values ​​to a specific range, standardizing text data, and dividing the training set, validation set, and test set. The common ratio is 8:1:1, which can be adjusted according to actual conditions to pave the way for subsequent model training.

[0021] S2: Model selection and initialization operations

[0022] Determine the teacher model and student model based on the specific application scenario and task objectives. For example, in image classification tasks, if a large-scale pre-trained model is used as the teacher model, you can choose classic convolutional neural network (CNN) models such as ResNet and VGG. These models have been trained on large data sets such as ImageNet, have strong representation capabilities and rich knowledge; the student model can use lightweight CNN architectures such as MobileNet and ShuffleNet, so that it can absorb the knowledge of the teacher model while achieving lightweightness, which is conducive to the deployment of applications on resource-constrained devices. When initializing the model, the default initialization method (such as the default parameter initialization method of the "nn.Module" class in PyTorch) is usually used in the deep learning framework, or a specific initialization strategy (such as Xavier initialization, He initialization, etc.) is used based on experience to initialize the model weight parameters. For example, Xavier initialization is used for linear layer weight initialization, and its calculation formula is:

[0023]

[0024] Where W represents the weight parameter, n in is the number of input neurons, n out is the number of output neurons, and U represents uniform distribution, which ensures that the parameters are reasonably distributed in the initial stage of model training, accelerates the training convergence speed and improves the training effect. In the process of knowledge distillation, loss calculation and monitoring are required.

[0025] S3: Loss calculation and monitoring during knowledge distillation

[0026] In each round of training iteration, the teacher model and the student model receive the input training data and perform forward propagation operations to obtain their respective output probability distributions (such as the probability values ​​of each category in the classification task). Suppose the output probability distribution of the teacher model is P T (x), the student model output probability distribution is P S (x) (x is the sample corresponding to the input data), the KL divergence calculation formula is When actually calculating the KL divergence value, follow these steps:

[0027] ① Initialize the KL divergence loss value to 0, and then traverse the data batches in the data loader. For each batch of data, get the input data from the data loader, input it into the teacher model and the student model respectively, and get the teacher model output and the student model output.

[0028] ② Use a custom function to calculate KL divergence, pass the teacher model output and student model output as parameters, calculate the KL divergence value of the batch of data, and add it to the total KL divergence loss value.

[0029] ③ Divide the total KL divergence loss value by the number of data batches in the data loader to get the average KL divergence loss value.

[0030] For the prediction results and true labels (obtained from the training data) output by the student model, the cross entropy loss function is used to calculate the probability distribution of the student model prediction as P S (x), the one-hot encoding vector corresponding to the true label is y (the true category position in the multi-classification task is 1, and the rest are 0), and the cross entropy loss calculation formula is H(y, P S )=-∑ i y i logP S (x) i , the cross entropy loss is calculated in the code by the following steps:

[0031] ① Initialize the cross entropy loss value to 0 and create an instance of the cross entropy loss function class that comes with PyTorch.

[0032] ② Traverse the data batches in the data loader. For each batch of data, get the input data and target label from the data loader, input the input data into the student model to get the output, use the cross entropy loss function class instance to calculate the cross entropy loss value of the batch of data, and accumulate it to the total cross entropy loss value.

[0033] ③ Divide the total cross entropy loss value by the number of data batches in the data loader to get the average cross entropy loss value.

[0034] After the calculation is completed, the KL divergence value and the cross entropy loss value are combined according to a certain weight to obtain the final comprehensive loss value that measures the knowledge distillation effect and the model state. Assume that the KL divergence loss weight is α, the cross entropy loss weight is β (α+β=1), and the comprehensive loss L calculation formula is L=αD KL (P T ||P S )+βH(y,P S ), the comprehensive loss value is calculated as follows:

[0035] Set α to 0.5 (can be adjusted according to actual conditions), then β = 1-α. Get the KL divergence loss value and cross entropy loss value calculated previously, multiply the KL divergence loss value by α, multiply the cross entropy loss value by β, and then add the two together to get the comprehensive loss value. Monitor the comprehensive loss value in real time to understand the model training stage and the effectiveness of knowledge distillation.

[0036] S4: PID controller parameter setting and temperature adjustment calculation

[0037] First initialize the PID controller parameters, that is, set the proportional coefficient K p , integral coefficient K i and the differential coefficient K d The values ​​of these coefficients are critical to the effect of temperature parameter adjustment and need to be determined through multiple experiments based on the specific model and data set characteristics. For example, for a simple model and data set K p The value can be relatively large to quickly respond to loss changes. For complex and volatile situations, K should be appropriately reduced. p And reasonably set K i and K d To ensure system stability and adjustment accuracy, the initial setting can refer to the common experience value range (such as K p Between 0.1 and 1, K i Between 0.01 and 0.1, K dThe proportional term generates an adjustment component proportional to the error based on the difference in the current comprehensive loss. Suppose the comprehensive loss value at the current moment is L(t), and the comprehensive loss value at the previous moment is L(t-1), then the error e(t) = L(t)-L(t-1), and the proportional term generates an adjustment component u p (t) is calculated as u p (t) = K p e(t), the steps to calculate the proportional adjustment component are as follows:

[0038] ① Initialize the loss value of the previous round of training to 0. In each round of training iteration, obtain the current comprehensive loss value, and calculate the difference between the current loss value and the previous round of loss value as the error.

[0039] ②Multiply the error by the proportionality factor K p , and obtain the proportional adjustment component.

[0040] ③ Update the current loss value to the previous round loss value for the next round of calculation.

[0041] The integral term eliminates the steady-state error by accumulating the comprehensive loss error over a period of time. Assume that the integral time interval is Δt (determined by the training iteration time interval, such as each training iteration is a time interval unit), and the integral term adjusts the component u i (t) is calculated by accumulating the error e(t), and the calculation formula is The steps to calculate the integral term adjustment component are as follows:

[0042] Initialize the integral sum to 0. In each training iteration, calculate the difference between the current loss value and the previous loss value as the error, and add the error to the integral sum. Then multiply the integral sum by the integral coefficient K i and time interval to obtain the integral term adjustment component.

[0043] The differential term adjusts the predicted loss change trend according to the comprehensive loss change rate. Suppose the loss change rate at the current moment is The approximate calculation is obtained by dividing the difference in losses at adjacent moments by the time interval, that is, The differential term adjusts the component u d (t) is calculated as The steps to calculate the differential adjustment component are as follows:

[0044] Initialize the error value of the previous round to 0. In each training iteration, calculate the difference between the current loss value and the previous loss value as the current error, and calculate the difference between the current error and the previous error value divided by the time interval to get the loss change rate. Then multiply the loss change rate by the differential coefficient K dFinally, the current error value is updated to the error value of the previous round for the next round of calculation.

[0045] Finally, the adjustment components calculated by the proportional term, integral term and differential term are combined to obtain the total temperature adjustment ΔT, which is calculated as ΔT = u p (t)+u i (t)+u d (t).

[0046] S5: Temperature parameter update

[0047] Set the intermediate temperature threshold T mid = 2.0, and the hyperparameter k that controls the steepness of the curve is 0.5. The negative label attention adjustment factor Y is calculated based on the current temperature T:

[0048]

[0049] Then, the current temperature parameter T is updated according to the temperature adjustment amount ΔT to obtain the new temperature parameter T new , the update formula is:

[0050] T new =T+γ·ΔT

[0051] In the code implementation, follow these steps:

[0052] First, obtain the current temperature parameter T and the calculated temperature adjustment value ΔT. Set T mid =2.0 and k=0.5, calculate the negative label attention adjustment factor γ according to the above formula. Then multiply the adjustment factor γ by the temperature adjustment amount ΔT, and add the result to the current temperature parameter T to obtain the new temperature parameter T new .

[0053] S6: Model training iteration

[0054] The updated temperature parameters are applied to the subsequent training process of knowledge distillation, such as being used in some knowledge distillation formulas based on temperature parameter adjustment probability distribution, and then the next round of training iterations is continued, repeating the entire process from model forward propagation, loss calculation, PID controller adjustment of temperature parameters to temperature parameter update and re-training until the training stop conditions are met. Common training stop conditions include reaching a preset number of training rounds (such as 100 rounds, etc.), the accuracy on the verification set reaches a certain threshold (such as 90%, etc.), or the comprehensive loss value converges to a sufficiently small range.

[0055] S7: Model evaluation and validation and application deployment

[0056] First, prepare test set data that is independent of the training set and validation set to ensure that it can represent the data distribution of the actual application scenario and is not involved in the model training process, so as to objectively evaluate the final performance of the model. Then, select appropriate performance evaluation indicators based on the specific deep learning task. For example, for image classification tasks, you can use indicators such as accuracy (Accuracy, that is, the proportion of correctly classified samples to the total number of test samples), recall (Recall, measuring the model's ability to correctly recall positive examples), and F1 value (a comprehensive consideration of the harmonic mean of accuracy and recall). In addition to classification-related indicators, target detection tasks also consider mean average precision (mAP, a comprehensive indicator that measures the accuracy of detection box positioning and classification accuracy). Then, the trained student model is used to predict the test set data and the corresponding values ​​are calculated according to the selected performance indicators to evaluate the model performance. At the same time, it is compared with the model trained by the traditional fixed temperature parameter knowledge distillation method to observe the differences in various performance indicators on the same test set, and verify the advantages of the method of the present invention in improving distillation efficiency and enhancing the generalization ability of the student model. For example, the accuracy of the model in the image classification task under the two methods is compared. If the accuracy of the model trained by the method of the present invention is higher or the training time is shorter under similar accuracy, it proves that the method is effective. Finally, practical application and deployment are carried out. The trained lightweight student model is deployed to the corresponding target device, such as mobile devices (smartphones, tablets, etc.), edge computing devices (Internet of Things gateways, etc.) or servers for online services. When deploying, it is necessary to consider the hardware resource limitations (computing power, memory size, etc.) of the target device and the compatibility of the software environment. The model may need to be further optimized and adapted, such as model quantization (converting floating-point parameters to integers to reduce memory usage and calculation amount), model pruning (removing redundant parts) and other operations. After the model is deployed, it is applied to actual deep learning task scenarios, such as real-time image classification and recognition on smartphones, natural language processing services on servers, etc. During the application process, the model performance and response to different input data are continuously monitored, and feedback information is collected for subsequent optimization of the model or adjustment of knowledge distillation related parameters, so as to continuously improve the effect and adaptability of the model in practical applications. Through the above implementation process, a dynamic adjustment method of temperature parameters for knowledge distillation based on PID controller can be fully realized, which effectively solves the problem of fixed temperature parameters in existing knowledge distillation technology, improves the performance of deep learning models and expands their application scope.

Claims

1. A method for dynamic adjustment of knowledge distillation temperature parameters based on PID controller, characterized in that: include: During the knowledge distillation process, the KL divergence and cross entropy loss between the student model and the teacher model are calculated in real time. The KL divergence is used to characterize the degree of deviation of the output distribution, and the cross entropy loss is used to measure the difference between the predicted and true labels, so as to monitor the model status and distillation effect. Introduce PID controller, according to the preset proportional coefficient K p (value range 0.1-1), integral coefficient K i (value range 0.01-0.1) and differential coefficient Kd (value range 0.01-0.1), calculate the temperature adjustment by integrating the current loss difference, historical accumulation and change rate, among which the proportional term responds quickly according to the real-time loss difference, the integral term eliminates the steady-state error, and the differential term introduces foresight to prevent over-adjustment; The knowledge distillation temperature parameters are updated in real time according to the calculated temperature adjustment amount and applied to the loss function calculation, so that the student model can learn in a dynamic temperature environment, achieve a balance between the KL divergence and the cross entropy loss between the student model and the teacher model, and ensure that the knowledge transfer is fitted with the real data. The entire process does not require manual intervention in temperature adjustment, which is suitable for a variety of deep learning models and data sets, effectively improving model performance and reducing the cost of manual parameter adjustment.

2. The dynamic adjustment method based on PID controller according to claim 1, characterized in that: include: set up The comprehensive loss value at the current moment is L(t), and the previous moment is L(t-1). The calculation error e(t) = L(t)-L(t-1), and the proportional term adjustment component u p (t) = K p e(t); Integral term adjustment component (Δt is the integration time interval); Derivative adjustment component Final temperature adjustment ΔT = u p (t)+u i (t)+u d (t).

3. The method for adjusting the temperature parameters of knowledge distillation according to claim 1, characterized in that: include: Set the intermediate temperature threshold T mid = 2.0, control curve steepness hyperparameter k = 0.5, calculate negative label attention adjustment factor based on current temperature T New temperature parameter T new =T+γ·ΔT, so that the change in the degree of attention to negative labels is taken into account when adjusting the temperature parameter.

4. The method according to claim 1, characterized in that: include: The model training is iterated until the preset number of training rounds (such as 100 rounds) is met, the accuracy of the validation set reaches a certain threshold (such as 90%), or the comprehensive loss value converges to a small enough range, and then stops training; Model evaluation and verification uses a test set independent of the training set and validation set, selects appropriate performance evaluation indicators (such as the accuracy of image classification tasks, etc.) according to the deep learning task, and verifies the advantages of this method in improving distillation efficiency and enhancing the generalization ability of student models by comparing it with the model trained by the traditional fixed temperature parameter knowledge distillation method; In the actual application and deployment stage, the trained lightweight student model is deployed to mobile devices (such as smartphones, tablets), edge computing devices (such as IoT gateways) or servers. During deployment, the hardware resource limitations of the target device (computing power, memory size, etc.) and software environment compatibility are considered, and the model is optimized and adapted (such as model quantization and model pruning). During the model application process, the performance and response to different input data are continuously monitored, and feedback information is collected for subsequent optimization of the model or adjustment of knowledge distillation related parameters.

Images