Logits knowledge distillation method and system based on fuzzy adaptive PID temperature control

Abstract

The invention discloses a Logits knowledge distillation method and system based on fuzzy adaptive PID temperature control. The method comprises the following steps: preprocessing input image data; inputting the image into a student network and a teacher network to respectively obtain student output and teacher output; calculating a knowledge distillation error of the student network and the teacher network to obtain a normalized error; outputting PID parameters according to the defined fuzzy rule and the fuzzy set; the current distillation temperature is calculated, the normalized error serves as error input, and the self-adaptive temperature is obtained; performing KL divergence calculation on output of the student network and the teacher network to obtain knowledge distillation loss, and forming total loss; updating student network parameters to realize knowledge distillation training; and dynamically adjusting the temperature to realize self-adaptive knowledge distillation. According to the method, fuzzy control and a PID regulation mechanism are introduced into distillation temperature regulation and control, so that the distillation process has dynamic response capability, and the stability and convergence performance of a student network are effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and involves the combination of deep learning and model compression technologies. Specifically, it relates to a Logits knowledge distillation method and system based on fuzzy adaptive PID temperature control. Background Technology

[0002] Knowledge distillation (KD) is an important method in deep learning to improve the performance of lightweight models. It is a key step in achieving efficient model transfer and intelligent optimization, and also an important link in promoting the practical application of artificial intelligence technology. In recent years, academia and industry have been continuously exploring optimization strategies for knowledge distillation to improve the learning effect of student models on teacher model knowledge.

[0003] However, current distillation processes still face numerous challenges. In traditional knowledge distillation, the temperature parameter is typically set to a fixed value to balance the soft label distribution between teacher and student models. While a fixed temperature is computationally simple, it cannot dynamically adapt to changes in distillation loss at different training stages and sample difficulty levels. This can easily lead to excessively large gradients in the early stages of training or learning stagnation in later stages, affecting overall convergence and model performance. Therefore, designing an adaptive temperature regulation mechanism to achieve intelligent and dynamic optimization of the knowledge distillation process has become an urgent problem to solve. To address these challenges, leveraging the combined advantages of fuzzy control and PID control techniques has become an important research direction. Fuzzy control possesses the ability to perform adaptive reasoning for uncertain systems, while PID control can achieve continuous and stable error correction. Combining the two provides a new technical approach for establishing a dynamic temperature control mechanism.

[0004] Current research methods mainly achieve distillation temperature control through fixed temperature or simple linear decay strategies. Although this improves training stability to some extent, it still has several bottlenecks in actual distillation: First, fixed or static temperature mechanisms cannot dynamically respond to changes in distillation loss based on real-time errors, resulting in insufficient teacher information transmission efficiency and limited student model learning ability; Second, although traditional PID control can make dynamic adjustments, the manual setting of parameters is highly sensitive to system stability and is difficult to adapt to complex nonlinear distillation scenarios. Summary of the Invention

[0005] Purpose of the invention: To address the problems existing in current knowledge distillation techniques, such as fixed distillation temperature, insufficient matching degree between teacher and student features, and poor stability of the distillation process, this invention provides a Logits knowledge distillation method and system based on fuzzy adaptive PID temperature control. By introducing fuzzy control and PID adjustment mechanism into the distillation temperature regulation, the distillation process has dynamic response capability, effectively improving the stability and convergence performance of the student network, and has good interpretability and versatility.

[0006] Technical Solution: To achieve the above objectives, this invention provides a Logits knowledge distillation method based on fuzzy adaptive PID temperature control, comprising the following steps:

[0007] S1: Preprocess the input image data;

[0008] S2: Input the preprocessed image into the student network and the teacher network to obtain the student output logits_student and the teacher output logits_teacher, respectively.

[0009] S3: Based on the student output logits_student and the teacher output logits_teacher, calculate the knowledge distillation error between the student network and the teacher network, and dynamically normalize it through a sliding window to obtain the normalized error;

[0010] S4: Input the normalized error into the fuzzy PID temperature controller, and output the PID parameters according to the defined fuzzy rules and fuzzy sets;

[0011] S5: Calculate the current distillation temperature using the PID parameters from step S4, take the normalized error as the error input, perform integral, derivative and proportional calculations, and normalize it using the Sigmoid function before mapping it to the specified temperature range to obtain the adaptive distillation temperature;

[0012] S6: Using the adaptive distillation temperature calculated in step S5, perform KL divergence calculation on the outputs of the student network and the teacher network to obtain the knowledge distillation loss, and combine it with the cross-entropy loss of the student network and the real label to form the total loss.

[0013] S7: Based on the total loss calculated in step S6, update the student network parameters through backpropagation to achieve knowledge distillation training;

[0014] S8: Repeat steps S3 to S7 in each training iteration, and use a fuzzy PID controller to dynamically adjust the temperature to achieve adaptive knowledge distillation.

[0015] Furthermore, the formula for calculating the knowledge distillation error in step S3 is as follows:

[0016]

[0017] in, For temperature coefficient, for The function `teacher` represents the network output for teachers, and `Student` represents the network output for students.

[0018] Furthermore, in step S3, the current error is linearly mapped to the [0,1] interval using historical error values ​​within the sliding window, eliminating the influence of differences in the absolute value of the error. The calculation formula for the dynamically normalized error is as follows:

[0019]

[0020] in, and These represent the maximum and minimum errors, respectively.

[0021] Furthermore, the fuzzy set defined in step S4 is:

[0022] The domain of the variable error is [0,7]. Seven equally spaced sampling points are generated, and the fuzzy set is {S, M, L}, which represent three fuzzy sizes: "small", "medium", and "large", respectively.

[0023] The domain of the variable delta_error is [-2,2]. Seven equally spaced sampling points are generated, and the fuzzy sets are {N, Z, P}, which represent "negative number, zero, and positive number" respectively.

[0024] Furthermore, the PID parameters in step S4 include proportional coefficients. Integral coefficient With differential coefficients Each of them generates 7 equally spaced sampling points, and the fuzzy sets are {L, M, H}.

[0025] Furthermore, the formula for calculating the adaptive temperature in step S5 is as follows:

[0026]

[0027]

[0028] in, The length of the sliding window. This is the previous error value. For adaptive distillation temperature; The temperature prediction value is obtained based on PID control. and These represent the maximum and minimum distillation temperatures, respectively. Let be the error value at the i-th sampling time within the sliding window.

[0029] Further, step S6 specifically includes:

[0030] A1: Input the output logits of the teacher network and the student network into the normalization module respectively to ensure that the distribution scale of the two is consistent;

[0031] A2: Divide the logits of both by the temperature coefficient. The probability distribution of the teacher model and the prediction distribution of the student model are obtained through the Softmax function.

[0032] A3: The KL divergence is used to measure the degree of fit between the student distribution and the teacher distribution, thus forming the distillation loss:

[0033]

[0034] in, and The outputs of logits for the teacher model and the student model are respectively. For temperature coefficient, This represents the KL divergence, used to measure the difference between the probability distributions of two entities.

[0035] A4: Distillation loss Multiply by the square of the temperature To maintain gradient scale balance and achieve stable knowledge transfer under temperature regulation:

[0036]

[0037] This method improves the learning ability of the student model by preserving the soft label information of the teacher model through temperature smoothing.

[0038] Furthermore, in step S7, the gradient of the student network is pruned to limit its norm to a preset range to ensure the stability of the training process; the parameters of the student network are updated using stochastic gradient descent, and the parameter update is completed based on the combined effect of the learning rate and the weight decay term.

[0039] Furthermore, the update formula for the student network parameters in step S7 is as follows:

[0040]

[0041] in, Indicates student network parameters, Indicates the learning rate. Represents cross-entropy loss, and These represent the weight coefficients for cross-entropy loss and knowledge distillation loss, respectively.

[0042] This invention also provides a Logits knowledge distillation system based on fuzzy adaptive PID temperature control, comprising:

[0043] The preprocessing module is used to preprocess the input image data;

[0044] The network output module is used to input the preprocessed image into the student network and the teacher network, and obtain the student output logits_student and the teacher output logits_teacher, respectively.

[0045] The normalization module is used to calculate the knowledge distillation error between the student network and the teacher network, and to perform dynamic normalization through a sliding window to obtain the normalized error.

[0046] The PID parameter output module inputs the normalized error into the fuzzy PID temperature controller and outputs PID parameters according to the defined fuzzy rules and fuzzy sets.

[0047] The adaptive temperature module takes the normalized error as the error input, performs integral, differential and proportional calculations, and then normalizes it using the Sigmoid function before mapping it to a specified temperature range to obtain the adaptive temperature.

[0048] The total loss module calculates the knowledge distillation loss by performing KL divergence on the outputs of the student network and the teacher network, and combines the cross-entropy loss of the student network and the real labels to form the total loss.

[0049] The knowledge distillation training module updates student network parameters through backpropagation to achieve knowledge distillation training.

[0050] The dynamic adjustment module uses a fuzzy PID controller to dynamically adjust the temperature, achieving adaptive knowledge distillation.

[0051] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0052] 1. This invention introduces a fuzzy adaptive PID control mechanism, which can adjust the temperature in real time according to the distillation error, realize adaptive knowledge distillation with dynamic temperature regulation, and improve the distillation stability and model convergence speed.

[0053] 2. The fuzzy PID tuner proposed in this invention uses fuzzy rules to automatically adjust the proportional, integral and derivative parameters, making the temperature change smoother and avoiding the problems of insufficient information transmission or overfitting in traditional fixed-temperature distillation.

[0054] 3. This invention enables efficient transfer of teachers' knowledge in complex task scenarios, significantly improves students' network performance, and has good versatility. It can be applied to various visual tasks such as image classification, object detection, and semantic segmentation, and has broad application prospects. Attached Figure Description

[0055] Figure 1 This is a flowchart of the method of the present invention;

[0056] Figure 2 The results are visualizations of t-SNE, with the left figure showing the KD method and the right figure showing the method of this invention.

[0057] Figure 3 The method visualizes the difference in teacher-student logits across different categories. The left figure shows the KD method, and the right figure shows the method of this invention. Detailed Implementation

[0058] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0059] Example 1:

[0060] like Figure 1 As shown, this embodiment provides a Logits knowledge distillation method based on fuzzy adaptive PID temperature control, including the following steps:

[0061] S1: Preprocess the input image data;

[0062] First, data augmentation operations are performed, including random horizontal flipping and random cropping, to enhance data diversity. Then, pixel value normalization is performed to convert pixel values ​​in the range of [0,255] to the range of [0,1]. Finally, standardization is performed using the statistical parameters of the dataset (mean, standard deviation) to ensure the uniformity of the network input.

[0063] S2: Input the preprocessed image into the student network and the teacher network to obtain the student output logits_student and the teacher output logits_teacher, respectively.

[0064] The student and teacher networks perform forward computation. The teacher network maintains `eval()` mode throughout training to ensure consistent behavior of the BatchNorm and Dropout layers. The `torch.no_grad()` context manager is used to wrap the teacher network's forward propagation process, avoiding the computation and storage of gradient information, saving GPU memory and improving computational efficiency. The student network maintains `train()` mode, allowing gradient computation and parameter updates. During the forward propagation, all intermediate feature maps and activation values ​​retain gradient information for subsequent backpropagation. The forward propagation of the two networks yields the student output `logits_student` and the teacher output `logits_teacher`, respectively. These raw outputs, without softmax processing, contain the class discrimination information learned by the networks.

[0065] S3: Based on the student output logits_student and the teacher output logits_teacher, calculate the knowledge distillation error between the student network and the teacher network, and dynamically normalize it through a sliding window to obtain the normalized error;

[0066] The formula for calculating the knowledge distillation error is:

[0067]

[0068] in, For temperature coefficient, for The function `teacher` represents the network output for teachers, and `Student` represents the network output for students.

[0069] By using historical error values ​​within a sliding window, the current error is linearly mapped to the [0,1] interval, eliminating the influence of differences in absolute error values ​​and obtaining the normalized error. The formula for calculating the dynamic normalized error is as follows:

[0070]

[0071] in, and These represent the maximum and minimum errors, respectively.

[0072] S4: Input the normalized error into the fuzzy PID temperature controller, and output the PID parameters according to the defined fuzzy rules and fuzzy sets;

[0073] The defined fuzzy set is:

[0074] The domain of the variable error is [0,7]. Seven equally spaced sampling points are generated, and the fuzzy set is {S, M, L}, which represent three fuzzy sizes: "small", "medium", and "large", respectively.

[0075] The domain of the variable delta_error is [-2,2]. Seven equally spaced sampling points are generated, and the fuzzy sets are {N, Z, P}, which represent "negative number, zero, and positive number" respectively.

[0076] The defined fuzzy rules are:

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086] PID parameters include proportional coefficient. Integral coefficient With differential coefficients Each of them generates 7 equally spaced sampling points, and the fuzzy sets are {L, M, H}.

[0087] S5: Calculate the current distillation temperature using the PID parameters from step S4, take the normalized error as the error input, perform integral, derivative and proportional calculations, and normalize it using the Sigmoid function before mapping it to the specified temperature range to obtain the adaptive temperature.

[0088] The formula for calculating the adaptive temperature is as follows:

[0089]

[0090]

[0091] in, The length of the sliding window. This is the previous error value. For adaptive distillation temperature; The temperature prediction value is obtained based on PID control. and These represent the maximum and minimum distillation temperatures, respectively. Let be the error value at the i-th sampling time within the sliding window.

[0092] S6: Using the adaptive distillation temperature calculated in step S5, perform KL divergence calculation on the outputs of the student network and the teacher network to obtain the knowledge distillation loss, and combine it with the cross-entropy loss of the student network and the real label to form the total loss.

[0093] Step S6 specifically includes:

[0094] A1: Input the output logits of the teacher network and the student network into the normalization module respectively to ensure that the distribution scale of the two is consistent;

[0095] A2: Divide the logits of both by the temperature coefficient. The probability distribution of the teacher model and the prediction distribution of the student model are obtained through the Softmax function.

[0096] A3: The KL divergence is used to measure the degree of fit between the student distribution and the teacher distribution, thus forming the distillation loss:

[0097]

[0098] in, and The outputs of logits for the teacher model and the student model are respectively. For temperature coefficient, This represents the KL divergence, used to measure the difference between the probability distributions of two entities.

[0099] A4: Distillation loss Multiply by the square of the temperature To maintain gradient scale balance and achieve stable knowledge transfer under temperature regulation:

[0100]

[0101] This method improves the learning ability of the student model by preserving the soft label information of the teacher model through temperature smoothing.

[0102] S7: Based on the total loss calculated in step S6, update the student network parameters through backpropagation to achieve knowledge distillation training. The update formula is as follows:

[0103]

[0104] in, Indicates student network parameters, Indicates the learning rate. Represents cross-entropy loss, and These represent the weight coefficients for cross-entropy loss and knowledge distillation loss, respectively.

[0105] The backpropagation algorithm is invoked to calculate the gradient of the total loss with respect to all learnable parameters of the student network. During this process, the parameters of the teacher network remain frozen, and the parameters of the adaptive temperature controller are not adjusted through gradient calculation but are updated independently by the PID control algorithm. To prevent gradient explosion, the gradients of the student network are pruned, limiting their norm to a preset range to ensure the stability of the training process. Subsequently, stochastic gradient descent is used to update the student network parameters, with parameter updates based on the combined effect of the learning rate and weight decay terms. The learning rate is gradually reduced during training according to a set scheduling strategy to improve the model's convergence efficiency and generalization ability. In some stages, warm-up and cosine decay are used to achieve smooth changes in the learning rate, thereby obtaining better training results.

[0106] S8: Repeat steps S3 to S7 in each training iteration. In each training batch, strictly follow the order of steps: error calculation → PID tuning → temperature update → loss calculation → parameter update. Normalize the error to output new PID parameters for the fuzzy PID temperature controller, thereby dynamically adjusting the adaptive temperature and realizing the cyclic training of knowledge distillation for temperature adaptive adjustment.

[0107] In summary, the method of this invention introduces a dynamic temperature regulation mechanism combining fuzzy control and a PID controller during the distillation process of the teacher and student networks to achieve adaptive adjustment of the distillation temperature. Specifically, this invention first uses a sliding window mechanism to dynamically normalize the distillation loss error between the outputs of the teacher and student models, and uses the error and its rate of change as input to the fuzzy controller. The proportional, integral, and derivative coefficients in the PID controller are dynamically adjusted using a predefined fuzzy rule base. In each training round, the PID controller predicts the new distillation temperature in real time based on the updated parameters, thus achieving continuous adaptive temperature change during the distillation process. This method increases the temperature in the early stages of distillation to enhance dark knowledge learning and decreases the temperature in the later stages to focus on the main category features, thereby effectively improving the convergence speed and generalization performance of the student model. Experimental results show that this invention outperforms traditional methods with fixed temperatures or manual parameter tuning in terms of stability, robustness, and accuracy improvement.

[0108] Example 2:

[0109] This embodiment provides a Logits knowledge distillation system based on fuzzy adaptive PID temperature control, including:

[0110] The preprocessing module is used to preprocess the input image data;

[0111] The network output module is used to input the preprocessed image into the student network and the teacher network, and obtain the student output logits_student and the teacher output logits_teacher, respectively.

[0112] The normalization module is used to calculate the knowledge distillation error between the student network and the teacher network, and to perform dynamic normalization through a sliding window to obtain the normalized error.

[0113] The PID parameter output module inputs the normalized error into the fuzzy PID temperature controller and outputs PID parameters according to the defined fuzzy rules and fuzzy sets.

[0114] The adaptive temperature module takes the normalized error as the error input, performs integral, differential and proportional calculations, and then normalizes it using the Sigmoid function before mapping it to a specified temperature range to obtain the adaptive temperature.

[0115] The total loss module calculates the knowledge distillation loss by performing KL divergence on the outputs of the student network and the teacher network, and combines the cross-entropy loss of the student network and the real labels to form the total loss.

[0116] The knowledge distillation training module updates student network parameters through backpropagation to achieve knowledge distillation training.

[0117] The dynamic adjustment module uses a fuzzy PID controller to dynamically adjust the temperature, achieving adaptive knowledge distillation.

[0118] Example 3:

[0119] To verify the effectiveness and impact of this invention, this embodiment applies the method to the common classification datasets CIFAR-100 and ImageNet for experiments. The CIFAR-100 dataset contains 100 categories and a total of 60,000 32×32 color images, with 600 images per category. The dataset features more detailed category divisions and smaller inter-class feature differences, placing higher demands on the model's feature extraction and classification capabilities. The ImageNet dataset is a classic large-scale image classification dataset in computer vision, containing over 14 million images, of which approximately 1 million are labeled for training. Its 1000 categories cover a wide range of objects, from animals and plants to everyday items. While the resolution of each image varies, most models uniformly adjust them to 224×224 or 256×256 to facilitate processing by convolutional neural networks. The ImageNet dataset is not only large in scale but also exhibits significant inter-class differences, along with numerous complex backgrounds and multi-scale objects. To measure the actual effectiveness of the method and system of this invention, this experiment uses only classification accuracy as the core evaluation metric. Classification accuracy is calculated by taking the proportion of correctly predicted samples out of the total number of samples. The higher the value of this indicator, the better the classification performance and the stronger the system's ability to process complex category data.

[0120] The results of distillation on the CIFAR-100 dataset using the method of this invention are shown in Tables 1 and 2.

[0121] Table 1 - Performance data of the first batch of datasets

[0122]

[0123] Table 2 - Performance data of the second batch of datasets

[0124]

[0125] Tables 1 and 2 show the performance comparison of different Teacher models, Student models, and the Vanilla KD (conventional knowledge distillation) method with Ours (the method of this invention) on homogeneous and heterogeneous models. In all groups, the performance of the method of this invention is better than that of the Vanilla KD method, and only slightly lower in a few groups, which shows the overall advantage of the method of this invention in knowledge distillation tasks.

[0126] Table 3 shows a comparison of the performance of five basic distillation methods (PKT, SP, RC, RLD, DKD) and their combinations with "Ours (the method of this invention)".

[0127] Table 3 - Performance data of combining the basic distillation method with the method of this invention

[0128]

[0129] As can be seen from Table 3, in the vast majority of groups, the performance of the version with the method of the present invention is better than the corresponding basic distillation method, which shows the overall enhancement effect of the method of the present invention on the performance of basic distillation.

[0130] The effects of using different sliding windows in the method of this invention on the performance of the model after distillation are shown in Table 4.

[0131] Table 4 - Performance data for different sliding windows

[0132]

[0133] As can be seen from Table 4, the model exhibits the best performance when the sliding window size is 100.

[0134] The effects of different fuzzy functions (Triangular / Trapezoidal / Gaussian) on the distillation effect are shown in Table 5.

[0135] Table 5 - Performance data corresponding to fuzzy functions

[0136]

[0137] As can be seen from Table 5, the Triangular fuzzy function performs best on almost all models, while Trapezoidal generally performs slightly worse, especially on ResNet56 where it drops significantly. Gaussian performs close to Triangular on ResNet32×4, but is slightly lower on WRN-40-2 and ResNet110.

[0138] The effects of different PID parameter combinations on the model distillation performance are shown in Table 6. In the table, "√" indicates that the corresponding PID parameter is enabled.

[0139] Table 6 - Performance data corresponding to PID parameter combinations

[0140]

[0141] As can be seen from Table 6, the present invention exhibits the best performance when using the three parameters P, I, and D simultaneously.

[0142] To verify the universality of the method of the present invention, experiments were conducted on the ImageNet dataset to test the KD, RLD and DKD methods, as shown in Table 7.

[0143] Table 7 - General Experimental Data

[0144]

[0145] Compared to the smaller CIFAR-100 dataset, ImageNet offers more categories and higher resolution, making the task more challenging. The experimental results in Table 7 show that the model combined with PID achieves stable performance improvements under KD, RLD, and DKD, indicating that the PID proposed in this invention has universal applicability for knowledge distillation.

[0146] To visually demonstrate the superiority of the method of this invention, such as... Figure 2 The figure shows the t-SNE visualization results of student network features under different methods. The left figure represents the student feature distribution trained using the traditional Knowledge Distillation (KD) method, while the right figure represents the student feature distribution trained using the method of this invention. It can be observed that, compared to the traditional KD method, the method of this invention can form a more compact intra-class distribution and clearer inter-class boundaries in the feature space. This indicates that through the adaptive temperature adjustment mechanism, the student network can more fully learn the discriminative knowledge of the teacher network, thereby improving its feature representation ability.

[0147] Figure 3 This visualization shows the differences in logits output by the teacher and student networks under different category conditions. The left figure shows the results of the traditional KD method, and the right figure shows the results of the method proposed in this invention. Figure 3 As can be seen, the traditional KD method has a large difference between teacher and student outputs in some categories. However, the method of this invention can effectively reduce the distribution difference of teacher-student logits in different categories, making the output of the student network in each category closer to that of the teacher network, thereby improving the stability and consistency of the distillation process.

[0148] Combination Figure 2 and Figure 3 As can be seen, this invention, through a fuzzy PID temperature adaptive adjustment mechanism based on distillation error, not only enhances the class discrimination ability of the student network at the feature space level, but also effectively reduces the prediction difference between teachers and students at the output level, thereby achieving more stable and efficient knowledge distillation.

Claims

1. A Logits knowledge distillation method based on fuzzy adaptive PID temperature control, characterized in that, Includes the following steps: S1: Preprocess the input image data; S2: Input the preprocessed image into the student network and the teacher network to obtain the student output logits_student and the teacher output logits_teacher, respectively. S3: Based on the student output logits_student and the teacher output logits_teacher, calculate the knowledge distillation error between the student network and the teacher network, and dynamically normalize it through a sliding window to obtain the normalized error; S4: Input the normalized error into the fuzzy PID temperature controller, and output the PID parameters according to the defined fuzzy rules and fuzzy sets; S5: Calculate the current distillation temperature using the PID parameters from step S4, take the normalized error as the error input, perform integral, derivative and proportional calculations, and normalize it using the Sigmoid function before mapping it to the specified temperature range to obtain the adaptive distillation temperature; S6: Using the adaptive distillation temperature calculated in step S5, perform KL divergence calculation on the outputs of the student network and the teacher network to obtain the knowledge distillation loss, and combine it with the cross-entropy loss of the student network and the real label to form the total loss. S7: Based on the total loss calculated in step S6, update the student network parameters through backpropagation to achieve knowledge distillation training; S8: Repeat steps S3 to S7 in each training iteration, and use a fuzzy PID controller to dynamically adjust the temperature to achieve adaptive knowledge distillation.

2. The Logits knowledge distillation method based on fuzzy adaptive PID temperature control according to claim 1, characterized in that, The formula for calculating the knowledge distillation error in step S3 is as follows: ； in, For temperature coefficient, for The function `teacher` represents the network output for teachers, and `Student` represents the network output for students.

3. The Logits knowledge distillation method based on fuzzy adaptive PID temperature control according to claim 2, characterized in that, In step S3, the current error is linearly mapped to the [0,1] interval using historical error values ​​within the sliding window, eliminating the influence of differences in absolute error values. The formula for calculating the dynamically normalized error is as follows: ； in, and These represent the maximum and minimum errors, respectively.

4. The Logits knowledge distillation method based on fuzzy adaptive PID temperature control according to claim 3, characterized in that, The fuzzy set defined in step S4 is: The domain of the variable error is [0,7]. Seven equally spaced sampling points are generated, and the fuzzy set is {S, M, L}, which represent three fuzzy sizes: "small", "medium", and "large", respectively. The domain of the variable delta_error is [-2,2]. Seven equally spaced sampling points are generated, and the fuzzy sets are {N, Z, P}, which represent "negative number, zero, and positive number" respectively.

5. The Logits knowledge distillation method based on fuzzy adaptive PID temperature control according to claim 4, characterized in that, The PID parameters in step S4 include the proportional coefficient. Integral coefficient With differential coefficients Each of them generates 7 equally spaced sampling points, and the fuzzy sets are {L, M, H}.

6. The Logits knowledge distillation method based on fuzzy adaptive PID temperature control according to claim 5, characterized in that, The formula for calculating the adaptive distillation temperature in step S5 is as follows: ； ； in, The length of the sliding window. This is the previous error value. For adaptive distillation temperature; The temperature prediction value is obtained based on PID. and These represent the maximum and minimum distillation temperatures, respectively. Let be the error value at the i-th sampling time within the sliding window.

7. The Logits knowledge distillation method based on fuzzy adaptive PID temperature control according to claim 6, characterized in that, Step S6 specifically includes: A1: Input the output logits of the teacher network and the student network into the normalization module respectively to ensure that the distribution scale of the two is consistent; A2: Divide the logits of both by the temperature coefficient. The probability distribution of the teacher model and the prediction distribution of the student model are obtained through the Softmax function. A3: The KL divergence is used to measure the degree of fit between the student distribution and the teacher distribution, thus forming the distillation loss: ； in, and The outputs of logits for the teacher model and the student model are respectively. For temperature coefficient, This represents the KL divergence, used to measure the difference between the probability distributions of two entities. A4: Distillation loss Multiply by the square of the temperature To maintain gradient scale balance and achieve stable knowledge transfer under temperature regulation: 。 8. The Logits knowledge distillation method based on fuzzy adaptive PID temperature control according to claim 7, characterized in that, In step S7, the gradient of the student network is pruned to limit its norm to a preset range to ensure the stability of the training process; the parameters of the student network are updated using stochastic gradient descent, and the parameter update is completed based on the combined effect of the learning rate and the weight decay term.

9. The Logits knowledge distillation method based on fuzzy adaptive PID temperature control according to claim 8, characterized in that, The update formula for student network parameters in step S7 is as follows: ； in, Indicates student network parameters, Indicates the learning rate. Represents cross-entropy loss, and These represent the weight coefficients for cross-entropy loss and knowledge distillation loss, respectively.

10. A Logits knowledge distillation system based on fuzzy adaptive PID temperature control, characterized in that, For implementing the method of claim 1, the system comprises: The preprocessing module is used to preprocess the input image data; The network output module is used to input the preprocessed image into the student network and the teacher network, and obtain the student output logits_student and the teacher output logits_teacher, respectively. The normalization module is used to calculate the knowledge distillation error between the student network and the teacher network, and to perform dynamic normalization through a sliding window to obtain the normalized error. The PID parameter output module inputs the normalized error into the fuzzy PID temperature controller and outputs PID parameters according to the defined fuzzy rules and fuzzy sets. The adaptive temperature module takes the normalized error as the error input, performs integral, differential and proportional calculations, and then normalizes it using the Sigmoid function before mapping it to a specified temperature range to obtain the adaptive temperature. The total loss module calculates the knowledge distillation loss by performing KL divergence on the outputs of the student network and the teacher network, and combines the cross-entropy loss of the student network and the real labels to form the total loss. The knowledge distillation training module updates student network parameters through backpropagation to achieve knowledge distillation training. The dynamic adjustment module uses a fuzzy PID controller to dynamically adjust the temperature, achieving adaptive knowledge distillation.

Images