Multi-model knowledge distillation training method for sensing dynamic temperature based on uncertainty

Abstract

The invention discloses a multi-model knowledge distillation training method for sensing dynamic temperature based on uncertainty, and aims to solve the problems of large multi-model prediction divergence and unstable distillation supervision information in an unbalanced sample data scene by introducing a sample-level uncertainty evaluation mechanism based on multi-teacher model prediction consistency. And carrying out self-adaptive adjustment on temperature parameters in the knowledge distillation process. The method comprises the following steps: acquiring and preprocessing unbalanced sample data; constructing a plurality of mutually independent teacher models and outputting prediction probability distribution; calculating a sample-level uncertainty index based on fusion of multi-teacher prediction results; a dynamic temperature parameter is generated in a self-adaptive mode according to the uncertainty index, and temperature scaling is carried out on teacher model output so as to construct a soft label; and jointly training a target model by using the soft label and a real label. According to the method, differential distillation control is carried out on different uncertain samples, so that the stability of a model training process is improved on the premise of not increasing the complexity of a model structure, and the classification effect in an unbalanced sample data scene is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a multi-model knowledge distillation training method based on uncertainty-perceived dynamic temperature. Background Technology

[0002] With the development of computer technology and machine learning methods, data-driven classification modeling techniques have been widely applied in complex data analysis and intelligent decision-making scenarios. In practical applications, due to factors such as data acquisition costs and differences in sample distribution, training data often exhibits an imbalanced class distribution, meaning that the number of samples in the minority class is significantly less than the number of samples in the majority class. In imbalanced data scenarios, traditional classification models are easily dominated by majority class samples during training, leading to insufficient discrimination ability of the model against minority class samples, thus affecting the overall performance and stability of the model.

[0003] To address the classification problem of imbalanced sample data, existing technologies have proposed various solutions, such as adjusting the sample distribution through oversampling and undersampling. However, these methods usually rely on human experience to set rules or parameters, and their stability is limited under complex data distribution conditions. In particular, when the sample distribution changes or there is a lot of noise, the model performance fluctuates greatly.

[0004] In recent years, knowledge distillation technology has been introduced into the model training process. It utilizes the soft-label information output by the teacher model to supervise the training of the student model, thereby improving model performance while maintaining a relatively simple model structure. To further improve distillation results, existing technologies have proposed multi-teacher knowledge distillation methods. These methods introduce multiple teacher models with different structures or training strategies to reduce the impact of single-model prediction bias on the distillation process. However, in multi-teacher knowledge distillation scenarios, different teacher models often produce significantly divergent predictions for the same sample due to structural differences or different training strategies. Directly performing uniform distillation on these predictions can easily introduce unstable or biased supervisory information.

[0005] Chinese patent document CN113222034A discloses a fine-grained multi-class imbalanced fault classification method based on knowledge distillation. This method combines knowledge distillation and clustering to classify and model imbalanced sample data. However, this method mainly relies on static sample partitioning and a fixed distillation strategy during the distillation process, failing to distinguish the differences in consistency of prediction results among different samples. When there is significant discrepancy in the predictions of multiple models, it may still lead to distortion of soft label information, thereby affecting the training stability of the student model.

[0006] Furthermore, while some existing technologies have introduced dynamic temperature regulation mechanisms to improve the information transfer effect during knowledge distillation, these methods mostly adjust temperature parameters based on training rounds or global statistical information, failing to finely control the distillation intensity by considering the discrepancies in the prediction results of multi-teacher models at the sample level. When there are significant discrepancies in the prediction results of multi-teacher models for the same sample, existing methods struggle to reduce the distillation supervision intensity in a timely manner, which can easily amplify unreliable soft label information and affect the stability of model training.

[0007] Therefore, existing technologies still lack a knowledge distillation training method that can combine multi-teacher model prediction consistency information to implement differentiated distillation control for samples with different prediction uncertainties in imbalanced sample data scenarios, so as to improve the stability of the model training process and classification performance without increasing the complexity of the model structure. Summary of the Invention

[0008] The main objective of this invention is to provide a multi-model knowledge distillation training method based on uncertainty-aware dynamic temperature. Addressing the issues of significant discrepancies in predictions for the same sample by multiple teacher models and insufficient stability of distillation supervision information in imbalanced data scenarios, this invention introduces a sample-level uncertainty assessment mechanism based on the prediction consistency of multiple teacher models. This allows for adaptive adjustment of the temperature parameter during the knowledge distillation process, achieving differentiated distillation control for different samples. Therefore, without increasing the complexity of the model structure, the stability and classification performance of the model training process are improved. The specific technical solution is as follows: A multi-model knowledge distillation training method based on uncertainty-aware dynamic temperature includes the following steps: S1, Obtain the imbalanced sample data to be processed, and preprocess the sample data. The preprocessing includes at least sample feature normalization and sample distribution adjustment for categories with a small number of samples. S2, construct multiple independent teacher models, wherein the teacher models differ in at least one of the following: network structure, parameter initialization method, or training strategy; train each teacher model using preprocessed sample data, and output the predicted probability distribution of the corresponding sample respectively; S3, the predicted probability distributions of multiple teacher models for the same sample are fused, and a sample-level uncertainty index is calculated based on the fused predicted probability distribution to characterize the consistency of predictions of multiple teacher models. The sample-level uncertainty index is used to characterize the statistical dispersion between the prediction results of multiple teacher models. S4. Based on the sample-level uncertainty index, adaptive dynamic temperature parameters are generated for different samples. The dynamic temperature parameters are limited to a preset temperature range, and the prediction output of each teacher model is temperature scaled using the dynamic temperature parameters to obtain the distillation soft label corresponding to the sample, so as to reduce the impact of samples with large prediction discrepancies in the multi-teacher model on the stability of distillation supervision. S5. Based on the distilled soft label and the true label of the sample, the target model is jointly supervised and trained to obtain the target model for classifying imbalanced sample data.

[0009] Furthermore, the sample distribution adjustment in step S1 includes at least one of the following processing methods: oversampling, sample generation, or resampling for samples of categories with a smaller sample size.

[0010] Furthermore, the sample-level uncertainty index in step S3 is obtained by calculating the prediction entropy after fusing the prediction probability distributions output by multiple teacher models; It is used to characterize the degree of disagreement among multiple teacher models in predicting the same sample, and the larger the value of the uncertainty index, the greater the prediction disagreement. The dynamic temperature parameter is generated by mapping the uncertainty index corresponding to the sample to a preset temperature range, which is defined by a preset minimum temperature value and a preset maximum temperature value.

[0011] Furthermore, in step S4, the predicted outputs of multiple teacher models after temperature scaling are weighted and fused to generate distillation soft labels corresponding to the samples.

[0012] Furthermore, in step S5, the training of the target model is optimized based on a joint loss function, which includes a first loss term between the target model's prediction result and the true label, and a second loss term between the target model's output and the distilled soft label.

[0013] Compared with the prior art, the present invention has at least the following beneficial effects: 1. By introducing a sample-level uncertainty assessment mechanism based on the consistency of multi-teacher model predictions, the degree of divergence between different samples in the multi-model prediction results can be effectively characterized, avoiding the soft label information distortion problem caused by using a uniform distillation strategy on uncertain samples, and improving the reliability of distillation supervision information.

[0014] 2. By adaptively generating dynamic temperature parameters based on sample-level uncertainty, differentiated control of the knowledge distillation process is achieved, enabling samples with high prediction uncertainty to obtain smoother supervision information, and samples with low prediction uncertainty to obtain more discriminative supervision information, thereby improving the stability of the model training process.

[0015] 3. Compared with knowledge distillation methods that adjust temperature based on fixed temperature or global statistical information, this invention drives temperature parameter adjustment through sample-level uncertainty, avoiding the problem of over-enhancing erroneous distillation supervision on samples with large discrepancies in multi-teacher predictions, thereby further improving the stability of the distillation process and the reliability of model training. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a structural diagram of the multi-teacher model and student model in the method described in this invention. Detailed Implementation

[0017] To make the technical solution, implementation process, and beneficial effects of the present invention clearer, the present invention will be further described below with reference to specific embodiments. It should be understood that the following embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0018] Example 1 A multi-model knowledge distillation training method based on uncertainty-perceived dynamic temperature, such as Figure 1 As shown, it includes the following steps: S1, Imbalanced Sample Data Acquisition and Preprocessing: In this embodiment, the imbalanced sample dataset to be processed is first acquired. The dataset includes multiple samples, wherein the first... The feature vector of each sample is represented as: The corresponding real category label is represented as The number of samples is unevenly distributed across different categories.

[0019] The sample data is preprocessed, including but not limited to: 1. Normalize or standardize the sample features to eliminate the impact of differences in the dimensions of different features on model training; 2. Clean up abnormal samples or samples with missing values; 3. For categories with a small number of samples, adjust the sample distribution through oversampling, sample generation, or resampling to mitigate the adverse effects of class imbalance on model training.

[0020] S2, Multi-Teacher Model Construction and Prediction: In this embodiment, multiple heterogeneous models are constructed. The teacher models differ in at least one of the following: network structure, parameter initialization method, or training strategy, resulting in diverse prediction results.

[0021] Each teacher model was trained using preprocessed imbalanced sample data. After training, the heterogeneous models were used to train the sample data. Make a prediction and obtain the corresponding prediction probability distribution: (1); in, Indicates the first A heterogeneous model for samples The predicted probability vector.

[0022] In one specific implementation, the number of teacher models can be set according to the computing resource conditions and task complexity, for example, two, three or more; the fusion weights of multiple teacher models can be set in an equal weight manner, or set according to the prediction performance of each teacher model on the validation dataset.

[0023] S3, Calculation of Sample Uncertainty Index: In order to characterize the consistency of prediction results of multiple heterogeneous models for the same sample, this embodiment calculates the sample uncertainty index based on the prediction probability distribution of multiple teacher models.

[0024] First, the predicted probability distributions of multiple teacher models are fused to obtain the sample. Average prediction probability distribution: (2); Based on the fused predicted probability distribution, the predicted entropy of the sample is calculated as a sample-level uncertainty index: (3); in, Indicates the number of categories. Indicates sample Belongs to the The predicted probability of a class.

[0025] Predicting entropy Used to characterize samples from multiple heterogeneous models The degree of uncertainty in the prediction results is indicated by the prediction entropy value. The larger the prediction entropy value, the greater the discrepancy in the model's predictions; the smaller the prediction entropy value, the more consistent the model's prediction results.

[0026] S4, Dynamic Temperature Parameter Generation and Soft Tag Construction: In this embodiment, based on the uncertainty index corresponding to the sample... This process adaptively generates corresponding dynamic temperature parameters for different samples. Specifically, the predicted entropy is mapped to a preset temperature range to obtain the dynamic temperature parameters of the samples. (4); in, and These represent the minimum and maximum values ​​of the prediction entropy in the training samples, respectively.

[0027] Subsequently, the output logits of each teacher model are temperature-scaled using the dynamic temperature parameters to obtain the temperature-adjusted prediction probability distribution: (5); in, Indicates the first A heterogeneous model for samples The output logits.

[0028] Furthermore, the predicted probability distributions of multiple teacher models after temperature scaling are weighted and fused to generate distilled soft labels corresponding to the samples: (6); in, Indicates the first The fusion weights corresponding to each teacher model, and satisfying the following conditions: .

[0029] S5, Target Model Training and Joint Loss Optimization: In this embodiment, a target model is constructed, and the target model is jointly supervised and trained using the real labels of the samples and the distilled soft labels.

[0030] The target model output is: (7); in, Represents the target model.

[0031] The training of the target model is optimized based on a joint loss function, which is defined as follows: (8); in, These are the weighting coefficients used to balance the two losses.

[0032] The first loss term is the supervision loss between the target model's prediction results and the true labels: (9); The second loss term is the distillation loss between the target model output and the distillation soft label.

[0033] By minimizing the joint loss function, the parameters of the target model are updated, ultimately obtaining a trained model for classifying imbalanced sample data.

Claims

1. A multi-model knowledge distillation training method based on uncertainty-perceived dynamic temperature, characterized in that, Includes the following steps: S1, Obtain the imbalanced sample data to be processed, and preprocess the sample data. The preprocessing includes at least sample feature normalization and sample distribution adjustment for categories with a small number of samples. S2, construct multiple independent teacher models, wherein the teacher models differ in at least one of the following: network structure, parameter initialization method, or training strategy; train each teacher model using preprocessed sample data, and output the predicted probability distribution of the corresponding sample respectively; S3, the predicted probability distributions of multiple teacher models for the same sample are fused, and a sample-level uncertainty index is calculated based on the fused predicted probability distribution to characterize the consistency of predictions of multiple teacher models. The sample-level uncertainty index is used to characterize the statistical dispersion between the prediction results of multiple teacher models. S4. Based on the sample-level uncertainty index, adaptive dynamic temperature parameters are generated for different samples. The dynamic temperature parameters are limited to a preset temperature range, and the prediction output of each teacher model is temperature scaled using the dynamic temperature parameters to obtain the distillation soft label corresponding to the sample, so as to reduce the impact of samples with large prediction discrepancies in the multi-teacher model on the stability of distillation supervision. S5. Based on the distilled soft label and the true label of the sample, the target model is jointly supervised and trained to obtain the target model for classifying imbalanced sample data.

2. The multi-model knowledge distillation training method according to claim 1, characterized in that, The sample distribution adjustment in step S1 includes at least one of the following processing methods: oversampling, sample generation, or resampling for samples in categories with a small number of samples.

3. The multi-model knowledge distillation training method according to claim 1, characterized in that, The sample-level uncertainty index in step S3 is obtained by calculating the prediction entropy after fusing the prediction probability distributions output by multiple teacher models.

4. The multi-model knowledge distillation training method according to claim 1, characterized in that, The sample-level uncertainty index is used to characterize the degree of disagreement among multiple teacher models regarding the prediction results of the same sample, and the larger the uncertainty index value, the greater the prediction disagreement.

5. The multi-model knowledge distillation training method according to claim 1, characterized in that, The dynamic temperature parameter is generated by mapping the uncertainty index corresponding to the sample to a preset temperature range, which is defined by a preset minimum temperature value and a preset maximum temperature value.

6. The multi-model knowledge distillation training method according to claim 1, characterized in that, In step S4, the predicted outputs of multiple teacher models after temperature scaling are weighted and fused to generate distillation soft labels corresponding to the samples.

7. The multi-model knowledge distillation training method according to claim 1, characterized in that, In step S5, the training of the target model is optimized based on a joint loss function, which includes a first loss term between the target model's prediction result and the true label, and a second loss term between the target model's output and the distilled soft label.

Images