Temperature regulation and control parameter estimation method for distribution offset data

Abstract

The invention discloses a temperature regulation and control parameter estimation method for distribution offset data. The method comprises the following steps: firstly, dividing marked source field data into a training set and a verification set; dividing unmarked target domain data into a test set and an anchor point set; training the model by using a source field data training set, so that the model has good prediction capability of the source field; gaussian noise is added to a source domain verification set, two different alignment means are adopted to enable the source domain verification set and an unmarked target domain anchor point set to have approximate estimated temperature, and the following two alignment means specifically include the first step of minimizing the maximum confidence coefficient in the two classes or the entropy difference of output distribution, and the second step of minimizing the distribution difference of the two classes; the aligned source field verification set is called as an alternative verification set, and the optimal temperature for temperature scaling is searched on the alternative verification set by using a search technology; the obtained optimal temperature is used for post-event calibration of a model evaluation target domain test set. Compared with an existing method, the method is obviously optimized.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and machine learning technology, and in particular designs a temperature control parameter estimation method for distribution offset data. Background Art

[0002] Cutting-edge deep neural networks have been shown to suffer from poor confidence calibration. Existing approaches for uncertainty estimation and confidence calibration in deep neural networks are primarily categorized into two types. One uses regularization techniques to penalize the confidence of deep neural network outputs during training to prevent the model from becoming overconfident. The other considers further processing of the model output after training, known as post-hoc calibration. In the case of domain shift, the uncertainty calibration problem becomes even more complex, and research has shown that existing post-hoc calibration methods cannot be directly applied in out-of-domain (OOD) settings.

[0003] Calibration method in the training phase: Confidence calibration in the training phase mainly uses some regularization methods. Representative methods include: label smoothing technology, which can effectively improve the calibration problem of the neural network; FocalLoss - a new loss function that can effectively change the overconfidence problem of the neural network.

[0004] Post-processing calibration methods: Representative methods for confidence calibration in the post-processing stage include: Temperature Scaling (TS) method, which improves the confidence calibration performance by adjusting the temperature value of the neural network softmax layer; Histogram Binning (HB) method, which effectively improves the confidence performance of the prediction model by assigning specific confidence values to samples in a specific confidence interval. Many new post-calibration methods have emerged for out-of-domain (OOD) situations, among which the representative ones include: synthesizing out-of-domain (OOD) data in the source domain validation set before the post-calibration stage to analogize existing post-calibration methods to out-of-domain (OOD) conditions; using multiple source domains as validation set data, but determining the post-calibration parameters in an instance-dependent manner, where the parameters of the test instance (e.g., temperature in temperature scaling) depend on its nearest instance cluster in the validation set; adaptively integrating two predefined calibrators from the in-domain (ID) and out-of-domain (OOD) for calibration. However, these existing methods do not take into account the complexity of calibration under out-of-domain (OOD) conditions, and the calibration performance varies greatly in different target areas, making them not adaptable.

[0005] Therefore, in order to address some problems existing in the existing technology, such as (1) due to the different input distributions of the source domain and target domain data, the post-calibration method performed on the in-domain data cannot be directly transferred to the target domain, and the performance of the post-calibration method directly used on the out-of-domain (OOD) data is poor. (2) The current calibration method designed for out-of-domain (OOD) does not fully consider the complexity of the target domain data, and the calibration performance for different target domains varies greatly, and its adaptability is poor. It is urgent to propose a temperature control parameter estimation method for distribution offset data to improve the accuracy of calibration. Summary of the Invention

[0006] To solve the above problems, the present invention discloses a temperature control parameter estimation method for distribution-shifted data. The post-confidence calibration of the model under domain shift conditions is significantly optimized compared with existing methods.

[0007] A method for estimating temperature control parameters for distribution offset data, the method specifically comprising the following steps:

[0008] Step 1: Obtain N training data from the source domain training set, and label information corresponding to each of the N training data, where the label information is used to represent the category to which the training data belongs, and N is an integer greater than 1;

[0009] Step 2: Extract n data and the corresponding label information of each data from the N training data as the validation set, and the remaining data as the training set;

[0010] Step 3: Obtain M unlabeled information data from the target domain dataset;

[0011] Step 4: Extract m data from the M training data as the anchor set, and the remaining data as the test set;

[0012] Step 5: Train the model using the training set to obtain a model f with good prediction accuracy for the source domain;

[0013] Step 6: Process the model f through the BN layer to obtain the adjusted model

[0014] Step 7: Add Gaussian noise to the validation set, and then align it with the anchor set using the maximum confidence score or output distribution entropy as a measurement, or by minimizing the distribution difference by using the maximum mean difference (MMD). The aligned validation set is used as the replacement set.

[0015] Step 8: The alignment operation is to search on the validation set and the anchor set to obtain a temperature that is very similar to that used for confidence calibration;

[0016] Step 9: Searching for the optimal temperature on the alternative set that can minimize the expected calibration error ECE of the model;

[0017] Step 10: Using the optimal temperature, predict the output using the temperature scaled calibration model.

[0018] Furthermore, step 5 includes adjusting the parameters of the BN layer according to the statistical mean and variance of the target domain data to normalize the data.

[0019] Furthermore, the operation used in step 7 to align the anchor set and the validation set includes using the model adjusted by the BN layer. Training anchor set data.

[0020] Furthermore, the operation for aligning the anchor set and the validation set in step 7 includes training the validation set with Gaussian noise added using the model f.

[0021] Furthermore, in step 7, the maximum confidence in the class is used as a measurement method, including the mean square error (MSE) of the maximum confidence of samples of different categories between the anchor set and the training set.

[0022] Furthermore, in step 7, the entropy of the output distribution is used as a measurement method, including the mean square error (MSE) of the output means of samples of different categories between the anchor set and the training set.

[0023] Furthermore, the operation used to align the anchor set and the validation set in step 7 includes adjusting the parameter value β added to the Gaussian noise N(0,βI) to minimize the difference in maximum confidence of different categories between the anchor set and the training set.

[0024] Furthermore, the operation used to align the anchor set and the validation set in step 7 includes adjusting the parameter value β added to the Gaussian noise N(0,βI) to minimize the difference in the output mean of samples of different categories between the anchor set and the training set.

[0025] Furthermore, the operations used to align the anchor set and the validation set include adjusting the parameter value β in the Gaussian noise N(0,βI), calculating the maximum mean difference MMD loss in each mini-batch training iteration using the validation set and anchor set data, and using the stochastic gradient descent SGD optimizer while ensuring that β remains non-negative to obtain the parameter value β that minimizes the maximum mean difference MMD loss.

[0026] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements calibration steps of a temperature control parameter estimation method for distribution offset data.

[0027] Beneficial effects of the present invention:

[0028] (1) This technology achieves superior confidence calibration performance on existing benchmark datasets and is highly valuable for classification scenarios that require more robust uncertainty estimation.

[0029] (2) It has certain adaptability to the confidence calibration problem in different target fields and shows good confidence calibration performance in many fields.

[0030] (3) The present invention experimentally verifies that average confidence and prediction error rate can be used as effective measurement means for temperature estimation. The method of using average confidence and average entropy as measurement, and the method of minimizing distribution difference (MMD) are used to adjust the noise intensity added to the in-domain (ID) verification set, so that the in-domain (ID) verification set and the out-of-domain (OOD) anchor set can obtain an approximate calibration temperature, and then the obtained calibration temperature is used to calibrate the output distribution of the out-of-domain (OOD) test set.

[0031] (4) Through the validation set alignment operation, the dilemma of confidence calibration in the out-of-domain (OOD) scenario is effectively solved; experiments show that the method proposed in this paper exhibits better calibration performance than existing methods on datasets such as SVHN, CIFAR-10, CIFAR-100, and Tiny-ImageNet. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a diagram of the overall framework of the model; (a) represents the overall framework of the model, and (b) represents the generation process diagram of the alternative validation set).

[0033] Figure 2 : Relationship between calibration temperature and prediction error rate; (Resnet-18 is trained on CIFAR-10 and evaluated on CIFAR-10-C. (a) shows the relationship between calibration temperature and prediction error rate without BN layer adjustment, and (b) shows the relationship between calibration temperature and prediction error rate after BN layer adjustment).

[0034] Figure 3 : Relationship diagram between average confidence, prediction error rate and calibration temperature; (Resnet-18 is trained on CIFAR-10 and evaluated on CIFAR-10-C; (a) shows the correlation between average confidence and prediction error rate before BN layer adjustment, (b) shows the correlation between average confidence and prediction error rate after BN layer adjustment, (c) shows the correlation between average confidence and calibration temperature before BN layer adjustment, (d) shows the correlation between average confidence and calibration temperature after BN layer adjustment). DETAILED DESCRIPTION

[0035] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, and the terms "inward" and "outward" refer to directions toward or away from the geometric center of a particular component, respectively.

[0036] Figure 1 Schematic diagram created for this solution's architecture diagram and alternative set:

[0037] 3.2.1 Problem Setting

[0038] In the case of domain shift, each The data points are independent and identically distributed in the source data distribution p s (x,y), but the trained classifier will be t The target set of (x,y) Assume that there is an independent unlabeled dataset D u , where the data points conform to the target distribution p t (x,y). It is also assumed that an additional source distribution p can be obtained s (x,y) labeled validation set D v The validation set is used for the temperature at which the model is searched for calibration.

[0039] The present invention aims to utilize D S Train a model f to make it t The model has good classification accuracy and good confidence calibration performance. Accuracy measures the proportion of correct labels that the model f can predict, while confidence calibration can be measured by the expected calibration error (ECE). The expected calibration error can be calculated as follows: First, all test samples are grouped into M equal intervals according to the confidence output of the sample. Then calculate the weighted average of the difference between the accuracy and confidence of samples in all intervals. The formal calculation is as follows:

[0040]

[0041] 3.2.2 Exploring the metrics for alignment between validation set and anchor set

[0042] Figure 2The correlation between accuracy and optimal temperature is shown. Different colors represent different models, and each point corresponds to an evaluation case on a specific out-of-domain (OOD) target set. The optimal temperature for each evaluation case is determined by minimizing the expected calibration error (ECE) on the corresponding target set. It should be noted that there is a difference between the two figures: Figure 2 In (a), the trained model is directly evaluated during testing; Figure 2 In (b), we adjust the mean and standard deviation of the batch normalization (BN) layer according to the statistics of each target dataset before evaluation. Figure 2 In (b), the temperature change is relatively small. In addition, after adjusting the BN layer, the accuracy on the target dataset can be used as an ideal metric for temperature estimation. The solid line in the figure represents the validation set generated from a set of These validation sets are generated by adding Gaussian noise of varying intensities to the original features. After adjustment by the BN layer, these solid lines fit the data points from different domains well. Therefore, the perturbation of Gaussian noise can be used to generate alternative validation sets for post-hoc calibration.

[0043] We conducted sample experiments to verify that there is a strong correlation between average confidence and accuracy, such as Figure 3 (a) and 3(b). We use the average maximum confidence across all categories as a reflection of the confidence score. Based on these observations, we propose that the average confidence of unlabeled target data can be used as a useful metric for predicting temperature parameters in out-of-domain calibration. Figure 3 (c) and 3(d) show the correlation between the confidence score and the optimal temperature. We observe that when the BN layer is adjusted, the confidence score becomes a reliable indicator of temperature estimation; on the contrary, without the adjustment process, the correlation is significantly weakened.

[0044] 3.2.3 Constructing an alternative set for the out-of-domain (OOD) validation set

[0045] The key to analyzing the validation set alignment metrics is to determine the noise intensity during the perturbation process. Specifically, D v Each sample x in i By adding random noise z i Perform interference, that is, generate transformed samples Among them, z i with x i have the same dimensions and are sampled from a Gaussian distribution N(0,βI), where β controls the intensity of the noise. Our goal is to find the optimal β value to achieve and D u The best alignment between:

[0046]

[0047] Among them, M can be the maximum confidence between categories or the entropy of the output distribution, represents a modified version of the trained model whose batch normalization (BN) layer has been adapted to the target data D u Make adjustments.

[0048] We used two methods to construct the replacement set:

[0049] (1) Search-based construction: Since only one coefficient β needs to be adjusted, a grid search strategy can be adopted: first, β is selected from the candidate pool P and the corresponding validation sets are generated; then the confidence scores of these sets are calculated and compared with D u The confidence scores of D are compared; finally, the u A perturbed validation set with minimal differences in metrics to estimate temperature parameters in temperature scaling (TS) calibration.

[0050] (2) Optimization-based construction: using maximum mean difference (MMD) to perform and D u The distribution matching between the sample output distributions in

[10] . The MMD loss function is:

[0051]

[0052] in is the feature map that maps the sample to the Reproducing Kernel Hilbert Space (RKHS). H is the RKHS.

[0053] Both search-based and optimization-based strategies involve using an unlabeled target dataset D u Adjust each BN layer to build a modified version of the trained model f The adjustment of the BN layer can be easily achieved through a series of forward propagations. In the optimization-based strategy, we compute the maximum mean discrepancy (MMD) loss in each mini-batch training iteration and use the stochastic gradient descent (SGD) optimizer while ensuring that β remains non-negative.

[0054] 3.2.4 Search for the optimal temperature and calibrate

[0055] After the replacement set is constructed, the trained model is used to search for the scaling temperature that minimizes ECE on the replacement set. This temperature is used as the optimal calibration temperature. The optimal calibration temperature is then used to perform temperature scaling (TS) on the out-of-domain test data for post-calibration.

[0056] 3.3 Beneficial effects:

[0057] (1) This technology achieves superior confidence calibration performance on existing benchmark datasets and is highly valuable for classification scenarios that require more robust uncertainty estimation.

[0058] (2) This technology has certain adaptability to the confidence calibration problem in different target fields and has demonstrated good confidence calibration performance in many fields.

[0059] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above-mentioned embodiment, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A method for estimating temperature control parameters for distribution offset data, characterized in that: The method specifically comprises the following steps: Step 1: Obtain N training data from the source domain training set, and label information corresponding to each of the N training data, where the label information is used to represent the category to which the training data belongs, and N is an integer greater than 1; Step 2: Extract n data and the corresponding label information of each data from the N training data as the validation set, and the remaining data as the training set; Step 3: Obtain M unlabeled information data from the target domain dataset; Step 4: Extract m data from the M training data as the anchor set, and the remaining data as the test set; Step 5: Train the model using the training set to obtain a model f with good prediction accuracy for the source domain; Step 6: Process the model f through the BN layer to obtain the adjusted model Step 7: Add Gaussian noise to the validation set, and then align it with the anchor set using the maximum confidence score or output distribution entropy as a measurement, or by minimizing the distribution difference by using the maximum mean difference (MMD). The aligned validation set is used as the replacement set. Step 8: The alignment operation is to search on the validation set and the anchor set to obtain a temperature that is very similar to that used for confidence calibration; Step 9: Searching for the optimal temperature on the alternative set that can minimize the expected calibration error ECE of the model; Step 10: Using the optimal temperature, predict the output using the temperature scaled calibration model.

2. A method for estimating temperature control parameters for distribution offset data according to claim 1, characterized in that: The step 5 includes adjusting the parameters of the BN layer according to the statistical mean and variance of the target domain data to normalize the data.

3. The method for estimating temperature control parameters for distribution offset data according to claim 1, characterized in that: The operation used in step 7 to align the anchor set and the validation set includes using the model adjusted by the BN layer Training anchor set data.

4. The method for estimating temperature control parameters for distribution offset data according to claim 1, wherein: The operation for aligning the anchor set and the validation set in step 7 includes training the validation set with Gaussian noise added using the model f.

5. The method for estimating temperature control parameters for distribution offset data according to claim 1, characterized in that: In step 7, the maximum confidence in the class is used as a measurement method, including the mean square error (MSE) of the maximum confidence of samples of different categories between the anchor set and the training set.

6. The method for estimating temperature control parameters for distribution offset data according to claim 1, characterized in that: In step 7, the entropy of the output distribution is used as a measurement method, including the mean square error (MSE) of the output means of samples of different categories between the anchor set and the training set.

7. The method for estimating temperature control parameters for distribution offset data according to claim 1, characterized in that: The operation used to align the anchor set and the validation set in step 7 includes adjusting the parameter value β added to the Gaussian noise N(0, βI) to minimize the difference in maximum confidence of different categories between the anchor set and the training set.

8. The method for estimating temperature control parameters for distribution offset data according to claim 1, characterized in that: The operation used to align the anchor set and the validation set in step 7 includes adjusting the parameter value β added to the Gaussian noise N(0, βI) to minimize the difference in the output mean of samples of different categories between the anchor set and the training set.

9. The method for estimating temperature control parameters for distribution offset data according to claim 1, characterized in that: The operations used to align the anchor set and the validation set include adjusting the parameter value β in the Gaussian noise N(0,βI), calculating the maximum mean difference MMD loss in each mini-batch training iteration using the validation set and anchor set data, and using the stochastic gradient descent SGD optimizer while ensuring that β remains non-negative to obtain the parameter value β that minimizes the maximum mean difference MMD loss.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the calibration step of the temperature control parameter estimation method for distribution offset data according to any one of claims 1 to 9 is implemented.

Images