An uncertainty perception-based image classification model regression error reduction method

Abstract

The application discloses an image classification model regression error reduction method based on uncertainty perception, and aims at the problem that a new version of an image classification model is difficult to be compatible with an old version of the image classification model after the image data is updated or the model architecture is updated. A model integration strategy is used to estimate and obtain an integrated prediction result of the model according to the uncertainty of the new and old image classification models in the prediction result. The integrated strategy is discussed for two cases of data-free and a small amount of label-free. For the data-free case, the uncertainty of the image classification model is estimated by using an image disturbance or a model disturbance. For the case of a small amount of label-free, the uncertainty of the old version of the image classification model is calibrated by using a temperature scaling method, so as to be aligned to the new version of the image classification model. The application reduces the regression error and obtains an integrated image classification model with forward compatibility, and has a better performance effect.

Description

Technical Field

[0001] This invention relates to a method for reducing regression errors in image classification models based on uncertainty perception, belonging to the technical field of forward compatibility optimization of image classification models during the updating process. Background Technology

[0002] Image classification models are now widely used in various aspects of life, such as facial recognition, autonomous driving, and photo search. As a key component of these modern software systems, image classification models, like traditional software, are constantly being updated to provide better services to users.

[0003] However, continuous updates to image classification models can introduce regression errors. This means that some image samples will always be correctly predicted in older versions of the model but incorrectly predicted in newer versions. The existence of regression errors undermines the forward compatibility of the new model. Most image classification models are based on deep neural network (DNN) architectures. Since DNNs are uncertain models, regression errors after updates are unavoidable. For example, given the same image classification model and the same image dataset, if all other parameters are kept constant, and the model is trained independently twice, although the prediction accuracy after the two training sessions may be similar, the image samples correctly predicted by the model will not be exactly the same.

[0004] The prevalence of regression errors in image classification models can have a detrimental impact on production practices and user experience. For example, updates to photo search apps can lead to different search results, causing a drop in the app's rating in app stores. Similarly, updates to medical classification models for X-ray images can alter the image samples that were prone to misclassification, requiring doctors to readjust. These issues reflect the regression errors that occur after updating image classification models, leading to decreased forward compatibility and negatively impacting user experience.

[0005] Current methods for reducing regression errors in image classification models still have many shortcomings. On the one hand, existing model training techniques cannot guarantee the forward compatibility of the new image classification model with the old one when image data or model architecture is updated, thus failing to meet the requirements of business applications. On the other hand, existing regression error reduction techniques can lead to a decrease in the performance of image classification models, resulting in poor practical effects and failing to meet the requirements of business applications. Summary of the Invention

[0006] Purpose of the invention: To address the problems of poor forward compatibility and performance degradation of new image classification models when image data or image classification model architecture is updated in existing technologies, this invention provides a lightweight application technology for reducing regression errors in image classification models based on uncertainty awareness. From the perspective of uncertainty analysis of image classification models, it adopts an integrated approach to reduce regression errors in image classification models during update scenarios and ensure the lower bound of the performance of image classification models.

[0007] Technical Solution: A regression error reduction method for image classification models based on uncertainty awareness is proposed to reduce regression errors generated during image classification model updates, while taking into account the forward compatibility and performance of the image classification model. The method includes: 1) training the old and new image classification models independently during model updates; 2) using image perturbation or model perturbation strategies to estimate the uncertainty of the image classification model in the absence of data; 3) using temperature scaling strategies to calibrate the uncertainty of the old model in the presence of a small amount of unlabeled data; and 4) using a simple averaging model ensemble method to obtain an ensemble model.

[0008] In the above 1), the reasons for updating the image classification model are usually as follows: (1) More image data is collected for model training. For example, an app that provides photo search services to users collects more image data while providing the service, and this image data can be used to train the model for updates; (2) The underlying neural network architecture of the image classification model is updated. For example, the VGG network architecture was a popular image classification model used nearly ten years ago, while newer network architectures such as ResNet have been proven to provide higher accuracy, so an upgrade is needed. When updating the model for these two reasons, we need to train the old model and the new model independently on the corresponding image dataset using the corresponding network architecture, and the new model obtained is used to update the original old model. However, after the model is updated, regression error will inevitably be introduced. For example, when the ResNet-50 model is updated to the DenseNet-169 model, the regression error is as high as 9.44% on the ImageCLEF image dataset. For the image dataset D, the samples that the old model predicts correctly but the new model predicts incorrectly are called regression errors.

[0009] Consider a common scenario during model updates where the image dataset used for model training cannot be determined due to privacy or proprietary restrictions on image data. In this case, the strategy for analyzing the uncertainty of the model is either image perturbation or model perturbation.

[0010] In step 2), when no additional data is available to reduce the regression error of the image classification model, the uncertainty of the old and new models is adjusted by estimating the prediction variance for each image input sample. For similar image inputs, the image classification model should produce similar prediction outputs; therefore, for similar image inputs, the model with a larger prediction variance has greater uncertainty—this is image perturbation. Similarly, for the same input, the larger the prediction variance after a small change in the image classification model, the greater the model's uncertainty—this is model perturbation.

[0011] Based on this conclusion, two ways to introduce disturbances can be derived:

[0012] (1) Perturb the input image with random noise;

[0013] (2) Add dropout perturbation model to image classification model.

[0014] The old model is denoted as M. old The new model is denoted as M. new For the input image x, the predicted output of the old model is denoted as... The predicted output of the new model is denoted as The prediction variances of the old and new models are calculated using image perturbation or model perturbation, respectively. and

[0015] For the input image x, the scaling factors α1(x) and α2(x) are defined as follows:

[0016]

[0017]

[0018] Based on the input data x, the calculated scaling factors α1(x) and α2(x) are used to adjust the model predictions, achieving point-by-point scaling of the predictions. The predictions of the old model and the new model after adjustment are as follows:

[0019]

[0020]

[0021] This scaled simple average is equivalent to inverse variance weighting, when M... old and M new When there is no correlation between the components, the variance will be minimized. At this point, the ensemble model... Predicted output This involves simply averaging the adjusted model predictions:

[0022]

[0023] In point 3), when some unlabeled data is available to reduce regression errors in image classification models, temperature scaling can better handle regression errors. Temperature scaling is a simple yet effective technique that can be used for confidence calibration. It involves scaling the input Logitz layer of the softmax layer before the softmax layer of the neural network. M Divide by the scalar T (called temperature), that is:

[0024]

[0025] in This represents the predicted output of model M at temperature T. Models using temperature T are denoted as M. (T) Temperature scaling does not change the category of the maximum value in the Softmax output, thus preserving the model's predictions. Based on this property, model M... (T) The prediction results are completely consistent with those of model M.

[0026] Mean Squared Error (MSE) represents the difference between the model's predicted value and the true label. Therefore, MSE(M) reflects the uncertainty of model M to a certain extent; the larger the difference between the model's predicted value and the true label, the larger the corresponding MSE, indicating greater uncertainty in the model. In the scenario of updating image classification models, the old model often has greater uncertainty, which is why model updates are necessary. Therefore, it is necessary to update the old image classification model M. old Temperature scaling is performed to obtain By reducing the uncertainty of the old model and making it approximately equal to the uncertainty of the new model, the uncertainties of the old and new models are aligned. Then, a simple averaging ensemble method can be used to obtain an ensemble model with the best performance.

[0027] Note that model uncertainty is measured using mean squared error (MSE), therefore only the following needs to be implemented:

[0028]

[0029] To achieve the above objectives, a small set of unlabeled image data was used. Specifically, the old model was temperature-scaled onto this set of unlabeled image data. And solve for the optimal temperature T. * Make For the optimal temperature T *The solution was obtained using the quasi-Newton method L-BFGS with multiple initial points.

[0030] It is worth noting that the above method only requires using this set of unlabeled image data to obtain the optimal temperature value T before the old model makes its first prediction. * For subsequent inputs, this temperature value T will be used each time. * Without changing anything, the predicted output of the ensemble model is obtained through the following expression.

[0031]

[0032] in, This represents the predicted probability distribution of the old model after temperature scaling. This represents the predicted probability distribution of the new model. Note the optimal temperature value T. * Since it only requires one calculation, this method is also efficient and feasible.

[0033] In section 4), the simple averaging model ensemble method is to average the prediction results of multiple models. Specifically, for the regression error problem of image classification models, it means to simply average the prediction probability distribution of the old model and the new model on the input data.

[0034] In sections 2) and 3), uncertainty analysis methods were presented for two scenarios: no data and a small amount of unlabeled data. Furthermore, scaling factors or temperature scaling techniques were used to adjust the predictions of the old and new models, respectively. Let the adjusted prediction of the old model be denoted as... And the predictions of the adjusted new model are At this point, a simple averaging of these two predictions yields the ensemble model we desire, which combines forward compatibility and performance. Prediction

[0035]

[0036] 1) When updating the image classification model, the old and new models are trained independently. Note that the training of the old and new models is performed independently, and they may use different image datasets and deep neural network architectures. The old and new models are trained using the following process:

[0037] 101) Construct the model architecture and randomly initialize the model parameters;

[0038] 102) Read the image training dataset in batches and input it into the model;

[0039] 103) Calculate the cross-entropy between the model's predicted values ​​and the true label values ​​to obtain the model's loss.

[0040] 104) The backpropagation algorithm is used to update the parameters of each neural network layer of the model based on the loss.

[0041] 105) Train the model until it converges to obtain a trained model.

[0042] Uncertainty analysis is performed on the old and new image classification models. Depending on the situation, the uncertainty analysis process in 2) for the case without data or the uncertainty analysis process in 3) for the case with a small amount of unlabeled data can be adopted respectively.

[0043] 2) The uncertainty analysis process in the absence of data is as follows:

[0044] 201) Give the old model (M) old ) and the new model (M) new Given the same image data (x), obtain the corresponding prediction outputs. and

[0045] 202) Evaluate the uncertainty of the model by perturbing the image or the model;

[0046] (2021) Regarding the method of image perturbation:

[0047] i) Generate a set of image inputs x that are similar to x by adding random noise (e.g., Gaussian noise). 1,…,n ;

[0048] ii) Input this set of images into x 1,…,n Simultaneously input to M old and M new The predicted outputs generated by the old model and the new model are obtained respectively, and the variance of the predicted outputs is calculated. and

[0049] (2022) Regarding the method of model perturbation:

[0050] i) For the old model M old And the new model M new By setting up network dropout to randomly discard the output of some neurons, a set of randomized models similar to the old model can be generated. and a set of similar stochastic models corresponding to the new model

[0051] ii) Input the image x simultaneously to and They were obtained respectively and The generated set of predicted outputs is used to calculate the variance of the predicted outputs. and

[0052] 203) Based on the image perturbation or model perturbation, obtain the prediction variances of the old and new models, and calculate the scaling factors α1(x) and α2(x) of the old and new models respectively. The calculation formula is as follows:

[0053]

[0054]

[0055] 204) Based on the scaling factor, obtain the predicted outputs of the old model and the new model with respect to the image input x after adjustment by the scaling factor:

[0056]

[0057]

[0058] 3) The uncertainty analysis process for situations with a small amount of unlabeled data is as follows:

[0059] 301) Given the old model (M) old ), new model (M) new Given a small dataset of unlabeled images D and an input image x to be predicted, we obtain the corresponding prediction outputs. and

[0060] 302) Determine whether the optimal temperature T has been calculated for dataset D. * :

[0061] i) If yes, then proceed to step 304 to execute;

[0062] ii) If not, proceed to step 303 to execute;

[0063] 303) On dataset D, according to The optimal temperature T is obtained by using the quasi-Newton method L-BFGS with multiple initial points. * ;

[0064] 304) Employs temperature scaling technology, using the optimal temperature T * Scaling M old Logit Obtain the old model M after temperature scaling adjustment old Predicted output:

[0065]

[0066] 305) New Model M new Predicted output Temperature scaling is not required, so it remains unchanged.

[0067] 4) Based on the uncertainty analysis results of the new and old models, the prediction outputs of the new and old models are integrated. The process is as follows:

[0068] 401) After uncertainty analysis (through process 2 or 3), the prediction outputs of the old model and the new model are denoted as follows: and

[0069] 402) Through uncertainty analysis, the second-level uncertainties of the old and new models are kept consistent. At this point, a suitable integrated prediction output can be obtained through simple averaging.

[0070]

[0071] 403) That is, to integrate the old model M old And the new model M new Then, the predicted output is generated from the image input x.

[0072] 404) Finally, the ensemble model The predicted classification category for input x is the ensemble prediction output. The index class (1,…,K) with the maximum value, i.e.:

[0073]

[0074] Compared with existing technical solutions, the present invention has the following characteristics:

[0075] 1) Using model ensemble methods can effectively address the regression error caused by updating image classification models, thus achieving a balance between forward compatibility and performance.

[0076] 2) From the perspective of uncertainty in deep neural networks, a complete and effective method is presented to analyze and quantify the uncertainty of deep neural networks;

[0077] 3) In view of the actual situation of image classification model updates in production practice, two solutions are proposed, which can effectively solve the problem in the case of no data and with a small amount of unlabeled data;

[0078] 4) This method is characterized by high efficiency, lightweight and universality. The process does not require a large amount of image data. The whole process is fast, efficient and timely, and has application value. Attached Figure Description

[0079] Figure 1 This is an overall execution flowchart of an embodiment of the present invention;

[0080] Figure 2 A flowchart for performing uncertainty analysis on new and old models in the absence of data, provided as an embodiment of the present invention;

[0081] Figure 3 This is a flowchart illustrating uncertainty analysis of old and new models when some unlabeled data is available, provided as an embodiment of the present invention.

[0082] Figure 4 This is a flowchart illustrating the measurement of model uncertainty through image perturbation and model perturbation, provided as an embodiment of the present invention. Detailed Implementation

[0083] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0084] Figure 1 The overall execution flowchart of this invention is described, emphasizing that this invention is a technique for reducing regression errors in image classification models based on uncertainty awareness. The main method used is model ensemble, which includes both an old model and a new model. Figure 2 and Figure 3 The processes are described in the case of no data and the case of having some unlabeled data. The basic idea is to analyze the uncertainty of the old model and the new model and then perform a simple averaging model ensemble method. Figure 4 The illustrations illustrate the specific process of analyzing model uncertainty through image perturbation and model perturbation in the absence of data.

[0085] The following is a detailed implementation description. First, let's describe the model preparation part:

[0086] Step 1: The old and new models are trained independently, usually using different image datasets and deep neural network architectures, and the new model generally has higher accuracy on the current test set.

[0087] The following section describes the uncertainty analysis of the model, starting with the uncertainty analysis in the absence of data:

[0088] Step 2: In the absence of data, the old model M will be used as the basis for the calculation. old New model M new The uncertainty of the model is analyzed by taking the image input x to be predicted and using either image perturbation or model perturbation.

[0089] Image perturbation involves adding random noise to the input image x to generate a set of similar input images x. 1,…,n Input these images into M respectively old and M new And calculate the variance of the model's predicted output. and

[0090] The method for model perturbation is to set M old and M new Dropout is used to randomly discard a portion of the outputs of neurons in order to generate a set of similar random models. and Then, the image to be predicted is input as x, and this input is given to both models respectively. The variance of the predicted output of each model is then calculated. and

[0091] Step 3: Obtain the scaling factors α1(x) and α2(x) of the model based on the model's prediction variance. Its essence is inverse variance weighting.

[0092] Step 4: Based on the scaling factors α1(x) and α2(x), and the prediction results of the old and new models for the image input x. and After weight scaling, the prediction results are obtained after aligning the uncertainties of the old and new models. and

[0093] The uncertainty analysis process for cases with some unlabeled data is as follows:

[0094] Step 2: In the case of having some unlabeled data, the old model M will be used as a basis. old New model M new We use an unlabeled dataset D to analyze the uncertainty of the model, employing a temperature scaling method.

[0095] Step 3: Determine the optimal temperature T * Does it exist? If it exists, skip this step. Otherwise, on dataset D, according to... The optimal temperature T is obtained by using the quasi-Newton method L-BFGS with multiple initial points. * .

[0096] Step 4: For input x, for M old The generated predicted Logit uses temperature T * After scaling, the predicted output is obtained by passing it through a Softmax layer. And M new Since temperature scaling is not used, its predicted output remains the same.

[0097] Finally, the model integration section:

[0098] Step 5: For ease of description, the predicted output generated by the old model in Step 4 of the uncertainty analysis in the above two cases is denoted as... The predicted output generated by the new model is denoted as... At this point, the prediction outputs of the old and new models have undergone uncertainty alignment and possess uniform uncertainty. Therefore, a satisfactory ensemble model prediction output can be obtained through simple averaging. The calculation method is as follows: Finally, found The index category containing the maximum value of an element in a vector is the ensemble model. Predict the category of the image input x.

[0099] Image classification models typically employ deep neural network models, including ResNet_v1, ResNet_v2, WideResNet (WRN), ResNeXt, VGG, etc. These models consist of a neural network structure with an input layer, an output layer, and several hidden layers, and usually include convolutional layers, pooling layers, batch normalization layers, fully connected layers, and activation layers.

[0100] Through the above method, the present invention realizes a lightweight application technology for reducing regression error of image classification model based on uncertainty perception. The method adopts a model ensemble approach, which first performs uncertainty analysis on the old and new models before and after the update, and then performs simple averaging model ensemble to obtain an ensemble model with forward compatibility and good performance.

[0101] Obviously, those skilled in the art should understand that the steps of the lightweight application technology for reducing regression errors in image classification models based on uncertainty perception described in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be implemented using computer-executable program code, thereby constituting one of the components in an application software system. Therefore, the embodiments of the present invention are not limited to any specific hardware and software combination.

Claims

1. A method for reducing regression error in an image classification model based on uncertainty perception, characterized in that, The process includes the following steps: 1) When updating the image classification model, the old and new image classification models are trained independently; 2) For cases without data, image perturbation or model perturbation strategies are used to estimate the uncertainty of the image classification model; 3) For cases with unlabeled data, temperature scaling strategies are used to calibrate the uncertainty of the old model; 4) An ensemble model is obtained using a simple averaging model ensemble method. In step 3), when some unlabeled data is available to reduce the regression error of the image classification model, temperature scaling is used to process the regression error, thus improving the old image classification model. Temperature scaling is performed to obtain By using the mean squared error (MSE) to measure model uncertainty, the following can be achieved: Using a set of unlabeled image data, the old model is temperature-scaled onto this set of unlabeled image data to obtain... And solve for the optimal temperature. Make For the optimal temperature The solution is obtained using the quasi-Newton method L-BFGS with multiple initial points; For subsequent input images, this temperature value is used every time. Without changing anything, the predicted output of the ensemble model is obtained through the following expression. : 。 2. The method for reducing regression error in image classification models based on uncertainty perception according to claim 1, characterized in that, In the above 1), the reasons for updating the image classification model are usually as follows: (1) more image data is collected for model training; (2) the underlying neural network architecture of the image classification model is updated. When the image classification model is updated due to these two reasons, the old model and the new model of image classification are trained independently on the corresponding image dataset using the corresponding network architecture. The new model obtained by training is used to update the original old model. After the model is updated, regression error is introduced. For the image dataset D, the samples that are correctly predicted by the old model and incorrectly predicted by the new model are called regression error.

3. The method for reducing regression error in image classification models based on uncertainty perception according to claim 1, characterized in that, In step 2), during image perturbation, the input image is perturbed by random noise: for the input image Add random noise to generate a set of similar input images. Input these images into the old model respectively. and new models And calculate the old model and new models Prediction variance and .

4. The method for reducing regression error in image classification models based on uncertainty perception according to claim 1, characterized in that, In step 2), the model perturbation method is to set the old model. and new models Dropout is used to randomly discard a portion of the outputs of neurons in order to generate a set of similar random models. and Then the input image to be predicted The inputs are given to these two sets of models respectively. and Calculate the variance of the predicted output for each model group. and .

5. The method for reducing regression error in an image classification model based on uncertainty perception according to claim 1, characterized in that, In section 4), the simple averaging model ensemble method, specifically for the regression error problem of image classification models, refers to simply averaging the predicted probability distributions of the old and new models on the input data.

6. The method for reducing regression error in an image classification model based on uncertainty perception according to claim 1, characterized in that, When updating the image classification model, the old model and the new model are trained independently. The old model and the new model are trained using the following process: 101) Construct the model architecture and randomly initialize the model parameters; 102) Read the image training dataset in batches and input it into the model; 103) Calculate the cross-entropy between the model's predicted values ​​and the true label values ​​to obtain the model's loss. 104) The backpropagation algorithm is used to update the parameters of each neural network layer of the model based on the loss. 105) Train the model until it converges to obtain a trained model.

7. The method for reducing regression error in an image classification model based on uncertainty perception according to claim 1, characterized in that, Uncertainty analysis is performed on the old and new image classification models. Depending on the situation, the uncertainty analysis process can be adopted for either the case without data or the case with unlabeled data. 2) The uncertainty analysis process in the absence of data is as follows: 201) Give the old model and new models Input the same image data The corresponding prediction outputs are obtained respectively. and ; 202) Evaluate the uncertainty of the model by perturbing the image or the model; (2021) Regarding the method of image perturbation: i) Input based on image By adding random noise to generate A similar set of image inputs ; ii) Input this set of images Simultaneously input to and The predicted outputs generated by the old model and the new model are obtained respectively, and the variance of the predicted outputs is calculated. and ; (2022) Regarding the method of model perturbation: i) Regarding the old model and new models By setting up network dropout to randomly discard the output of some neurons, a set of randomized models similar to the old model can be generated. and a set of similar stochastic models corresponding to the new model ; ii) Input the image Simultaneously input to and , respectively obtained and The generated set of predicted outputs is used to calculate the variance of the predicted outputs. and ; 203) Obtain the prediction variances of the old and new models based on the image perturbation or model perturbation, and calculate the scaling factors of the old and new models respectively. and The calculation formula is: 204) Based on the scaling factor, obtain the old and new models with respect to the image input after adjustment by the scaling factor. Predicted output: 。 8. The method for reducing regression error in an image classification model based on uncertainty perception according to claim 7, characterized in that, 3) The uncertainty analysis process for situations with a small amount of unlabeled data is as follows: 301) Given the old model New model A set of unlabeled image datasets and the image input to be predicted The corresponding prediction outputs are obtained respectively. and ; 302) Determine if there is a problem in the dataset. Has the optimal temperature been calculated? : i) If yes, then proceed to step 304 to execute; ii) If not, proceed to step 303 to execute; 303) In the dataset Above, according to The optimal temperature was obtained by using the quasi-Newton method L-BFGS with multiple initial points. ; 304) Employs temperature scaling technology to use the optimal temperature. Scaling Logit The old model after temperature scaling adjustment is obtained. Predicted output: 305) New Model Predicted output Temperature scaling is not required, so it remains unchanged.

Images