Self-adaptive target identification method based on sample uncertainty estimation during online test

Abstract

The invention discloses a sample uncertainty estimation-based self-adaptive target recognition method during online testing, which belongs to the technical field of computer vision and comprises the following steps of: firstly, downloading a data set and augmenting the data set; then sequentially sending each batch of data into the source domain model, evaluating the current batch of data twice through the source domain model, and calculating an uncertainty value of each sample; carrying out model learning on the batch of samples and carrying out model training updating; and finally, predicting the samples by using the model which learns the current batch of target data to obtain a classification result of the batch of samples. According to the invention, the problem that the target domain data cannot be obtained and real-time identification needs to be carried out in the adaptive framework during the online test at present is well solved, and the accuracy of identifying the vehicle target object by the intelligent traffic during the online test is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to an adaptive target recognition method during online testing based on sample uncertainty estimation. Background Art

[0002] Deep learning has made great progress in the field of images, but it is overly dependent on a large amount of training data. The effectiveness of the model is closely related to the probability distribution of training data and test data. Reusing test data to train a new model requires not only obtaining a large amount of test data, but also manual annotation, which requires huge costs. In real-life applications, such as autonomous driving and intelligent transportation, due to the change in lighting during the day, from day to night, even images taken at the same location will be different. In addition, the change in weather, from sunny to rainy days, at the same time for two consecutive days, such as 8 am, and at the same location, the difference in images will be large. These changes at all times or suddenly will make it impossible for the pre-deployed model to perform real-time object recognition.

[0003] In order to simultaneously deal with the two challenges of different data distribution of images and online adaptation, the main idea of ​​online test-time adaptation is proposed: the source domain data is used as training data, and the model obtained after training is used as the source domain model; the target domain data is used as test data, which has a similar but different distribution to the source domain data; the goal of online test-time adaptation is to adapt the standard pre-trained source domain model to the target domain, where the test data enters as a batch stream, so the adaptation needs to be performed in parallel with inference.

[0004] At present, the methods of online test-time adaptation are mainly divided into three categories: batch normalization statistical correction methods adapt the BN layer of the source model to the statistics of the target data offset. However, batch normalization statistical correction methods depend on the size of the batch, and the estimation of statistics on small batches of data may not be accurate enough, which may lead to a decrease in model performance. Self-training methods generate pseudo labels for target data and adapt in a supervised manner. Y. Iwasawa (Y. Iwasawa and Y. Matsuo, "Test-time classifier adjustment module for model-agnostic domain generalization," in Proc. NeurIPS, 2021, pp. 2427-2440) et al. proposed to obtain pseudo labels by only using online unlabeled data to calculate class prototypes and classify each unlabeled sample based on its distance to these prototypes. However, when the target data distribution is very different from the source domain data distribution, the model trained based on the source data may not be able to effectively calculate accurate class prototypes, thereby affecting the quality of pseudo labels. Gao (Gao, J., Zhang, J., Liu, X., Darrell, T., Shelhamer, E., Wang, D.: Back to the source: Diffusion-driven adaptation to test-time corruption. In: CVPR. pp. 11786-11796 (2023)) et al. proposed an online test-time adaptation method that uses a diffusion model to align the target image and the source domain image. This method aims to make the target image visually closer to the source domain through image style transfer without adjusting the parameters of the source model. The diffusion model may cause loss of image quality when transferring the style of the target image to match the source domain, especially in details and textures, and in the process of transferring the style of the target image to match the source domain, some important features in the target domain may be lost, which may be critical for classification tasks. Summary of the invention

[0005] The present invention provides an adaptive target recognition method during online testing based on sample uncertainty estimation, which can well solve the problem that source domain data cannot be obtained during domain adaptation framework training and the model needs to be adapted in real time, resulting in poor accuracy.

[0006] To achieve the above objectives, the present invention adopts the following technical solutions:

[0007] An adaptive target recognition method during online testing based on sample uncertainty estimation comprises the following steps:

[0008] Step 1: Dataset acquisition process: First download the dataset and augment it;

[0009] Step 2: Neural network adaptation process: Use the model to evaluate the current batch of data twice, calculate the uncertainty value of each sample, and perform model learning updates;

[0010] Step 3: Test the image recognition process: Use the model that has learned the current batch of target data to predict the samples and obtain the classification results of the batch of samples.

[0011] In the above steps, step 1 specifically includes the following steps:

[0012] Step 1-1: Download the dataset from the website and get several 32×32 images;

[0013] Step 1-2: Perform a random combination of augmentations on each image, including center cropping and random horizontal flipping.

[0014] Step 2 specifically includes the following steps:

[0015] Step 2-1: Use the drop layer of the neural network as the basis for uncertainty estimation, and perform two model evaluations on each sample x. The specific process is: repeatedly send the sample to the neural network model twice, perform feature extraction and classification output, and obtain two prediction results of the model, namely:

[0016] P T (y|x)=f θ (x) (1)

[0017] Among them, P T (y|x) represents the model prediction result, T represents the number of model predictions, and f θ (.) is the convolutional neural network model, which is specifically expressed as follows: Convolutional neural network is a deep learning model that extracts image features and classifies them through a series of convolutional layers, pooling layers, and fully connected layers;

[0018] Step 2-2: Calculate the difference between the two model evaluation results to obtain the uncertainty estimate for each sample. The calculation formula is as follows:

[0019]

[0020] Where C represents the number of categories, represents the probability value of the current category c, and t represents the tth model prediction result;

[0021] Step 2-3: Optimize the original entropy-based domain adaptation loss, weight it according to the uncertainty value of the sample, use the network output as the calculation value, and calculate the loss according to the uncertainty weight value. The weighted loss is as follows:

[0022]

[0023] Where C represents the number of categories, and the value of c is 1, 2, 3...C, P c Indicates the probability value when belonging to category c;

[0024] Step 2-4: Set the learning rate and batch size value for the adaptive hyperparameter network training on the neural network, and use the Adam optimizer for optimization.

[0025] Step 3 specifically includes the following steps:

[0026] Step 3-1: Send the test image and the two augmented images into the updated neural network;

[0027] Step 3-2: Use the model to predict the original image and the two augmented images to obtain three prediction results; that is, the original image is provided to the model as input, and features are extracted through the network's convolution operation and predicted and classified through the fully connected layer. At the same time, the two augmented images are also provided to the model. Finally, three prediction results can be obtained, that is, three C-dimensional vectors are obtained.

[0028] Step 3-3: Average the three prediction results to obtain the final result.

[0029] Beneficial effect: The present invention provides an adaptive target recognition method during online testing based on sample uncertainty estimation. By introducing uncertainty evaluation of samples, the key problem of unreliable pseudo-label prediction by the model is solved. By utilizing weighted entropy loss, the characteristic structure information and underlying distribution information of target domain data can be more effectively mined, thereby improving the accuracy of object classification. Compared with the prior art, the present invention is easier to implement, and well solves the problem that the target domain data cannot be obtained in the adaptive framework during online testing and real-time recognition is required, thereby improving the accuracy of intelligent transportation in identifying vehicle target objects during online testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0031] Figure 2 This is a method flow chart of step 1 of an embodiment of the present invention.

[0032] Figure 3 This is a method flow chart of step 2 of an embodiment of the present invention.

[0033] Figure 4 This is a method flow chart of step 3 of an embodiment of the present invention.

[0034] Figure 5 This is a test result diagram of the identification method in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0036] like Figure 1 As shown, this embodiment is applied to real-time object recognition in an actual traffic environment, specifically an online intelligent traffic adaptive target recognition method based on sample uncertainty estimation, comprising the following steps:

[0037] Step 1: Figure 2 As shown in the figure, data is downloaded from the official website of CIFAR-C and CLAD-C datasets to obtain several 32×32 images. Each image is randomly enhanced, including center cropping and random horizontal flipping. The original image is center cropped to the original size, and the probability of random horizontal flipping is 0.5.

[0038] Step 2: If Figure 3 As shown, neural network learning adaptation is performed, the model is used to evaluate the current batch of data twice, and then the uncertainty value of each sample is calculated based on the two evaluations, and the weighted loss calculation and model learning update are performed; the specific steps include the following:

[0039] Step 201: Introduce the drop layer feature of the network, randomly ignore some neurons of the model, repeatedly send the sample into the neural network model twice, perform feature extraction and classification output, and obtain two prediction results, namely:

[0040] P T (y|x)=f θ (x) (1)

[0041] Among them, P T (y|x) represents the model prediction result, T represents the number of model predictions, and f θ (.) is a neural network model. A convolutional neural network is a deep learning model that extracts image features and classifies them through a series of convolutional layers, pooling layers, and fully connected layers.

[0042] Step 202: Perform a difference calculation on the two model prediction results, and use the difference calculation result as the uncertainty estimate of each sample. The calculation formula is as follows:

[0043]

[0044] Where C represents the number of categories; represents the probability value of the current category c; t represents the tth model prediction result;

[0045] Step 203: The uncertainty value of the sample is used as the entropy loss weight for weighting. The entropy-based domain adaptation loss in which the weight value of each sample is 1 is improved. The model prediction output is used as the entropy calculation value, and the uncertainty value is introduced as the weight value to calculate the weighted entropy loss. The weighted loss is as follows:

[0046]

[0047] Where C represents the number of categories, and the value of c is 1, 2, 3...C, P c Indicates the probability value when belonging to category c;

[0048] Step 204: Set hyperparameters for the neural network model during adaptation, set the learning rate to 0.001, the batch size to 110, use the Adam optimizer to optimize and perform model adaptation.

[0049] Step 3: If Figure 4 As shown, the model that has learned the current batch of target data is used to predict the samples to obtain the classification results of the batch of samples, which specifically includes the following steps:

[0050] Step 301: Send the original image into the updated neural network, and also send the two augmented images into the neural network for prediction;

[0051] Step 302: Provide the original image as input to the model, perform feature extraction through the network convolution operation, and perform prediction classification through the fully connected layer. At the same time, the two augmented images are also provided to the model. Finally, three prediction results can be obtained, that is, three C-dimensional vectors are obtained, which are the model's predictions for the original image and the two augmented images.

[0052] Step 403: Perform arithmetic averaging on the three prediction results to obtain the final prediction result.

[0053] Figure 5In order to obtain the recognition results using the above method, the experiment was conducted on two TITAN Xp graphics cards. The learning rate was set to 0.001 during model adaptation. The accuracy of the batch data was output to the terminal every 6 batches to facilitate observation of the overall convergence. Finally, the overall accuracy was calculated at the end and the verification results were output. In each domain adaptation task, the source domain and target domain corresponding to the task will be indicated. For example, brightness 5 means that the network currently trained by the source domain CIFAR dataset has a damage type of brightness and a damage degree of 5 in the target domain CIFAR10-C, which can achieve a classification accuracy of 92.09%.

[0054] The above-described embodiments are only for illustrating the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. An adaptive target recognition method during online testing based on sample uncertainty estimation, characterized in that: The following steps are involved: Step 1: Dataset acquisition process: First download the dataset and augment it; Step 2: Neural network adaptation process: Use the model to evaluate the current batch of data twice, calculate the uncertainty value of each sample, and perform model learning updates; Step 3: Test the image recognition process: Use the model that has learned the current batch of target data to predict the samples and obtain the classification results of the batch of samples.

2. The method for adaptive target recognition during online testing based on sample uncertainty estimation according to claim 1, characterized in that: Step 1 specifically includes the following steps: Step 1-1: Download the dataset from the website and get several images; Step 1-2: Perform random enhancement combinations on each image.

3. The method for adaptive target recognition during online testing based on sample uncertainty estimation according to claim 2, characterized in that: A random enhancement combination is performed on each image, including center cropping and random horizontal flipping. The center cropping size is the same as the original image size, and the probability of random horizontal flipping is 0.

5.

4. The method for adaptive target recognition during online testing based on sample uncertainty estimation according to claim 1, characterized in that: Step 2 specifically includes the following steps: Step 2-1: Use the drop layer of the neural network as the basis for uncertainty estimation, perform two model evaluations on each sample x, and obtain two prediction results of the model; Step 2-2: Calculate the difference between the two model evaluation results to obtain the uncertainty estimate for each sample Step 2-3: Optimize the original entropy-based domain adaptation loss, weight it according to the uncertainty value of the sample, use the network output as the calculation value, and calculate the loss according to the uncertainty weight value; Step 2-4: Set the learning rate and batch size value for the adaptive hyperparameter network training on the neural network, and use the Adam optimizer for optimization.

5. The method for adaptive target recognition during online testing based on sample uncertainty estimation according to claim 4, characterized in that: The two prediction results of the model in step 2-1 are: P T (y|x)=f θ (x) (1) Among them, P T (y|x) represents the model prediction result, T represents the number of model predictions, and f θ (.) is the neural network model.

6. The method for adaptive target recognition during online testing based on sample uncertainty estimation according to claim 4, characterized in that: The calculation formula for calculating the difference between the two model evaluation results in step 2-2 is as follows: Where C represents the number of categories; It represents the probability value of the current category C; t represents the tth model prediction result.

7. The method for adaptive target recognition during online testing based on sample uncertainty estimation according to claim 4, characterized in that: The weighted loss in step 2-3 is: Where C represents the number of categories, and the value of c is 1, 2, 3...C, P c Indicates the probability value when it belongs to category c.

8. The method for adaptive target recognition during online testing based on sample uncertainty estimation according to claim 1, characterized in that: Step 3 specifically includes the following steps: Step 3-1: Send the test image and the two augmented images into the updated neural network; Step 3-2: Perform feature extraction through the convolution operation of the neural network and perform prediction classification through the fully connected layer to obtain the model's prediction of the original image and the two augmented images, and obtain three prediction results; Step 3-3: Average the three prediction results to obtain the final result.

Images