An online test time adaptive target recognition method based on sample uncertainty estimation

Abstract

The application discloses an online test self-adaptive target recognition method based on sample uncertainty estimation and belongs to the technical field of computer vision. The method comprises the following steps: firstly, downloading a data set and performing augmentation; then, sequentially feeding each batch of data into a source domain model, obtaining twice evaluation of the current batch of data through the source domain model, and calculating the uncertainty value of each sample; learning the model for the batch of samples and updating the model training; finally, predicting the samples by using the model that has learned the current batch of target data to obtain the classification result of the batch of samples. The application solves the problem that target domain data cannot be obtained in the current online test adaptive framework and real-time recognition is required, and improves the accuracy of intelligent traffic vehicle target object recognition during online test.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an online test-time adaptive target recognition method based on sample uncertainty estimation. Background Technology

[0002] Deep learning has made significant progress in image processing, but it relies heavily on large amounts of training data. The performance of a model is closely related to the probability distribution of the training and test data. Retraining a new model using test data requires not only a large amount of test data but also manual annotation, which is extremely costly. In real-world applications, such as autonomous driving and intelligent transportation, the changing light throughout the day, from day to night, can cause differences even between images taken at the same location. Furthermore, unpredictable weather changes, from sunny to rainy, can lead to significant differences between images taken at the same time and location on two consecutive days, such as 8 AM. These constant or sudden changes can prevent pre-deployed models from performing real-time object recognition.

[0003] To address the challenges of varying image data distributions and online adaptation simultaneously, we propose the following main idea for online testing-time adaptation: source domain data is used as training data, and the model obtained after training is used as the source domain model; target domain data is used as test data, which has a similar but different distribution from the source domain data; the goal of online testing-time adaptation is to adapt the standard pre-trained source domain model to the target domain, where the test data enters as a batch stream, so adaptation needs to be performed in parallel with inference.

[0004] Currently, online test-time adaptation methods are mainly divided into three categories: Batch Normalized Statistical Correction (BN) methods adapt the source model's BN layer to the statistics of the target data offset. However, BN methods depend on the batch size, and the estimation of statistics on small batches of data may not be accurate enough, which may lead to a decrease in model performance. Self-training methods generate pseudo-labels for the target data and perform adaptation in a supervised manner. Y. Iwasawa et al. (Y. Iwasawa and Y. Matsuo, "Test-time classifier adjustment module for model-agnostic domain generalization," in Proc. NeurIPS, 2021, pp. 2427-2440) proposed obtaining pseudo-labels by calculating class prototypes using online unlabeled data and classifying each unlabeled sample based on its distance to these prototypes. However, when the distribution of the target data differs greatly from that of the source domain data, the model trained on the source data may not be able to effectively calculate accurate class prototypes, thus affecting the quality of the pseudo-labels. Gao et al. (Gao, J., Zhang, J., Liu, X., Darrell, T., Shelhamer, E., Wang, D.: Back to the source: Diffusion-driven adaptation to test-time corruption. In: CVPR.pp.11786-11796(2023)) proposed an online test-time adaptation method that uses a diffusion model to align target and source domain images. This method aims to make the target image visually closer to the source domain through image style transfer without adjusting the parameters of the source model. The diffusion model may cause a loss of image quality, especially in detail and texture, when style transferring the target image to match the source domain. Furthermore, some important features in the target domain may be lost during the process of style transferring the target image to match the source domain. These features may be crucial for classification tasks. Summary of the Invention

[0005] This invention provides an online test-time adaptive target recognition method based on sample uncertainty estimation, which can effectively solve the problem that the source domain data cannot be obtained during the training of the domain adaptation framework, resulting in poor accuracy due to the need for real-time adaptation of the model.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An online test-time adaptive target recognition method based on sample uncertainty estimation includes the following steps:

[0008] Step 1: Dataset Acquisition Process: First, download the dataset and augment it;

[0009] Step 2: Neural Network Adaptation Process: The model is used to evaluate the current batch of data twice, the uncertainty value of each sample is calculated, and the model is learned and updated.

[0010] Step 3: Test the image recognition process: Use the model that has learned the target data of the current batch to predict the samples and obtain the classification results of the samples in this batch.

[0011] Of the steps described above, step 1 specifically includes the following steps:

[0012] Step 1-1: Download the dataset from the website to obtain several 32×32 images;

[0013] Steps 1-2: Perform random enhancement combinations on each image, including center cropping and random horizontal flipping.

[0014] Step 2 specifically includes the following steps:

[0015] Step 2-1: Using the drop layer of the neural network as the basis for uncertainty estimation, perform model evaluation twice for each sample x. Specifically, feed the sample into the neural network model twice for feature extraction and classification output, obtaining the model's two prediction results, i.e.:

[0016] P T (y|x)=f θ (x) (1)

[0017] Among them, P T (y|x) represents the model prediction result, T represents the number of times the model prediction is performed, and f θ (.) represents the convolutional neural network model, which is specifically expressed as: a convolutional neural network is a deep learning model that extracts image features and performs classification through a series of convolutional layers, pooling layers, and fully connected layers;

[0018] Step 2-2: Calculate the difference between the two model evaluation results to obtain the uncertainty estimate for each sample. The calculation formula is as follows:

[0019]

[0020] Where C represents the number of categories. represents the probability value when the current class is c, and t represents the model prediction result of the tth time;

[0021] Steps 2-3: Optimize the original entropy-based domain adaptation loss by weighting the loss based on the uncertainty values ​​of the samples, using the network output as the calculated value, and calculating the loss according to the uncertainty weight values. The weighted loss is as follows:

[0022]

[0023] Where C represents the number of categories, and c takes values ​​of 1, 2, 3...C, P c This represents the probability value when the item belongs to category c.

[0024] Steps 2-4: Set the learning rate and batch size values ​​on the neural network to adapt to the hyperparameter network training, and optimize using the Adam optimizer.

[0025] Step 3 specifically includes the following steps:

[0026] Step 3-1: Feed the test image and two augmented images into the updated neural network;

[0027] Step 3-2: Use the model to predict the original image and the two augmented images to obtain three prediction results; that is, the original image is provided as input to the model, and features are extracted through the network's convolution operation and then predicted and classified through a fully connected layer. At the same time, the two augmented images are also provided to the model. Finally, three prediction results can be obtained, that is, three C-dimensional vectors.

[0028] Step 3-3: Average the three prediction results to obtain the final result.

[0029] Beneficial effects: This invention provides an online test-time adaptive target recognition method based on sample uncertainty estimation. By introducing uncertainty assessment of samples, it solves the key problem of unreliable model prediction of pseudo-labels. By utilizing weighted entropy loss, it can more effectively mine the feature structure information and underlying distribution information of target domain data, thereby improving the accuracy of object classification. This invention is easier to implement than existing technologies and effectively solves the problem that current online test-time adaptive frameworks cannot obtain target domain data and need to perform real-time recognition, thus improving the accuracy of intelligent transportation in recognizing vehicle target objects during online testing. Attached Figure Description

[0030] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0031] Figure 2 This is a flowchart of step 1 of an embodiment of the present invention.

[0032] Figure 3 This is a flowchart of step 2 of an embodiment of the present invention.

[0033] Figure 4 This is a flowchart of step 3 of an embodiment of the present invention.

[0034] Figure 5 This is a test result diagram of the identification method in an embodiment of the present invention. Detailed Implementation

[0035] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0036] like Figure 1 As shown, this embodiment is applied to real-time object recognition in actual traffic environments. Specifically, it is an online intelligent traffic adaptive target recognition method based on sample uncertainty estimation, including the following steps:

[0037] Step 1: As Figure 2 As shown, data was downloaded from the official websites of CIFAR-C and CLAD-C datasets, resulting in several 32×32 images. Each image was then randomly augmented, including center cropping and random horizontal flipping. The original images were center-cropped to the original size, and the probability of random horizontal flipping was 0.5.

[0038] Step 2: As Figure 3 As shown, neural network learning and adaptation are performed. The model is used to obtain two evaluations of the current batch of data. Then, the uncertainty value of each sample is calculated based on the two evaluations, and weighted loss is calculated and the model is learned and updated. The specific steps include the following:

[0039] Step 201: Introduce the drop layer feature of the network, randomly ignoring a portion of the model's neurons, and repeatedly feed the samples into the neural network model twice for feature extraction and classification output, obtaining two prediction results:

[0040] P T (y|x)=f θ (x) (1)

[0041] Among them, P T (y|x) represents the model prediction result, T represents the number of times the model prediction is performed, and f θ (.) represents a neural network model. A convolutional neural network is a deep learning model that extracts image features and performs classification through a series of convolutional layers, pooling layers, and fully connected layers.

[0042] Step 202: Calculate the difference between the two model predictions and use this difference as the uncertainty estimate for each sample. The calculation formula is as follows:

[0043]

[0044] Where C represents the number of categories; This represents the probability value when the current element belongs to category c; t represents the model prediction result for the t-th time.

[0045] Step 203: The uncertainty value of the sample is used as a weight for the entropy loss. This improves the original entropy-based domain adaptation loss, where the weight value for each sample is 1. The model's predicted output is used as the entropy calculation value, and the uncertainty value is introduced as a weight value to calculate the weighted entropy loss. The weighted loss is as follows:

[0046]

[0047] Where C represents the number of categories, and c takes values ​​of 1, 2, 3...C, P c This represents the probability value when the item belongs to category c.

[0048] Step 204: Set the hyperparameters for the neural network model during adaptation. Set the learning rate to 0.001 and the batch size to 110. Use the Adam optimizer to optimize and adapt the model.

[0049] Step 3: As Figure 4 As shown, the model that has learned the target data of the current batch is used to predict the samples and obtain the classification results of the samples in this batch. The specific steps include the following:

[0050] Step 301: Feed the original image into the updated neural network, and also feed the two augmented images into the neural network for prediction;

[0051] Step 302: The original image is provided as input to the model. After feature extraction through convolution operations and other processes, it is predicted and classified through a fully connected layer. At the same time, two augmented images are also provided to the model. Finally, three prediction results can be obtained, which are three C-dimensional vectors, representing the model's predictions of the original image and the two augmented images.

[0052] Step 403: Take the arithmetic mean of the three prediction results to obtain the final prediction result.

[0053] Figure 5To test the recognition results using the above method, the experiment was conducted on two TITAN Xp graphics cards. The learning rate was set to 0.001 during model adaptation. Every 6 batches, the accuracy value of the batch data was output to the terminal to facilitate observation of the overall convergence. Finally, the overall accuracy was calculated at the end, and the verification results were output. In each domain adaptation task, the source domain and target domain corresponding to the task will be indicated. For example, brightness 5 indicates that the current prediction result of the network trained on the source domain CIFAR dataset in the target domain CIFAR10-C with damage type of brightness and damage level of 5 can achieve a classification accuracy of 92.09%.

[0054] The embodiments described above are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. An online test-time adaptive target recognition method based on sample uncertainty estimation, characterized in that, Comprising the following steps: Step 1: Dataset acquisition process: first download the dataset and augment the dataset; Step 2: Neural network adaptation process: use the model to evaluate the current batch of data twice, calculate the uncertainty value of each sample, and update the model learning; Specifically, it includes the following steps: Step 2-1: Use the drop layer of the neural network as the basis for uncertainty estimation, evaluate each sample x twice, and get the model prediction results twice; Step 2-2: Calculate the difference between the two model evaluation results to get the uncertainty estimate value of each sample. The calculation formula for the difference between the two model evaluation results is as follows: ， where C represents the number of categories; represents the probability value when the current belongs to the C category; t represents the tth model prediction result, and T represents the number of model predictions. Step 2-3: Optimize the original entropy-based domain adaptation loss, use the sample uncertainty value as the loss weight for weighting, use the network output as the calculation value, and calculate the loss according to the uncertainty weight value. The weighted loss is: ， where C represents the number of classes, C takes values 1, 2, 3,..., C, probability value when belonging to the Cth class; Step 2-4: Set the learning rate and batch size value of the adaptation hyperparameter network training on the neural network, and use the Adam optimizer for optimization; Step 3: Test image recognition process: use the model that has learned the current batch of target data to predict the sample and get the classification results of the batch of samples, which includes the following steps: Step 3-1: Send the test image and two augmented images into the updated neural network; Step 3-2: Perform feature extraction through convolution operation and prediction classification through fully connected layer to get the prediction of the original image and two augmented images, and get three prediction results; Step 3-3: Average the three prediction results to get the final result.

2. The online test-time adaptive target recognition method based on sample uncertainty estimation according to claim 1, characterized in that, Step 1 specifically includes the following steps: Step 1-1: Download the dataset from the website to get several images; Step 1-2: Randomly enhance each image.

3. The online test-time adaptive target recognition method based on sample uncertainty estimation according to claim 2, characterized in that, Randomly enhance each image includes center cropping and random horizontal flipping. The center cropping size is the original image size, and the random horizontal flipping probability is 0.

5.

4. The online test-time adaptive target recognition method based on sample uncertainty estimation of claim 1, wherein, The model prediction results in step 2-1 are: ， wherein, denotes the model prediction result, T denotes the number of times of performing model prediction, is a neural network model.

Images