Image classification method based on test time self-adaption

Abstract

The invention discloses an image classification method based on test time self-adaption, and relates to the technical field of computer vision, and the method specifically comprises the steps: 1, carrying out the destructive conversion of a test image data set; step 2, obtaining false label probability distribution through a pre-trained classifier f, and constructing false label probability difference loss; 3, calculating a false label probability difference score, and screening a test image set according to the false label probability difference score, and 4, inputting the screened images into an energy model to obtain an energy score and probability density; 5, constructing an energy optimization target; and step 6, a joint optimization target is constructed through the false label probability difference loss and the energy optimization target and is used for training and testing the time adaptive model so as to realize image classification. According to the method, the false label probability difference is introduced into the energy model, the test data is sensed from two angles of energy and shape information, and the classification accuracy of the model on a test image set is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to an image classification method based on test time adaptation. Background Art

[0002] The emergence of deep learning networks, especially the rise of convolutional neural networks (CNN), has greatly promoted the development of computer vision and achieved remarkable success in many aspects. For example, in the field of object detection, models represented by regional convolutional neural networks (R-CNN) and YOLO can locate and identify multiple objects in an image in real time, playing a key role in applications such as video surveillance, self-driving cars, and face recognition; in image segmentation tasks, deep learning models can accurately identify and distinguish each pixel in an image, which is of great significance to fields such as medical image analysis, satellite image processing, and augmented reality; in the agricultural field, the application of deep learning is becoming more and more extensive, and it plays an important role in improving agricultural production efficiency, monitoring crop health, optimizing resource utilization, and predicting crop yields; in view of this, research in related fields such as image classification has extremely important value and practical significance;

[0003] Deep learning network models usually show excellent performance under the assumption that the training and test data meet the independent and identical distribution. However, real-world application scenarios are often much more complicated than theoretical assumptions. In many cases, the training data and test data may come from different distributions. This phenomenon is called domain shift or domain bias. For example, an autonomous driving system that is well trained on images taken during the day may perform poorly at night or in bad weather conditions. When the distribution of training and test data is inconsistent, the performance of traditional deep learning models tends to decline significantly. This performance degradation not only affects the accuracy of the model, but also may affect the reliability and stability of its decision-making process. Therefore, how to solve the problem of domain bias has attracted widespread attention in the academic community. Scholars at home and abroad have proposed many test time adaptation methods, which aim to improve the generalization ability of the model by fine-tuning the deep learning network model in an unsupervised manner by utilizing unlabeled test data.

[0004] In the field of computer vision, test time adaptive algorithms have achieved certain results; many algorithms are used to deal with domain shift problems, including TENT, DeYO and TEA; the TENT algorithm proposes that the entropy of model prediction is closely related to the model error rate and the degree of data domain shift, and aims to minimize the predicted entropy of test data by optimizing the affine transformation parameters of each batch of data online. However, when the test data encounters the following three situations, the TENT algorithm will be at risk of accuracy collapse: the test batch size is too small, there are multiple distribution shifts in the test batch, and the test batch data shows time series correlation; the DeYO algorithm combines the pseudo-label probability difference with entropy minimization, giving priority to images that mainly contain shape information, and using entropy and pseudo-label probability differences as confidence indicators at the same time, which increases reliability; however, the DeYO algorithm will increase computational overhead, and the model performance depends on hyperparameters. Multiple attempts are required to adjust the hyperparameters on different data sets to achieve the best model performance; the TEA algorithm starts from the perspective of energy model and achieves perception of the test data set by reducing the energy on the test data set; however, TEA needs to consume a lot of additional computing power and space to generate negative samples during optimization;

[0005] In summary, existing test-time adaptive image classification algorithms face the following challenges:

[0006] First, most methods use entropy as a confidence indicator. However, entropy is not reliable when the test data distribution is complex, which is common in reality. This limits the application of the model in the real world.

[0007] Second, existing methods are often insensitive to the margins of the test data distribution, which causes the model to be overly dependent on the margins of the training data distribution, weakening the model's calibration ability on the test data and reducing the accuracy of the prediction.

[0008] Third, due to the limited access to the entire test dataset at the same time, most methods can only adjust the model based on batch data; however, when harmful samples appear in the batch data, it may mislead the model to make wrong predictions, resulting in errors. The errors continue to accumulate with multiple rounds of batch data iterations, which may eventually lead to a significant decrease in model accuracy. Summary of the invention

[0009] The present invention aims to address the deficiencies of the above-mentioned prior art and to provide an image classification method based on test time adaptation, in order to introduce the pseudo-label probability difference into the energy model, perceive the test data from the two perspectives of energy and shape information, thereby improving the classification accuracy of the model on the test image set.

[0010] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0011] The image classification method based on test time adaptation of the present invention is characterized in that it comprises the following steps:

[0012] Step 1: Get a test image dataset with C categories ,in, express The test images, express The total number of test images in ;

[0013] Applying Destructive Transformers For Test images After processing, the damaged image data set is obtained ,in, Represents a collection of corrupted image data The A destructive image, express The total number of corrupted images in ;

[0014] Step 2: Use a pre-trained classifier Predict the test image and the corrupted image to obtain the pseudo-label probability distribution, which is used to establish the pseudo-label probability difference loss ;

[0015] Step 2.1: and Input into the pre-trained classifier respectively In the corresponding The output probability distribution of and The output probability distribution of ,in, express Parameters;

[0016] Step 2.2: Construct the pseudo-label probability difference loss through formula (1) :

[0017] (1)

[0018] In formula (1), express The output probability on the cth class, express Output probability on the cth class;

[0019] Step 3: Calculate the pseudo-label probability difference score based on the pseudo-label probability distribution , and used to filter the test image data set , get the filtered image data set ;

[0020] Step 4: Input pre-trained classifier In, get Represent the set and input it into the energy model for processing to obtain the energy score and probability density;

[0021] Step 5: Build a dataset of negative imaginary images , and input into the energy model for processing to construct the optimization target of the energy model ;

[0022] Step 6: Use formula (9) to construct the joint optimization objective :

[0023] (9)

[0024] Step 7: Use the SGD optimizer to optimize The test-time adaptive network consisting of the score selector and the energy model is trained and computed To optimize the network parameters, the optimal image classification model is obtained for classifying unlabeled images in the test domain.

[0025] The image classification method based on test time adaptation described in the present invention is also characterized in that the step 3 includes the following steps:

[0026] Step 3.1: Calculate using formula (2) and The pseudo-label probability difference score :

[0027] (2)

[0028] In formula (2), Represents the i-th test image The pseudo labels on ;

[0029] Step 3.2: Use formula (3) to Filter and obtain the filtered image data set ,in, represents the hth filtered image, Indicates the number of images after filtering:

[0030] (3)

[0031] In formula (3), is the threshold for selecting samples.

[0032] Further, the step 4 comprises the following steps:

[0033] Step 4.1: Input pre-trained classifier Process and obtain of express ;

[0034] Step 4.2: Use formula (4) to convert of express Convert to Energy score :

[0035] (4)

[0036] In formula (4), represents the energy function, express On the cth class express;

[0037] Step 4.3: Through formula (5), we can get Probability density on energy model :

[0038] (5)

[0039] In formula (5), Represents differential.

[0040] Further, the step 4 comprises the following steps:

[0041] Step 5.1: Define the current number of iterations as , and initialize ;

[0042] Step 5.2: Randomly generate Negative imaginary image dataset of iterations , Indicates The iteration A fictional image, indicates the total number of negative fictional images;

[0043] Step 5.3: Input pre-trained classifier Processed in of express , and through formula (4) Convert to Energy score , we can get The probability density of ;

[0044] Step 5.4: Use formula (6) to get The iteration Fictional image , thus obtaining Negative imaginary image dataset of iterations ;

[0045] (6)

[0046] In formula (6), represents the learning rate, is random noise and follows normal distribution;

[0047] Step 5.5: After assigning t+1 to t, return to step 5.2 and execute sequentially until t>T, thus obtaining the first Negative imaginary image dataset of iterations , and recorded as the final negative fictional image data set ,in, For the The iteration A fictional image, Indicates A fictional image, Indicates the total number of iterations;

[0048] Step 5.6: Use equations (7) and (8) to construct the optimization objectives of the energy model respectively And its gradient update formula:

[0049] (7)

[0050] (8)

[0051] In formula (8), express exist expectations, express energy score.

[0052] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the image classification method, and the processor is configured to execute the program stored in the memory.

[0053] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the image classification method when the computer program is executed by a processor.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. In step 3, the present invention uses the pseudo-label probability difference score to screen the test image set, thereby avoiding the misleading of the model by harmful images mainly containing false correlation factors and reducing the accumulation of errors; at the same time, since harmful samples are excluded, the scale of the test image set is reduced, the computing power and space required for energy model optimization are reduced to a certain extent, the computational efficiency of the image classification model is improved, and it has a high practical value.

[0056] 2. In step 6, the present invention adds a new consideration to the energy model by introducing a new confidence indicator, namely, pseudo-label probability difference. Energy considers the performance of the entire set based on the test data set, while pseudo-label probability difference considers the difference of each image based on the shape information of the image. This enhances the model's ability to adapt to test data and improves the accuracy of the model's classification tasks in test image data.

[0057] 3. The present invention adopts an energy model to enhance the perception of the edge of the test image data distribution by reducing the energy score on the test image data set, thereby realizing the calibration between the training image data distribution and the test image data distribution; energy is more reliable than entropy, more robust in the face of complex distribution of test image data, better able to adapt to complex changes in the real world, and has higher universality;

[0058] 4. The present invention is intended for practical application fields, such as: the manufacturing industry can detect real-time images of products on the production line to immediately discover defective products and production anomalies, thereby ensuring product quality and reducing production waste; the agricultural sector can use satellite images and farmland images taken by drones to quickly assess crop growth conditions and pest and disease conditions, facilitate precise irrigation and pesticide spraying, and improve crop yield and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] See also Figure 1 As shown, an image classification method based on test time adaptation includes the following steps:

[0062] In the actual implementation process, the test data set will be divided into multiple batches of data. In order to elaborate on the technical solution of the present invention, the test image data set in the following steps is is a batch in the batch data. The model trains one batch of data each time and is gradually optimized through multiple batches. After traversing all batches of data, the optimal model is obtained, which is used to classify unlabeled images in the test domain.

[0063] Step 1: Get a test image dataset with C categories ,in, express The test images, express The total number of test images in ;

[0064] Applying Destructive Transformers For Test images After processing, the damaged image data set is obtained ,in, Represents a collection of corrupted image data The A destructive image, express The total number of corrupted images in ;

[0065] Destructive Converter The specific implementation method is: divide the image into multiple small blocks and randomly arrange the small blocks in a shuffled order while keeping the image size unchanged.

[0066] Step 2: Use a pre-trained classifier Predict the test image and the corrupted image to obtain the pseudo-label probability distribution, which is used to establish the pseudo-label probability difference loss ;

[0067] Step 2.1: and Input into the pre-trained classifier respectively In the corresponding The output probability distribution of and The output probability distribution of ,in, express Parameters;

[0068] definition is a factor positively correlated with the label, which is often a real correlation factor, such as the structure and shape of the object in the picture. Factors that are positively correlated with the label only during training are often false correlation factors, such as background, weather, etc. It can improve model training performance, but it will reduce model reasoning performance. If the model prediction mainly relies on , then the risk of misprediction is high under data distribution shift. is more reliable; pseudo label probability difference loss By reducing the gap between the test image distribution and the damage image distribution, Negative impact on the model, improve The influence of factors can enhance the robustness of the model. The specific process is shown in step 2.2;

[0069] Step 2.2: Construct the pseudo-label probability difference loss through formula (1) :

[0070] (1)

[0071] In formula (1), express The output probability on the cth class, express Output probability for the cth class.

[0072] Step 3: Calculate the pseudo-label probability difference score based on the pseudo-label probability distribution , and used to filter the test image data set , get the filtered image data set ;

[0073] Step 3.1: Calculate using formula (2) and The pseudo-label probability difference score :

[0074] (2)

[0075] In formula (2), Represents the i-th test image The pseudo labels on ;

[0076] Pseudo label probability difference score The main factor in evaluating an image is whether the shape information of the image before and after the damage is different by quantifying the classifier. still , and then determine whether this image will harm the model's reasoning performance;

[0077] By removing harmful samples in the test image set that have a negative impact on the model performance, The image with the main influencing factor can improve the reliability of model prediction, which is of great significance for reducing error accumulation. At the same time, it can reduce the scale of the test image data set and speed up the training of the model. The specific process is shown in step 3.2.

[0078] Step 3.2: Use formula (3) to Filter and obtain the filtered image data set ,in, represents the hth filtered image, Indicates the number of images after filtering:

[0079] (3)

[0080] In formula (3), is the threshold for selecting samples.

[0081] Step 4: Input pre-trained classifier In, get Represent the set and input it into the energy model for processing to obtain the energy score and probability density;

[0082] Step 4.1: Input pre-trained classifier Process and obtain of express ;

[0083] Step 4.2: Use formula (4) to convert of express Convert to Energy score :

[0084] (4)

[0085] In formula (4), express On the cth class express; represents the energy function, the energy function Assign an energy score to each image. In energy models, lower energy usually indicates that the sample is more consistent with the distribution learned by the model, and higher energy indicates that the sample does not conform to the distribution learned by the model; in a pre-trained classifier, the energy of a sample for the corresponding category can be represented as its energy generated by the trained classifier. express.

[0086] Step 4.3: Through formula (5), we can get Probability density on energy model :

[0087] (5)

[0088] In formula (5), express The differential of .

[0089] Step 5: Construct the optimization target of the energy model;

[0090] Step 5.1: Define the current number of iterations as , and initialize ;

[0091] Step 5.2: Randomly generate Negative imaginary image dataset of iterations , Indicates The iteration A fictional image, indicates the total number of negative fictional images;

[0092] Step 5.3: Input pre-trained classifier Processed in of express , and through formula (4) Convert to Energy score , we can get The probability density of ;

[0093] Step 5.4: Use formula (6) to get The iteration Fictional image , thus obtaining Negative imaginary image dataset of iterations ;

[0094] (6)

[0095] In formula (6), represents the learning rate, is random noise and follows normal distribution;

[0096] Step 5.5: After assigning t+1 to t, return to step 5.2 and execute sequentially until t>T, thus obtaining the first Negative imaginary image dataset of iterations , and recorded as the final negative fictional image data set ,in, Indicates A fictional image, Indicates the total number of iterations;

[0097] Negative fictional image collection Generated by stochastic gradient Langevin dynamics (SGLD), when iterated, Optimize along the direction of decreasing energy in the distribution of the pre-trained classifier, and finally generate a set of imaginary images with lower energy based on the pre-trained classifier .

[0098] Step 5.6: Use equations (7) and (8) to construct energy optimization objectives respectively And its gradient update formula:

[0099] (7)

[0100] (8)

[0101] Formula (7) maximizes the distance between the two energy distributions by reducing the energy of the test data distribution while increasing the energy of the fictitious image distribution. Reducing the energy of the test data distribution can effectively enhance the model's perception of the edge of the test data distribution, improve the model's performance on the test domain, and amplify the fictitious image set. The energy can effectively prevent the energy from being minimized indiscriminately in the entire space, thereby avoiding falling into a trivial solution and causing the model performance to collapse.

[0102] In formula (8), express exist expectations, express The optimization process of formula (8) can be understood as a maximum-minimum game. When the energy of the fictitious image distribution generated by the pre-trained classifier matches the energy of the test data distribution, the convergence point of the maximum-minimum game is reached. At this time, the classifier reaches a steady state relative to the test data, and a higher probability is given to samples that are consistent with the test data distribution, and a lower probability is given to samples that deviate from the test data distribution.

[0103] Step 6: Use formula (9) to construct the joint optimization objective :

[0104] (9)

[0105] Step 7: Use the SGD optimizer to optimize The test-time adaptive network consisting of the score selector and the energy model is trained and computed To optimize the network parameters, the optimal image classification model is obtained for classifying unlabeled images in the test domain.

[0106] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0107] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

[0108] As can be seen from the above, the present invention innovatively combines the pseudo-label probability difference with the energy model, effectively enhancing the model's perception of test data and improving the model's classification accuracy for unlabeled images in the test domain;

[0109] The present invention uses the pseudo-label probability difference as a new confidence indicator, reduces the distance between the pseudo-label probability distributions of the test image and the destroyed image, suppresses the influence of false correlation factors, and improves the reliability of model prediction.

[0110] In addition, the present invention uses pseudo-label probability difference scores to screen the test data set to avoid harmful samples mainly containing false correlation factors from misleading the model to make wrong predictions. At the same time, since the scale of test samples is reduced, the efficiency of energy model optimization is improved to a certain extent.

[0111] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A test time adaptive image classification method, characterized in that: The following steps are involved: Step 1: Get a test image dataset with C categories ,in, express The test images, express The total number of test images in ; Applying Destructive Transformers For Test images After processing, the damaged image data set is obtained ,in, Represents a collection of corrupted image data The A destructive image, express The total number of corrupted images in ; Step 2: Use a pre-trained classifier Predict the test image and the corrupted image to obtain the pseudo-label probability distribution, which is used to establish the pseudo-label probability difference loss ; Step 2.1: and Input into the pre-trained classifier respectively In the corresponding The output probability distribution of and The output probability distribution of ,in, express Parameters; Step 2.2: Construct the pseudo-label probability difference loss through formula (1) : (1) In formula (1), express The output probability on the cth class, express Output probability on the cth class; Step 3: Calculate the pseudo-label probability difference score based on the pseudo-label probability distribution , and used to filter the test image data set , get the filtered image data set ; Step 4: Input pre-trained classifier In, get Represent the set and input it into the energy model for processing to obtain the energy score and probability density; Step 5: Build a dataset of negative imaginary images , and input into the energy model for processing to construct the optimization target of the energy model ; Step 6: Use formula (9) to construct the joint optimization objective : (9) Step 7: Use the SGD optimizer to optimize The test-time adaptive network consisting of the score selector and the energy model is trained and computed To optimize the network parameters, the optimal image classification model is obtained for classifying unlabeled images in the test domain.

2. The image classification method based on test time adaptation according to claim 1, characterized in that: The step 3 comprises the following steps: Step 3.1: Calculate using formula (2) and The pseudo-label probability difference score : (2) In formula (2), Represents the i-th test image The pseudo labels on ; Step 3.2: Use formula (3) to Filter and obtain the filtered image data set ,in, represents the hth filtered image, Indicates the number of images after filtering: (3) In formula (3), is the threshold for selecting samples.

3. The image classification method based on test time adaptation according to claim 2 is characterized in that: The step 4 comprises the following steps: Step 4.1: Input pre-trained classifier Process and obtain of express ; Step 4.2: Use formula (4) to convert of express Convert to Energy score : (4) In formula (4), represents the energy function, express On the cth class express; Step 4.3: Through formula (5), we can get Probability density on energy model : (5) In formula (5), Represents differential.

4. The image classification method based on test time adaptation according to claim 3 is characterized in that: The step 4 comprises the following steps: Step 5.1: Define the current number of iterations as , and initialize ; Step 5.2: Randomly generate Negative imaginary image dataset for iterations , Indicates The iteration A fictional image, indicates the total number of negative fictional images; Step 5.3: Input pre-trained classifier Processed in of express , and through formula (4) Convert to Energy score , we can get The probability density of ; Step 5.4: Use formula (6) to get The iteration Fictional image , thus obtaining Negative imaginary image dataset for iterations ; （6） In formula (6), represents the learning rate, is random noise and follows normal distribution; Step 5.5: After assigning t+1 to t, return to step 5.2 and execute sequentially until t>T, thus obtaining the first Negative imaginary image dataset for iterations , and recorded as the final negative fictional image data set ,in, For the The iteration A fictional image, Indicates A fictional image, Indicates the total number of iterations; Step 5.6: Use equations (7) and (8) to construct the optimization objectives of the energy model respectively And its gradient update formula: (7) (8) In formula (8), express exist expectations, express energy score.

5. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the image classification method according to any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image classification method according to any one of claims 1 to 4 are executed.

Images