Image classification domain generalization method and device for test time, and electronic device

Abstract

The present invention discloses a method and device for image classification domain generalization at test time, and an electronic device, comprising: at test time, inputting a target domain image into a deep neural network model trained by source domain data, performing a first prediction, and constructing a pseudo label, wherein the deep neural network model is composed of a feature extractor and a linear classifier; replacing the linear classifier with a pseudo prototype classifier, and updating the pseudo prototype classifier based on the pseudo label using feature representations of target domain image samples whose predicted entropy values ​​are lower than a set value; performing secondary classification using the updated pseudo prototype classifier, and constructing secondary pseudo labels; performing initial model prediction using the deep neural network model that has not been updated with the target domain image; filtering and scoring the target domain image samples based on the first prediction and the initial model prediction; and updating the feature extractor based on the filtering information, the sample scores, and the secondary pseudo labels.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for generalizing image classification domains during test time, and an electronic device. Background Art

[0002] Image classification, which involves assigning a label from a set of class labels to an input image using a computer algorithm, is a fundamental task in computer vision and is widely used in numerous fields, including facial recognition, medical imaging diagnosis, and assisted driving. The rapid development of deep neural networks has made them an indispensable component of image classification systems, making the prevention of catastrophic network failure a crucial issue. One of the key causes of catastrophic network failure is distribution shift, where the characteristic distribution of the data used during network training differs from the characteristic distribution of the data encountered during actual network application. Numerous solutions have been proposed to address this distribution shift, with domain generalization being one such approach.

[0003] Existing domain generalization methods can be divided into two categories based on the application phase. One category involves methods applied in the training phase, and the other involves methods applied in the testing phase. Methods applied in the training phase aim to improve the generalization performance of neural network models by leveraging training data (source domain data) during the training phase. Methods applied in the testing phase aim to directly utilize test data (target domain data) during the testing phase to improve the generalization performance of neural network models.

[0004] Based on the different methods used to update deep neural network models, domain generalization methods used in the testing phase can be divided into two categories: methods based on classifier updates and methods based on feature extractor updates. The former aims to adjust the model's decision boundary by adaptively adjusting the classifier; the latter aims to update certain parameters in the feature extractor to improve the model's feature extraction capabilities for the target domain data.

[0005] Existing domain generalization methods used in the testing phase usually select the model's classifier or feature extractor for adaptive adjustment or update. These methods have the following disadvantages:

[0006] (1) Updating the classifier or feature extractor of a deep learning model alone limits the improvement of the model's generalization performance. The reason is that if the model's performance deteriorates in the target domain mainly because the model does not extract the target domain data features well enough, then adjusting the classifier will not effectively improve the model's generalization performance; if the model's performance deteriorates in the target domain mainly because the model's decision boundary is not clear enough, then updating the feature extractor will not effectively improve the model's generalization performance.

[0007] (2) Domain generalization methods based on feature extractor updates typically select a small number of parameters within the model for update, which limits the improvement of model generalization performance. At the same time, if existing methods use unlabeled target domain data to update all parameters within the feature extractor, due to the error signals introduced by the unsupervised target, not only will it not bring about an improvement in generalization performance, but it will also cause the model performance to deteriorate. Summary of the Invention

[0008] The purpose of the embodiments of the present application is to provide a method and device for generalizing the image classification domain during test time, and an electronic device, so as to solve the technical problems existing in the related art in that only the classifier or feature extractor of the deep learning model is updated separately during test time, and only a small part of the parameters of the feature extractor are selected for update.

[0009] According to a first aspect of an embodiment of the present application, a method for image classification domain generalization at test time is provided, comprising:

[0010] At test time, the target domain image is input into a deep neural network model trained with source domain data, and a first prediction is performed on it to obtain the feature representation and first classification results of the target domain image, and a pseudo label is constructed based on the classification results. The deep neural network model consists of a feature extractor and a linear classifier.

[0011] Using a pseudo prototype classifier to replace the linear classifier, and updating the pseudo prototype classifier based on the pseudo label using the feature representation of the target domain image sample whose predicted entropy value is lower than a set value;

[0012] Performing secondary classification on the feature representation of the target domain image using the updated pseudo prototype classifier to obtain a final classification result, and constructing a secondary pseudo label based on the secondary classification result;

[0013] Performing initial model prediction on the target domain image using the deep neural network model that has not been updated with the target domain image, and obtaining the initial model prediction confidence of the target domain image;

[0014] Filter unreliable target domain image samples according to the first prediction and the initial model prediction confidence, and score the image samples according to the reliability and the offset degree;

[0015] The feature extractor is updated based on the filtering information and the sample scores, and the update trend of the feature extractor is adjusted according to the secondary pseudo-labels.

[0016] Furthermore, at test time, the target domain image is input into the deep neural network model trained with the source domain data, and a first prediction is performed on it to obtain the feature representation and first classification result of the target domain image, and a pseudo label is constructed based on the classification result, including:

[0017] Scale the target domain image;

[0018] Input the scaled target domain image into the feature extractor to extract the feature representation of the target domain image;

[0019] The feature representation of the target domain image is input into the linear classifier to obtain the first classification result of the target domain image [p1, p2, ..., p K ], where K represents the number of categories, p K represents the probability that the image belongs to category K;

[0020] The p with the largest predicted probability u The corresponding category u is used as the pseudo label of the target domain image

[0021] Furthermore, a pseudo prototype classifier is used to replace the linear classifier, and based on the pseudo label, the pseudo prototype classifier is updated using the feature representation of the target domain image sample whose predicted entropy value is lower than the set value, including:

[0022] Create support set [S 1 ,S 2 ,…,S K ], the construction of pseudo-prototype classifier relies on the creation of support set;

[0023] Use the weights [w1,w2,…,w K ] for the support set [S 1 ,S 2 ,…,S K ] is initialized, which is equivalent to initializing the pseudo prototype classifier, where ω k is a vector with the same dimension as the feature extractor output;

[0024] by Represents the state of the support set corresponding to category k at time t, based on the pseudo label The feature representation of the target domain image belonging to category k is used to update the corresponding support set This updating process will be repeated N times, where N is the number of target domain image samples arriving at time t;

[0025] According to the size of the predicted entropy value, the support set Filter the inner elements.

[0026] Furthermore, the updated pseudo prototype classifier is used to perform secondary classification on the feature representation of the target domain image to obtain a final classification result, and a secondary pseudo label is constructed based on the secondary classification result, including:

[0027] Updated and filtered support set The weighted average of the elements in each support set is used to obtain the centroid of each support set That is, the weights corresponding to each category of the pseudo prototype classifier at time t;

[0028] According to the centroids of each support set, the target domain image is secondary classified by calculating the cosine approximation between the feature representation of the target domain image and the centroids of each support set, and the secondary classification result of the target domain image is obtained [p1 ′ ,p ′ 2,…,p ′ K ];

[0029] The p with the largest predicted probability ′ u The corresponding category u is used as the secondary pseudo label of the target domain image

[0030] Furthermore, performing initial model prediction on the target domain image using the deep neural network model that has not been updated with the target domain image to obtain the initial model prediction confidence of the target domain image includes:

[0031] Scale the target domain image;

[0032] Input the scaled target domain image into the feature extractor that has not been updated with the target domain image to extract the initial feature representation of the target domain image;

[0033] Input the initial feature representation of the target domain image into the linear classifier to obtain the initial model classification result of the target domain image

[0034] The most likely prediction As the initial model prediction confidence of the target domain image.

[0035] Furthermore, based on the first prediction and the initial model prediction, unreliable target domain image samples are filtered out, and the image samples are scored according to reliability and deviation, including:

[0036] According to the predicted entropy value of the target domain image sample calculated by the first prediction result, the reliability of the target domain image sample is judged, and unreliable samples with a value higher than a preset entropy threshold are filtered out to obtain reliable samples;

[0037] The target domain image samples are scored according to the reliability to obtain a reliability score. The greater the reliability, the higher the reliability score.

[0038] Predict confidence based on the initial model Determine the degree of deviation of the target domain image samples compared to the source domain data distribution, score the target domain image samples according to the degree of deviation, and obtain the deviation score. The smaller the deviation, the higher the deviation score.

[0039] Furthermore, the feature extractor is updated based on the filtering information and the sample score, and the update trend of the feature extractor is adjusted according to the secondary pseudo-label, including:

[0040] The filter information, reliability score and offset score of the target image sample are combined with the information maximization loss to construct the reliable sample information maximization loss.

[0041] According to the secondary classification results [p1 ′ ,p ′ 2,…,p ′ K ], if the p with the largest predicted probability ′ u If the confidence level is greater than the preset threshold, the secondary pseudo-label is used. Perform supervision and construct cross entropy loss

[0042] Maximize loss by combining reliable sample information and cross entropy loss The feature extractor is updated by stochastic gradient descent (SGD).

[0043] According to a second aspect of an embodiment of the present application, a device for image classification domain generalization at test time is provided, comprising:

[0044] A first prediction module is used to input the target domain image into the deep neural network model trained with the source domain data at test time, perform the first prediction on it, obtain the feature representation and first classification result of the target domain image, and construct a pseudo label based on the classification result. The deep neural network model consists of a feature extractor and a linear classifier;

[0045] a classifier updating module, configured to replace the linear classifier with a pseudo prototype classifier, and update the pseudo prototype classifier based on the pseudo label using a feature representation of a target domain image sample whose predicted entropy value is lower than a set value;

[0046] A secondary classification module is used to perform secondary classification on the feature representation of the target domain image using the updated pseudo prototype classifier to obtain a final classification result, and to construct a secondary pseudo label based on the secondary classification result;

[0047] an initial model prediction module, configured to perform initial model prediction on the target domain image using the deep neural network model that has not been updated with the target domain image, and obtain the initial model prediction confidence of the target domain image;

[0048] A reliable sample identification module is used to filter unreliable target domain image samples based on the first prediction and the initial model prediction confidence, and score the image samples according to the reliability and deviation degree;

[0049] The feature extractor update module is used to update the feature extractor based on the filtering information and the sample score, and adjust the update trend of the feature extractor according to the secondary pseudo-label.

[0050] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including:

[0051] one or more processors;

[0052] a memory for storing one or more programs;

[0053] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0054] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0055] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0056] It can be seen from the above embodiments that the present application is different from the existing methods. The present application uses the target domain data to collaboratively update the feature extractor and classifier in the deep neural network model during test time, which not only enables the feature extraction of the model to be adaptively adjusted, but also improves the decision boundary of the model. In addition, reliable target domain image samples are selected through a reliable sample recognition strategy to update all parameters in the feature extractor, which alleviates the limitation of improving the generalization performance of the model, thereby improving the classification accuracy of the model on the target domain data, and then improving the generalization ability of the model, effectively avoiding the possible shortcomings of the two existing methods.

[0057] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 The flowchart of a method for image classification domain generalization at test time is shown according to an exemplary embodiment.

[0060] Figure 2 The figure is a flowchart of a framework of a method for generalizing image classification domain at test time according to an exemplary embodiment.

[0061] Figure 3 The figure is a block diagram of a test-time image classification domain generalization device according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0063] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0064] Figure 1 The flowchart of a method for image classification domain generalization at test time is shown according to an exemplary embodiment. Figure 2 FIG. 1 is a flowchart of a framework for a method for generalizing image classification domains at test time according to an exemplary embodiment. Figure 1 and Figure 2 As shown, the method may include the following steps:

[0065] S1: At test time, the target domain image is input into the deep neural network model trained with source domain data, and the target domain image is predicted for the first time. The feature representation and the first classification result of the target domain image are obtained, and a pseudo label is constructed based on the classification result. The deep neural network model consists of a feature extractor and a linear classifier.

[0066] S2: using a pseudo prototype classifier to replace the linear classifier, and updating the pseudo prototype classifier based on the pseudo label using the feature representation of the target domain image sample whose predicted entropy value is lower than the set value;

[0067] S3: Using the updated pseudo prototype classifier to perform secondary classification on the feature representation of the target domain image to obtain a final classification result, and constructing a secondary pseudo label based on the secondary classification result;

[0068] S4: performing initial model prediction on the target domain image using the deep neural network model that has not been updated with the target domain image, and obtaining the initial model prediction confidence of the target domain image;

[0069] S5: filtering unreliable target domain image samples according to the first prediction and the initial model prediction confidence, and scoring the image samples according to the reliability and the offset degree;

[0070] S6: updating the feature extractor based on the filtering information and the sample score, and adjusting the update trend of the feature extractor according to the secondary pseudo-label.

[0071] It can be seen from the above embodiments that the present application uses target domain data to collaboratively update the feature extractor and classifier in the deep neural network model during test time, which not only enables the feature extraction of the model to be adaptively adjusted, but also improves the decision boundary of the model. In addition, reliable target domain image samples are selected through a reliable sample recognition strategy to update all parameters in the feature extractor, alleviating the limitations on improving the generalization performance of the model, thereby improving the classification accuracy of the model on the target domain data, and further improving the generalization ability of the model.

[0072] In the specific implementation of S1: at test time, the target domain image is input into the deep neural network model trained with the source domain data, and the first prediction is performed on it to obtain the feature representation and the first classification result of the target domain image, and a pseudo label is constructed based on the classification result, including:

[0073] S11: resizing the target domain image;

[0074] Specifically, the target domain image x is transformed into T The image is scaled to a size of 224px*224px so that the input image meets the input specifications of the deep neural network model.

[0075] The deep neural network model consists of a feature extractor and a linear classifier. θ It consists of a ResNet-50 network with the output dimension changed to d; the linear classifier q ω It consists of a fully connected layer and a Softmax activation function.

[0076] S12: Input the scaled target domain image into the feature extractor to extract the feature representation of the target domain image;

[0077] Specifically, the scaled target domain image is input into the feature extractor f θ , that is, the feature representation f of the target domain image obtained in the ResNet-50 network θ (x T ), which is used for subsequent first prediction, classifier update, secondary classification, and feature extractor update.

[0078] S13: Input the feature representation of the target domain image into the linear classifier to obtain the first classification result [p1, p2, ..., p K ], where K represents the number of categories, p K Indicates the probability that the image belongs to category K;

[0079] Specifically, the feature representation f of the extracted target domain image is θ (x T ) Input into the linear classifier q ω In the example, we get the first prediction result of the target domain image [p1, p2, ..., p K ].

[0080] S14: Take the category u corresponding to the pu with the largest predicted probability as the pseudo label of the target domain image

[0081] Specifically, according to the first prediction results [p1, p2, ..., p K ], the category u corresponding to the pu with the largest predicted probability is used as the pseudo label of the target domain image This allows the classifier to be updated later using target domain image samples with pseudo labels.

[0082] In a specific implementation of S2: using a pseudo prototype classifier to replace the linear classifier, and updating the pseudo prototype classifier based on the pseudo label using the feature representation of the target domain image sample whose predicted entropy value is lower than the set value, including:

[0083] S21: Create support set [S 1 , S 2 ,...,S K ], the construction of pseudo-prototype classifier relies on the creation of support set;

[0084] Specifically, create a support set [S 1 , S 2 ,...,S K ], K is the number of categories, S k It can contain at most M elements, pseudo prototype classifier γ cThe construction of relies on the creation of the support set, and the subsequent update of the pseudo-prototype classifier is achieved by adding elements to the support set.

[0085] S22: Use the weights [ω1, ω2, ..., ω K ] for the support set [S 1 , S 2 ,...,S K ] is initialized, which is equivalent to initializing the pseudo prototype classifier, where ω k is a vector with the same dimension as the feature extractor output;

[0086] Specifically, using the linear classifier q ω The weights corresponding to each category [ω1, ω2, ..., ω K ] for the support set [S 1 , S 2 ,...,S K ] is initialized, that is, the pseudo prototype classifier γ c Initialize, where ω k is the feature extractor f θ Outputs a consistent vector of dimension d. S k The initial state is: where ||ω k || represents the vector ω k The L2 norm of the linear classifier q is based on the assumption that ω The weights corresponding to each category [ω1, ω2, ..., ω K ] is the prototype corresponding to each category in the feature space.

[0087] S23: Represents the state of the support set corresponding to category k at time t, based on the pseudo label The feature representation of the target domain image belonging to category k is used to update the corresponding support set The updating process will be repeated N times, where N is the number of target domain image samples arriving at time t.

[0088] Specifically, according to the pseudo label The feature representation of the target domain image belonging to category k is used to update the corresponding support set The specific details are shown in formula (1). The update process will be repeated N times, where N is the number of target domain image samples arriving at time t.

[0089]

[0090] The benefits of using this set-based update method are twofold. First, the update process is reversible, and previous calibration errors can be corrected by deleting set elements. Second, the set size can be limited to prevent the model from deviating from the original source model during the test time update process.

[0091] S24: Support set based on predicted entropy value Filter the inner elements.

[0092] Specifically, due to the support set The update uses pseudo labels So there may be classification errors, for which we use prediction entropy Filter out unreliable pseudo-label data, as shown in formula (2).

[0093]

[0094] where α k Is a support collection After filtering, the support set can correct the previous classification errors to a certain extent.

[0095] In a specific implementation of S3: using the updated pseudo prototype classifier to perform secondary classification on the feature representation of the target domain image to obtain a final classification result, and constructing a secondary pseudo label based on the secondary classification result, including:

[0096] S31: Update and filter the support set The weighted average of the elements in each support set is used to obtain the centroid of each support set That is, the weights corresponding to each category of the pseudo prototype classifier at time t;

[0097] Specifically, the updated and filtered support set The weighted average of the elements in each support set is used to obtain the centroid of each support set The details are shown in formula (3).

[0098]

[0099] The centroids of each support set obtained That is, the prototype corresponding to each category in the feature space after the update. The subsequent secondary classification of the target domain image is based on the support set centroid conduct.

[0100] S32: Based on the centroids of each support set, the target domain image is secondary classified by calculating the cosine approximation between the feature representation of the target domain image and the centroids of each support set, and the secondary classification result of the target domain image [p′1, p′2, …, p′ K ];

[0101] Specifically, according to the centroid of each support set By calculating the feature representation f of the target domain image θ (x T ) and the centroid of each support set The cosine approximation between the two is used to perform secondary classification of the target domain image x T , as shown in formula (4), the secondary classification results of the target domain image [p′1,p′2,…,p′ K ], which is the final classification result.

[0102]

[0103] The pseudo-prototype classifier determines the category to which the target domain image belongs by judging which category's pseudo-prototype (i.e., the centroid of the set) the feature representation of the target domain image is closest to based on a certain metric in the feature space (Euclidean distance is used here).

[0104] S33: predict the maximum probability p′ u The corresponding category u is used as the secondary pseudo label of the target domain image

[0105] Specifically, according to the secondary classification results of the target domain image [p′1,p′2,…,p′ K ], and predict the p′ with the largest probability u The corresponding category u is used as the secondary pseudo label of the target domain image Used to guide subsequent feature extractor updates.

[0106] In a specific implementation of S4, performing an initial model prediction on the target domain image using the deep neural network model that has not been updated with the target domain image to obtain the initial model prediction confidence of the target domain image includes:

[0107] S41: resizing the target domain image;

[0108] Specifically, the target domain image x is transformed into T The image is scaled to a size of 224px*224px so that the input image meets the input specifications of the deep neural network model.

[0109] S42: Inputting the scaled target domain image into a feature extractor that has not been updated with the target domain image to extract an initial feature representation of the target domain image;

[0110] Specifically, the scaled target domain image is input into the feature extractor that has not been updated with the target domain image. The initial feature representation of the target domain image obtained Used for subsequent initial model predictions.

[0111] S43: Input the initial feature representation of the target domain image into the linear classifier to obtain the initial model classification result of the target domain image

[0112] Specifically, the initial feature representation of the target domain image is Input into the linear classifier q ω In the example above, we get the initial model classification result of the target domain image.

[0113] S44: The predicted probability is the highest As the initial model prediction confidence of the target domain image.

[0114] Specifically, according to the initial model classification results of the target domain image The maximum probability value is As the initial model prediction confidence of the target domain image, as the basis for judging the degree of deviation of the current target domain image sample compared with the source domain data distribution, it participates in the update of the subsequent feature extractor.

[0115] In a specific implementation of S5: filtering unreliable target domain image samples based on the first prediction and the initial model prediction, and scoring the image samples based on reliability and deviation, including:

[0116] S51: judging the reliability of the target domain image sample according to the first prediction entropy value of the target domain image sample calculated based on the first prediction result, filtering out unreliable samples with an entropy value higher than a preset threshold, and obtaining reliable samples;

[0117] Specifically, according to the model, the target domain image x T The first prediction of the corresponding first prediction entropy E(x T ), as shown in formula (5), the lower the first prediction entropy value is, the more reliable the sample is when participating in the feature extractor update, because the lower the prediction entropy value is, the more accurate the model prediction is.

[0118]

[0119] where p k =qω (Y=y k |f θ (x T )) is the target domain image x T The probability of belonging to category k at the first prediction. According to the first prediction entropy value E(x T ), filter out unreliable samples that are higher than the preset entropy threshold E0 and obtain reliable samples, as shown in formula (6).

[0120]

[0121] Among them {·} (·) is the indicator function. First, a batch of target domain image samples that are too unreliable are filtered out.

[0122] S52: scoring the target domain image samples according to the reliability to obtain a reliability score. The greater the reliability, the higher the reliability score.

[0123] Specifically, according to the first prediction entropy value E(x T ), for the target domain image sample x T Scoring, the first prediction entropy value E(x T ), the larger the reliability score, the lower the reliability score, and vice versa, the higher the reliability score, as shown in formula (7). The calculated reliability score is used to determine the contribution of different target domain image samples to the model parameter update. The higher the reliability score, the greater the contribution and the greater the impact on the model parameter update.

[0124]

[0125] S53: Predict confidence based on initial model Determine the degree of deviation of the target domain image samples compared to the source domain data distribution, score the target domain image samples according to the degree of deviation, and obtain the deviation score. The smaller the deviation, the higher the deviation score.

[0126] Specifically, according to the initial model prediction confidence For the target domain image sample x T Scoring, initial model prediction confidence The higher the value, the smaller the degree of deviation, so the deviation score is higher. Conversely, the deviation score is lower, as shown in formula (8). The calculated deviation score is also used to determine the contribution of different target domain image samples to the model parameter update. The higher the deviation score, the greater the contribution and the greater the impact on the model parameter update.

[0127]

[0128] where p this the prediction confidence threshold.

[0129] In a specific implementation of S6, the feature extractor is updated based on the filtering information and the sample score, and the update trend of the feature extractor is adjusted based on the secondary pseudo-label, including:

[0130] S61: Combine the filtering information, reliability score and offset score of the target image sample with the information maximization loss to construct the reliable sample information maximization loss

[0131] Specifically, the target image sample x T Filter information S0(x T ), reliability score S1(x T ) and the offset score S2(x T ) is combined with information maximization loss to construct reliable sample information maximization loss The details are shown in formulas (9), (10), and (11). The reliable sample information maximizes the loss Depend on and composition. The entropy minimization loss is used to encourage the model to be more confident in its predictions for a single target domain image sample. The average entropy maximization loss is used to force the model to make the average prediction distribution of samples in the current batch close to a uniform distribution.

[0132]

[0133]

[0134]

[0135] Where S(x T )=S0(x T )·S1(x T )·S2(x T ) is the total score of reliable samples, calculate S(x T ) does not involve any gradient backpropagation, is the average predicted probability of a batch of target domain image samples in category k at the current time t. This part of the update of the feature extractor does not involve the pseudo prototype classifier, ensuring the stability of the feature extractor update.

[0136] S62: Based on the secondary classification results [p′1,p′2,…,p′ K ], if the p with the largest predicted probability ′ uIf the confidence level is greater than the preset threshold, the secondary pseudo-label is used. Perform supervision and construct cross entropy loss

[0137] Specifically, using the target domain image x T The predicted probabilities of each category obtained after the second prediction [p′1,p′2,…,p′ K ], and according to the maximum probability p′ u The pseudo labels obtained If p′ u Greater than the preset confidence threshold p′ th , then the secondary pseudo label is used Perform supervision and construct cross entropy loss The details are shown in formula (12).

[0138]

[0139] The update of the feature extractor here incorporates the classification results and secondary pseudo-labels obtained by the pseudo-prototype classifier, so that there is a certain consistency between the update of the feature extractor and the update of the pseudo-prototype classifier.

[0140] S63: Maximizing loss by combining reliable sample information and cross entropy loss The feature extractor is updated by stochastic gradient descent (SGD).

[0141] Specifically, combining reliable sample information to maximize the loss and cross entropy loss Constructing the overall loss Specifically, as shown in formula (13), the feature extractor f is updated by stochastic gradient descent. θ All parameters, each target domain image sample participates in the feature extractor f at most once θ Parameter update.

[0142]

[0143] Where α is a hyperparameter used to balance Impact on total losses.

[0144] The present invention uses four datasets commonly used in domain generalization image classification to evaluate the performance of the present invention, and the datasets are VLCS, PACS, OfficeHome, and TerraIncognita. The four datasets are all composed of four domains, where the images of each domain in VLCS are divided into the same five categories, and the images of each domain in PACS are divided into the same seven categories. The images of each domain in OfficeHome are divided into the same 65 categories, and the images of each domain in TerraIncognita are divided into the same 10 categories. When conducting experiments on each dataset, one of the domains is selected as the inaccessible target domain each time, and the remaining three domains are selected as the source domains, so that four tasks can be composed. The present invention uses the accuracy commonly used in domain generalization image classification as an evaluation indicator, which can measure the performance of a model prediction in general.

[0145] As shown in Table 1, VLCS in the table represents the average accuracy of the four tasks when the VLCS dataset was used for the experiment, and Avg in the last column represents the average accuracy of the model on the four datasets. It can be seen that the present invention achieved the highest accuracy of 88.2% on the PACS dataset, which is significantly higher than the highest score of 86.2% in the previous report, and improved by 3.5% compared to the ERM method we reproduced, and improved by 2.7% compared to the T3A method that only adjusted the classifier. In addition, the average accuracy on the four datasets reached 69.9% (third). It should be noted that since the reproduced ERM method did not adjust the hyperparameters, it was 2.4% lower than the average accuracy of 69.0% in the previous report. As the basic model used by the present invention at test time, it also reduced the average performance of the present invention. In summary, the present invention effectively solves the domain generalization problem of image classification by collaboratively updating the feature extractor and classifier of the model at test time, and outperforms most existing methods in the accuracy of domain generalization image classification.

[0146] Table 1 Experimental results of domain generalization image classification on all datasets

[0147]

[0148]

[0149] Corresponding to the aforementioned embodiment of the image classification domain generalization method for test time, the present application also provides an embodiment of the image classification domain generalization device for test time.

[0150] Figure 3 FIG. 1 is a block diagram of a generalization device for image classification domain at test time according to an exemplary embodiment. Figure 3 , the device comprises:

[0151] First prediction module 1: During test time, the target domain image is input into the deep neural network model trained with source domain data, and the first prediction is performed on it to obtain the feature representation and first classification results of the target domain image, and construct pseudo labels based on the classification results. The deep neural network model consists of a feature extractor and a linear classifier.

[0152] A classifier updating module 2 is configured to replace the linear classifier with a pseudo prototype classifier, and update the pseudo prototype classifier based on the pseudo label using a feature representation of a target domain image sample whose predicted entropy value is lower than a set value;

[0153] A secondary classification module 3 is configured to perform secondary classification on the feature representation of the target domain image using the updated pseudo prototype classifier to obtain a final classification result, and construct a secondary pseudo label based on the secondary classification result;

[0154] An initial model prediction module 4 is configured to perform initial model prediction on the target domain image using the deep neural network model that has not been updated with the target domain image, and obtain the initial model prediction confidence of the target domain image;

[0155] A reliable sample identification module 5 is used to filter unreliable target domain image samples based on the first prediction and the initial model prediction confidence, and score the image samples according to the reliability and deviation degree;

[0156] The feature extractor updating module 6 is used to update the feature extractor based on the filtering information and the sample scores, and adjust the updating trend of the feature extractor according to the secondary pseudo-labels.

[0157] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0158] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0159] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned image classification domain generalization method for test time.

[0160] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned image classification domain generalization method for test time.

[0161] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0162] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for generalizing image classification domains at test time, characterized in that include: At test time, the target domain image is input into a deep neural network model trained with source domain data, and a first prediction is performed on it to obtain the feature representation and first classification results of the target domain image, and a pseudo label is constructed based on the classification results. The deep neural network model consists of a feature extractor and a linear classifier. Using a pseudo prototype classifier to replace the linear classifier, and updating the pseudo prototype classifier based on the pseudo label using the feature representation of the target domain image sample whose predicted entropy value is lower than a set value; Performing secondary classification on the feature representation of the target domain image using the updated pseudo prototype classifier to obtain a final classification result, and constructing a secondary pseudo label based on the secondary classification result; Performing initial model prediction on the target domain image using the deep neural network model that has not been updated with the target domain image, and obtaining the initial model prediction confidence of the target domain image; Filter unreliable target domain image samples according to the first prediction and the initial model prediction confidence, and score the image samples according to the reliability and the offset degree; updating the feature extractor based on the filtering information and the sample scores, and adjusting the updating trend of the feature extractor according to the secondary pseudo-labels; The method of replacing the linear classifier with a pseudo prototype classifier and updating the pseudo prototype classifier based on the pseudo label using the feature representation of the target domain image sample whose predicted entropy value is lower than a set value comprises: Creating a support set ,The construction of pseudo-prototype classifier relies on the creation of support set; Use the weights corresponding to each category of the linear classifier Support set Initialization is equivalent to initializing the pseudo prototype classifier, where is a vector with the same dimension as the feature extractor output; by represent Moment Category The state of the corresponding support set, according to the pseudo label , will belong to the category The feature representation of the target domain image is used to update the corresponding support set , the updating process will repeat Second-rate, for The number of target domain image samples arriving at the moment; According to the size of the predicted entropy value, the support set Filter the inner elements; The updated pseudo prototype classifier is used to perform secondary classification on the feature representation of the target domain image to obtain a final classification result, and a secondary pseudo label is constructed based on the secondary classification result, including: Updated and filtered support set The weighted average of the elements in each support set is used to obtain the centroid of each support set , that is, the pseudo prototype classifier is The weight corresponding to each category at the moment; According to the centroids of each support set, the target domain image is secondary classified by calculating the cosine approximation between the feature representation of the target domain image and the centroids of each support set to obtain the secondary classification result of the target domain image. ; The most likely prediction Corresponding category As secondary pseudo-labels for target domain images .

2. The method according to claim 1, characterized in that During test time, the target domain image is input into the deep neural network model trained with source domain data, and the first prediction is performed to obtain the feature representation and first classification results of the target domain image. The pseudo labels are constructed based on the classification results, including: Scale the target domain image; Input the scaled target domain image into the feature extractor to extract the feature representation of the target domain image; Input the feature representation of the target domain image into the linear classifier to obtain the first classification result of the target domain image ,in K represents the number of categories, Indicates that the image belongs to a category K probability; The most likely prediction Corresponding category As pseudo labels for target domain images .

3. The method according to claim 1, characterized in that Performing an initial model prediction on the target domain image using the deep neural network model that has not been updated with the target domain image to obtain the initial model prediction confidence of the target domain image, including: Scale the target domain image; Input the scaled target domain image into the feature extractor that has not been updated with the target domain image to extract the initial feature representation of the target domain image; Input the initial feature representation of the target domain image into the linear classifier to obtain the initial model classification result of the target domain image ; The most likely prediction As the initial model prediction confidence of the target domain image.

4. The method according to claim 1, wherein According to the first prediction and the initial model prediction, unreliable target domain image samples are filtered out, and the image samples are scored according to reliability and deviation, including: According to the predicted entropy value of the target domain image sample calculated by the first prediction result, the reliability of the target domain image sample is judged, and unreliable samples with a value higher than a preset entropy threshold are filtered out to obtain reliable samples; The target domain image samples are scored according to the reliability to obtain a reliability score. The greater the reliability, the higher the reliability score. Predict confidence based on the initial model , judge the degree of deviation of the target domain image samples compared with the source domain data distribution, score the target domain image samples according to the degree of deviation, and obtain the deviation score. The smaller the deviation, the higher the deviation score.

5. The method according to claim 1, characterized in that The feature extractor is updated based on the filtering information and the sample score. At the same time, the update trend of the feature extractor is adjusted according to the secondary pseudo-label, including: The filtering information, reliability score and offset score of the target image sample are combined with the information maximization loss to construct the reliable sample information maximization loss ; According to the secondary classification results , if the predicted probability is the largest If the confidence level is greater than the preset threshold, the secondary pseudo-label is used. Perform supervision and construct cross entropy loss ; Maximize loss by combining reliable sample information and cross entropy loss , the feature extractor is updated by stochastic gradient descent.

6. A device for generalizing image classification domains at test time, characterized in that: include: A first prediction module is used to input the target domain image into the deep neural network model trained with the source domain data at test time, perform the first prediction on it, obtain the feature representation and first classification result of the target domain image, and construct a pseudo label based on the classification result. The deep neural network model consists of a feature extractor and a linear classifier; a classifier updating module, configured to replace the linear classifier with a pseudo prototype classifier, and update the pseudo prototype classifier based on the pseudo label using a feature representation of a target domain image sample whose predicted entropy value is lower than a set value; A secondary classification module is used to perform secondary classification on the feature representation of the target domain image using the updated pseudo prototype classifier to obtain a final classification result, and to construct a secondary pseudo label based on the secondary classification result; an initial model prediction module, configured to perform initial model prediction on the target domain image using the deep neural network model that has not been updated with the target domain image, and obtain the initial model prediction confidence of the target domain image; A reliable sample identification module is used to filter unreliable target domain image samples based on the first prediction and the initial model prediction confidence, and score the image samples according to the reliability and deviation degree; A feature extractor update module is used to update the feature extractor based on the filtering information and the sample score, and adjust the update trend of the feature extractor according to the secondary pseudo-label; The method of replacing the linear classifier with a pseudo prototype classifier and updating the pseudo prototype classifier based on the pseudo label using the feature representation of the target domain image sample whose predicted entropy value is lower than a set value comprises: Creating a support set ,The construction of pseudo-prototype classifier relies on the creation of support set; Use the weights corresponding to each category of the linear classifier Support set Initialization is equivalent to initializing the pseudo prototype classifier, where is a vector with the same dimension as the feature extractor output; by represent Moment Category The state of the corresponding support set, according to the pseudo label , will belong to the category The feature representation of the target domain image is used to update the corresponding support set , the updating process will repeat Second-rate, for The number of target domain image samples arriving at the moment; According to the size of the predicted entropy value, the support set Filter the inner elements; The updated pseudo prototype classifier is used to perform secondary classification on the feature representation of the target domain image to obtain a final classification result, and a secondary pseudo label is constructed based on the secondary classification result, including: Updated and filtered support set The weighted average of the elements in each support set is used to obtain the centroid of each support set , that is, the pseudo prototype classifier is The weight corresponding to each category at the moment; According to the centroids of each support set, the target domain image is secondary classified by calculating the cosine approximation between the feature representation of the target domain image and the centroids of each support set to obtain the secondary classification result of the target domain image. ; The most likely prediction Corresponding category As secondary pseudo-labels for target domain images .

7. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images