Cross-domain image classification method

Abstract

The invention relates to a cross-domain image classification method. The method comprises the following steps: preparing an image classification public data set containing a plurality of source domains; resNet-50 is used as a feature extractor G; constructing a classifier C by using linear regression; dividing the data set into a training set and a verification set by adopting a one-taking method, and jointly training G and C to obtain weight parameters of the G and the C; a feature reconstructor H is connected after the G in sequence; two domains are randomly selected from the training set to serve as a comparison data set, a model M = {G, H, C} is jointly trained through sample alignment and data enhancement technologies, and an image classification model M is finally obtained by optimizing model parameters. According to the cross-domain image classification method, a cross-domain image recognition task is realized by approaching a domain invariant feature through a domain common feature.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and particularly relates to a cross-domain image classification method for using pictures in a known domain as prior knowledge to identify images in an unknown domain. Background Art

[0002] Image recognition technologies have developed rapidly. These technologies usually perform well in experimental environments, but their performance in unknown scenarios is not satisfactory. This is because the model itself does not have the ability to obtain invariant features of the recognition target under domain shift. To address the challenges brought by domain shift, the domain generalization (DG) strategy has been proposed. By using DG methods, the model is trained on multiple different domains and evaluated on a new, unseen domain. Mainstream domain generalization methods fall into the category of domain-invariant representation learning methods. This method is based on a widely recognized assumption that each domain contains its unique domain-related features, which often reflect spurious relationships in the data, while all domains share domain-invariant features that are general and robust on any unseen domain. Research has shown that if the feature representations remain invariant between different domains, they are general and transferable. Inspired by this theory, many studies have been devoted to identifying and encoding these domain-invariant features from the source domain. The core concept of these DG methods is to obtain a domain-invariant feature distribution P(F(X)) across different domains to ensure the robustness of the conditional distribution P(Y|F(X)), where F(X) represents the features extracted from the input X and Y is the corresponding label. Existing work has theoretically supported the existence of domain-invariant features. However, it is very difficult to decouple and capture domain-invariant features, and some methods identify other features as domain-invariant. Summary of the Invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a cross-domain image classification method that realizes cross-domain image recognition tasks by approximating domain-invariant features through domain common features.

[0004] The present invention solves its technical problems through the following technical solutions:

[0005] A cross-domain image classification method includes the following steps:

[0006] Step 1, prepare an image classification dataset D containing multiple different domains;

[0007] Step 2, use the leave-one-out method to divide the dataset D into a training set D train and a test set D val ;

[0008] Step 3: Construct an image classification model S, which includes a feature extractor G based on a convolutional neural network and a classifier C based on a linear model, and use the training set D train Jointly train S, and by continuously adjusting the training parameters, verify the performance of the S model on D val Select the result with the highest recognition accuracy from multiple verification results and save the corresponding {G, C} weight parameters w g and w c ;

[0009] Step 4: Build an image network M: Construct a feature extractor G' and a classifier C', make them have the same network structure as the feature extractor G and the classifier C respectively, and add a feature reconstructor H between them, and then use the weight parameters (w g and w c ) of the feature extractor G and the classifier C in Step 3 to initialize G' and C' respectively;

[0010] Step 5: Construct the loss function LOSS of the image network M;

[0011] Step 6: Preprocess the data set D in Step 2 train Generate several combinations from the data subsets included in D train Arbitrarily select a data subset combination {D p , D q}, and through sample alignment, make the number of each classification sample in D p and D q equal;

[0012] Step 7: Use the processed data subset pair D' p and D' q to train the image network M = {G', H, C'};

[0013] Step 8: Repeat Step 6 and Step 7 until the iteration number E s2 is reached, and obtain the weights and training parameters corresponding to the result with the highest recognition accuracy of the model M within no more than the iteration number E s2 ;

[0014] Step 9: Adjust the model training parameters by the control variable method, repeat Step 6, 7, 8, and the weight parameters corresponding to the result with the highest recognition accuracy of the model M.

[0015] Moreover, in the data set D = {D1, D2,... D x ,..., D n} in Step 1, where the data subsets D1, D2,... D x ,..., D nComing from different acquisition environments respectively, the target categories included in the data subsets are the same. At the same time, there are distribution differences between the data subsets, and the intersection of any two data subsets is empty.

[0016] Moreover, in step 2, the training set D train ={D1, D2, …, D n}, the validation set D val ={D x}, D = D train ∪D val .

[0017] Moreover, step 3 includes the following steps:

[0018] Step 3.1: Mix and shuffle the training set D train into D mix , randomly crop and output the sample data in D mix with a fixed size, and then perform a random horizontal flip with a probability of 50% to obtain the processed sample data;

[0019] Step 3.2: Use a convolutional neural network to construct a feature extractor G, whose mathematical representation is the feature extraction function G(·). Select an image recognition model ResNet-50 based on CNN as the feature extractor G and assume it is part of the ideal function;

[0020] Step 3.3: Add a classifier C after G to form an image classification model;

[0021] Step 3.4: Initialize the feature extractor G and the classifier C using the Xavier method;

[0022] Step 3.5: Adopt the supervised learning method, use the data processed in step 3.1 for the joint training of the feature extractor G and the classifier C, and use the cross-entropy loss function as the constraint.

[0023] Moreover, step 4 includes the following steps:

[0024] Step 4.1: Build an image network M: Construct a feature extractor G’ with the same network structure as the feature extractor G and a classifier C’ with the same network structures as the feature extractor G and the classifier C respectively, and then initialize G’ and C’ using the weight parameters of the feature extractor G and the classifier C in step 3;

[0025] Step 4.2, construct the feature extractor G': The final convolutional layer of the CNN-based feature extractor can be interpreted as similar object features. Therefore, the output of the last convolutional layer can be regarded as object features. Further object feature representations are obtained through the feature reconstructor H. Select ResNet-50 as the feature extractor G', and the final object feature representation is f = R(x i ) = H(G'(x i ));

[0026] Step 4.3, construct the feature reconstructor H: Add the feature reconstructor H after the feature representation G to reconstruct the features. Formally, the feature representation is first dimensionally upsampled and then downsampled to achieve the purpose of feature reconstruction. Use a 3-layer MLP to construct the function H(·), and adopt ReLU as the activation function. Place a batch normalization (BatchNormalization) layer between the linear layer and ReLU. Normalization normalizes the data to between (0, 1), accelerating the model convergence speed and improving the model effect;

[0027] Step 4.4, set constraints: The feature reconstructor H generates 4 outputs h u , h u' , h v , h v' . For the output results of every two reconstruction heads, use the loss function formula 1. To avoid the situation of difficult learning due to excessive differences, cross-domain and enhancement will not appear in the same comparison pair at the same time. Finally, construct 4 comparison pairs, (h p′ , h q ), (h p, h q′ ), (h p , h p′ )(h q , h q′ ), as shown in formula 2;

[0028]

[0029] Among them, B represents the size of batchsize, MSE ij represents the mean square error between the i-th sample in b p and the j-th sample in b q . λ is the weight parameter in the loss function, τ is the temperature coefficient, and ∈ is a small constant used to prevent the situation of division by zero,

[0030]

[0031] Step 4.5, set the classifier C': Adopt a 1-layer fully connected network as the classifier and set constraints to satisfy:

[0032]

[0033] Among them, is the cross-entropy loss function.

[0034] Moreover, step 5 includes that the loss function LOSS is:

[0035] LOSS = w1L ctr + w2L mse + w3L cls + γL2 norm (H) (5)

[0036] where w1, w2, w3, and γ are all weight parameters, and L2 norm (H) represents the L2 regularization of the feature reconstruction model H (parameters).

[0037] Moreover, the sample alignment technology is adopted in step 6, which ensures that sample data can be pairwisely enhanced and processed in the case of unbalanced cross-domain sample quantities. The specific algorithm is as follows:

[0038] Input: Source domain pair D of Datasets p and D q ;

[0039] Output: Source domain pair D' with sample alignment p , D' q ;

[0040]

[0041] and have lengths of l p , l q

[0042] if l p > l q :

[0043] Randomly add l - l p - l q samples from its own samples

[0044] if l p < l q :

[0045] Randomly add l - l q - l p samples from its own samples

[0046] return

[0047] Function datasets_pairs(D p ,D q ):

[0048] Initialize sequence D' p ,D' q

[0049] for l in L: # L is the label set of D p ,D q 's label set

[0050] Select samples with label l from D p to generate a set

[0051] Select samples with label l from D q to generate a set

[0052]

[0053] Append to D' p and D' q respectively

[0054] return D′ p ,D′ q

[0055] Moreover, step 7 includes the following steps:

[0056] Step 7.1, Second-stage data augmentation, introducing three types of data augmentation methods: 1) crop: RandomResizedCrop; 2) flipping: RandomHorizontalFlip; 3) transformation: ColorJitter, RandomGrayscale, GaussianBlur. This process includes two steps: preprocessing: processing all data; postprocessing: only processing one branch;

[0057] Step 7.2, Initialize the feature reconstructor H using the Xavier method;

[0058] Step 7.3, Set the number of network layers, the number of neurons, the learning rate, the number of network iterations, and the SGD parameters of H, verify the performance of model M on D val and save the weight parameters corresponding to the highest recognition accuracy result of model M.

[0059] The beneficial effects of the present invention are:

[0060] 1. The cross - domain image classification method of the present invention uses pictures in a known domain as prior knowledge to identify images in an unknown domain, and realizes the cross - domain image recognition task by approximating domain - invariant features through domain common features.

[0061] 2. The cross - domain image classification method of the present invention divides the latent feature space into five parts: domain - invariant, domain - specific, domain - shared, domain - unique, and domain - common. Through reasoning, it is obtained that: on the premise of an ideal data set, when the number of source domains tends to infinity, if there exists a function R(·) such that the features after extracting samples from different source domains are the same and this feature contains the minimum discriminant feature for the classification task, then the function R(·) is an ideal function, and the extracted feature f is a domain - invariant feature. Based on this, a mirror model structure M is proposed, and a training strategy is formulated to train M, so that the present invention achieves good results in the practical application of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a correlation graph of two - domain feature spaces;

[0063] Figure 2 It is a flowchart for establishing a contrastive learning framework of the present invention;

[0064] Figure 3 It is a flowchart of the present invention;

[0065] Figure 4 It is a comparison of saliency maps between ResNet - 50 and the present invention;

[0066] Figure 5 It is a comparison of feature visualizations between ResNet - 50 and the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0067] The present invention will be further described in detail below through specific embodiments. The following embodiments are only descriptive and not restrictive, and the protection scope of the present invention cannot be limited thereby.

[0068] For the convenience of subsequent description, we need to define the range of domain - invariant in the latent feature space, especially to clarify the boundary between domain - invariant and other features. Assume that the feature representation of the target sample x is f = R(x)=[f1, f2, f3……, f n = {t, w}, where R(·) is the feature extraction function. This representation divides the feature f into two parts: the domain - invariant feature t and other features w, and We extend this representation to the case where two domains intersect.

[0069] As Figure 1 shown, if where x u and x vHaving the same label l, under the condition of the function R(·), if there is:

[0070] f u = R(x u ) = {t, w u} = {t, w s , w′ u}

[0071] f v = R(x v ) = {t, w v} = {t, w s , w′ v}

[0072] Among them, then it is defined that:

[0073] Domain invariance: Under the function R(·), the feature t is called the domain invariance of the sample with label l (such as x u , x v ). The feature t reflects the essential characteristics of the sample and does not change with the change of the domain, so it can remain consistent across different domains.

[0074] Domain feature: Under the function R(·), w u and w v are called the domain features of x u and x v . The domain feature is composed of the domain-shared w c and the domain-specific w' u (w' v ).

[0075] Domain sharing: Under the function R(·), if it satisfies w s = w u ∩ w v , then w s is called the domain sharing of x u and x v .

[0076] Domain specific: Under the function R(·), if it satisfies then w' u and w' v are respectively called the domain specific of x u and x v . This means that x u and x v each have independent specific characteristics, and there is no intersection between these characteristics.

[0077] Domain commonality: Under the function R(·), the domain commonality is expressed as w c = f u ∩ f v = ws +t.

[0078] Based on the above definitions, we clearly conclude that domain commonality is not domain invariance. Unfortunately, existing work often equates domain commonality with invariant features, ignoring their differences. In fact, solving for domain invariance is difficult. If the domain sharing in domain commonality can be reduced, then domain commonality will approach domain invariance.

[0079] Suppose there are n observed source domains D = {D1, D2, … D x , …, D n}, which follow different distributions. For each domain D m , it contains multiple sample pairs (x u , y l ), and each x u uniquely corresponds to a label y l . All domains contain the same labels. The goal of domain generalization is to train a model F using the source domain D so that it performs well on unseen target domains D t . Existing DG work is based on the assumption of feature consistency and integrity of samples and aims to obtain a function that can extract complete invariant features of samples. Based on this, we further give the following definitions in the context of classification tasks:

[0080] Definition: Suppose there are n source domains D1, D2, … D x , …, D n with different distributions. Subsets of samples X1, X2, …, X n corresponding to class l are obtained from them. For any sample x i ∈ X m , all the domain-invariant features contained in x i are T. If the sample x i has the minimum discriminative feature set d l for distinguishing other classes, and then x i is called an ideal sample of the sample set D m . A dataset composed entirely of ideal samples is called an ideal dataset. If there exists a function R(·) such that the feature latent space f of the sample x i is f = R(x i ), satisfying then R(·) is called the ideal function of the sample set D m . That is, f is the domain-invariant feature containing the minimum discriminative features.

[0081] Under the concepts of the ideal dataset and the ideal function, we do not seek to capture all the features of the target class, but rather focus on the key discriminative features between classes, and these discriminative features have domain invariance. For example, suppose there is a picture dataset (CD) for identifying animals, which contains two classes: Maine coon cats and dogs. For a Maine coon cat sample x i , its feature representation R(x i ) may include features such as "pointed ears" and "black fur". In the distinction from dogs, the feature of "pointed ears" is sufficient to distinguish Maine coon cats and dogs, because dogs usually do not have the feature of "pointed ears", and "pointed ears" constitutes the minimum discriminative feature set for distinguishing the two If we consider another dataset (CC) that is used to distinguish Maine coon cats and Persian cats, although it contains the same Maine coon cat samples as CD, the trained feature extraction function R(x i ) needs to pay more attention to the subtle differences between Maine coon cats and Persian cats, such as the density of fur and the length of the tail. It can be seen that the differences in extracted features are determined by the samples themselves and the number of classifications.

[0082] Therefore, we redefine the assumptions and goals of the DG task, that is, given the source domain set Assume is the ideal dataset, and solve for 's ideal function R(·).

[0083] Under the condition that the number of source domains tends to infinity, when the condition f p = f q =... = f z is satisfied and it contains the minimum discriminative features, we will be able to obtain the ideal function. For this reason, we make the following inferences:

[0084] Condition: Assume that there are n ideal source domains D1, D2,..., D n with different distributions, where the subsets of samples with label l are X1, X2,..., X n . The feature representation of any sample x i in X i by the feature extraction function R(·) is f i = R(x i ) = {t i , w s , w' i},where t i is domain-invariant, w s is domain-shared, and w' i is domain-specific. f i contains the minimum discriminative feature set d l , x iThe included domain-invariant feature is T i .

[0085] Conclusion: When n → ∞, if f1 = f2 = … = f n , then R(·) is an ideal function.

[0086] Proof:

[0087] According to the definition of domain sharing, when n → ∞, the shared feature w among multiple source domains s tends to disappear:

[0088]

[0089] Because f1 = f2 = … = f n , and f i = {t i , w s , w' i}, we have:

[0090] f i = {t1, w′1} = {t2, w′2} = … = {t n , w′ n} (1)

[0091] Because we have:

[0092] t1 = t2 = … = t n (2)

[0093] Because w' i are domain-specific features that do not intersect, from equations (1) and (2) we get:

[0094]

[0095] From equations (1), (2), and (3) we can obtain:

[0096] f i = R(x i ) = t i

[0097] Because And According to the definition of an ideal function, R(·) is an ideal function.

[0098] The above reasoning process shows that as the number of source domains increases, the distributions of the source domains gradually tend to be the same. This means that although there are differences in data distribution among different source domains (w dThey are different, but these differences will be averaged or diluted as a large number of source domains are introduced, and only those features that are common to all source domains (i.e., invariant feature t) will remain. In other words, in a cross-domain classification task, if there exists a function R(·) that can complete the classification task in an infinite number of domains by obtaining the minimum discriminant features, then the minimum discriminant features obtained by R(·) must be domain-invariant.

[0099] Accordingly, we plan to construct a mirror network structure to learn the domain-invariant features of cross-domain samples by minimizing the feature differences of cross-domain samples. To avoid model underfitting, the assumption condition f p = f p = … = f z degenerates to f p = f q , f p = f z , …, f q = f z , and is implemented using a contrastive learning model. At the same time, supervised learning is used to enable the model to acquire the ability to obtain the minimum discriminant features. Formally, the network structure is simplified from designing a feature representation branch for each source domain, i.e., branch×n, to only designing branch×2 for the contrast source domain. For any sample pair (x p and D q ), (x p , l) and (x q , l), after data augmentation, we get (x' p , l) and (x' q , l). We believe that (x p , x' p , x q , x' q ) have common features, so they have the same label l. Therefore, an additional branch×2 is added to handle the samples from the contrast source domain after data augmentation.

[0100] Based on the above analysis, a cross-domain image classification method is proposed, including the following steps:

[0101] Step 1: Prepare an image classification dataset D = {D1, D2, … D x , …, D n} containing multiple different domains, where the data subsets D1, D2, … D x , …, D n come from different acquisition environments respectively, the target categories contained in the data subsets are the same, there are distribution differences between the data subsets, and the intersection of any two data subsets is empty.

[0102] Step 2: Use the leave-one-out method to divide the dataset D into a training set Dtrain = {D1, D2, …, D n}, the validation set D val = {D x}, D = D train ∪ D val ,

[0103] Step 3, construct an image classification model S, which includes a feature extractor G based on a convolutional neural network and a classifier C based on a linear model. Use the training set D train to jointly train S, and by continuously adjusting the training parameters, verify the performance of S on D val . Select the result with the highest recognition accuracy from multiple verification results. And save the corresponding {G, C} weight parameters w g and w c ;

[0104] Step 3.1, mix and shuffle the training set D train into D mix , and randomly crop the sample data in D mix with a ratio between (0.08, 1), and fix the size to output 224 × 224 pixels, and then perform a random horizontal flip with a probability of 50% to obtain the processed sample data;

[0105] torchvision method parameter ResizedCrop size = 224 HorizontalFlip default

[0106] Step 3.2, use the convolutional neural network model RestNet50 to construct the feature extractor G, and its mathematical representation is the feature extraction function G(·). In the field of computer vision, finding an ideal function has always been a basic research work. However, it is difficult to determine whether a feature extractor is an ideal function. Therefore, the present invention selects an image recognition model ResNet-50 based on CNN as the feature extractor G and assumes that it is part of the ideal function.

[0107] Step 3.2, add the classifier C after G to form an image classification model;

[0108] Step 3.3, initialize the feature extractor G and the classifier C using the Xavier method;

[0109] Step 3.4, adopt the supervised learning method, use the data processed in Step 3.1 for the joint training of the feature extractor G and the classifier C, and use the cross-entropy loss function as the constraint. The training algorithm is as follows:

[0110]

[0111] Step 4, build the mirror network M( Figure 2), which consists of a feature extractor G', a classifier C', and a feature reconstructor H added therebetween.

[0112] Step 4.1: Construct the feature extractor G' to have the same network structure as the feature extractor G, and then use the weight parameter w of the feature extractor G in Step 3 g to initialize G'; the final convolutional layer of the CNN-based feature extractor can be interpreted as similar object features, so the output of the final convolutional layer can be regarded as object features, and further object feature representations are obtained through the feature reconstructor H. Therefore, the final object feature representation is f = R(x i ) = H(G'(x i ))).

[0113] Step 4.2: Construct the feature reconstructor H. Add the feature reconstructor H after the feature representation G to reconstruct the features. Formally, after dimensionality increasing and then dimensionality decreasing the feature representation, the purpose of feature reconstruction is achieved. Use a 3-layer MLP to construct the function H(·), and adopt ReLU as the activation function. Place a batch normalization (BatchNormalization) layer between the linear layer and ReLU. Batch normalization normalizes the data to between (0, 1), accelerating the model convergence speed and improving the model effect.

[0114] Step 4.4: Set constraints. The feature reconstructor H generates 4 outputs h p , h p' , h q , h q' . For each pair of the outputs of the reconstruction heads, use the loss function formula 4. To avoid the situation of difficult learning due to excessive differences, cross-domain and enhancement do not appear in the same comparison pair at the same time. Finally, construct 4 comparison pairs (h p′ , h q ), (h p , h q′ ), (h p , h p′ ), (h q , h q′ ), as shown in Equation 5.

[0115]

[0116] Among them, B represents the size of the batchsize, MSE ij represents the mean square error between the i-th sample in b p and the j-th sample in b q . λ is the weight parameter in the loss function, τ is the temperature coefficient, and ∈ is a small constant used to prevent division by zero.

[0117]

[0118] Step 4.5: Construct a linear classifier C', which has the same network structure as classifier C, and then use the weight parameter w of the linear classifier C in Step 3 c Initialize C' and set constraints to satisfy:

[0119]

[0120] where, is the cross-entropy loss function.

[0121] Step 4.6: Initialize. Load the weight w g and w c Initialize G' and C' respectively, and initialize H using the Xavier method.

[0122] Step 5: Construct the loss function LOSS of the mirror network M.

[0123] LOSS = w1L ctr + w2L mse + w3L cls + γL2 norm (H) (8)

[0124] where, w1, w2, w3, γ are all weight parameters, and L2 norm (H) represents the L2 regularization of the reconstruction model H (parameters).

[0125] Step 6: Preprocess the dataset D in Step 2 train Perform preprocessing.

[0126] Step 6.1: Generate a set of combination pairs P by pairwise combining the data subsets contained in D. Select any combination pair {D train , D p} from P. Through the sample alignment technique, ensure that the number of each classification sample in D q and D p is equal. q

[0127] Step 6.2: The sample alignment technique ensures that in the case of unbalanced cross-domain sample quantities, pairwise data augmentation and processing of the sample data can be performed. The specific algorithm is as follows:

[0128]

[0129]

[0129] Step 7: Use the processed data subset pairs D' p and D' q to train the mirror network M = {G', H, C'}. By continuously adjusting the parameters, obtain the weight parameters corresponding to the highest recognition accuracy result of the model M. The implementation logic is as follows.

[0130] Step 7.1: For samples from D' p and D' q , perform data augmentation, including the methods: 1) crop: RandomResizedCrop; 2) flipping: RandomHorizontalFlip; 3) transformation: ColorJitter, RandomGrayscale, GaussianBlur. This process consists of two steps: 1) preprocessing: process all data; 2) postprocessing: only process one branch.

[0131]

[0132]

[0133] Among them, Color_jitter and Grayscale participate in data augmentation with probability p. See Figure 2 for the application location of each data augmentation.

[0134] Step 7.2: Initialize the feature reconstructor H using the Xavier method.

[0135] Step 7.3: Utilize the samples after data augmentation to train the model M by minimizing the loss function LOSS (Equation 8). Set the number of network layers, the number of neurons, the learning rate, the number of network iterations, and the SGD parameters of H, and verify the performance of the model M on D val . Save the weight parameters w g’ , w h , w c’ corresponding to the result with the highest recognition accuracy of the model M.

[0136] Step 7.4: Repeat Step 6 and Steps 7.1, 7.2 until all combined pairs in P have been traversed.

[0137] Step 8: Repeat Step 6 and Step 7 until the number of iterations E s2 is reached. The algorithm for Step 8 is as follows:

[0138]

[0139] Step 9: Adjust the training parameters such as the model learning rate, Batchsize, and SGD using the method of controlling variables, and repeat Steps 6, 7, 8 to obtain the weight parameters corresponding to the result with the highest recognition accuracy of the model M.

[0140] Finally, use M as the inference model and load the weight parameters of each module {G’, H, C’}. The overall process of the present invention is as shown in Figure 3As shown below. To verify the effectiveness and advancement of the present invention, we respectively used the publicly available datasets PACS and OfficeHome to verify the method of the present invention. Each experiment was run three times. After the training in each domain pair was completed, the performance of the model was tested (if the source domain contains 3 domains and 20 epochs of training are performed, we will obtain 3×20 phased results). The best phased result was taken as the final result of this experiment, and finally the average value of the three experiments was taken. As shown in Table 1 (PACS) and Table 2 (OfficeHome) below, we compared with the research results published in major academic conferences or journals in recent years. The results show that the method of the present invention is superior to the existing methods. As shown in Table 2, our framework achieved the highest average accuracy among all the compared methods. The method of the present invention improved by 0.2% compared with the SOTA method DomainDrop (ICCV’23). As Figure 4 shows the visualization results of the saliency maps generated by the method of the present invention and the baseline method. Generally speaking, compared with the baseline method, our method can better focus on the target itself, while the baseline method focuses on non-causal factors (such as the texture of the picture background), which leads to misclassification. In addition, our method is more accurate than the baseline in the recognition of multi-target samples. In the person domain, our method accurately recognizes the two people in the picture, while the baseline only recognizes one person. In a multi-class classification task, effective feature representations should exhibit high intra-class similarity and low inter-class similarity. To evaluate this, t-SNE was used to visualize the deep features learned by the model. Specifically, the last fully connected layer before the classification layer in the baseline model and the model of the present invention was extracted as the feature layer, and it is shown in Figure 5 the generated t-SNE visualization. In the visualization, each color (or number) corresponds to a different class, and each point represents an individual sample. Tighter clustering within a class indicates greater similarity of the sample features, while less overlap between classes indicates reduced correlation of the inter-class features. Taking PACS as an example, using the cartoon domain as the unseen target and ResNet-50 as the baseline, our method improved the recognition accuracy by 2.6% (see Table 2). This improvement is reflected in Figure 5 where the right image shows a better clustering effect than the left image. This shows that our method is more effective in identifying important, class-specific features, while reducing the impact of spurious correlations or domain-specific correlations, thereby enhancing the distinction between different class samples.

[0141] Table 1. Comparison with the latest methods on PACS

[0142]

[0143] Table 2. Comparison with the latest methods on OfficeHome

[0144]

[0145]

[0146] Although the embodiments and drawings of the present invention are disclosed for illustrative purposes, those skilled in the art can understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the content disclosed in the embodiments and drawings.

Claims

1. A cross-domain image classification method, characterized by: The following steps are involved: Step 1: Prepare an image classification dataset D containing multiple different fields; Step 2: Use the take-one method to divide the data set D into training set D train and the test set D val ; Step 3: Build an image classification model S, which includes a feature extractor G based on a convolutional neural network and a classifier C based on a linear model, using the training set D train The image classification model S is jointly trained and the training parameters are continuously adjusted to verify the image classification model S in D val Based on the performance on the test results, the result with the highest recognition accuracy is selected from multiple verification results, and its corresponding {G,C} weight parameter w is saved. g and w c ; Step 4: Build the mirror network M: Construct the feature extractor G' and classifier C' to make them have the same network structure as the feature extractor G and classifier C respectively, and add the feature reconstructor H in between, and then use the weight parameters (w g and w c ) Initialize G' and C' respectively; Step 5: Construct the loss function LOSS of the mirror network M; Step 6: For the data set D in step 2 train Preprocessing: D train The data subsets contained in the generated several combinations, and the optional data subset combination {D p , D q }, through sample alignment, so that D p and D q The number of samples in each category is equal; Step 7: Substitute the processed data subsets in step 6 into D' p and D' q Used to train the mirror network M = {G', H, C'}; Step 8: Repeat steps 6 and 7 until the number of iterations E is reached. s2 , obtain the model M within no more than the number of iterations E s2 The weights and training parameters corresponding to the result with the highest internal recognition accuracy; Step 9: Adjust the model training parameters by controlling the variable method, repeat steps 6, 7, and 8, and find the weight parameters corresponding to the result with the highest recognition accuracy of model M.

2. The cross-domain image classification method according to claim 1, characterized in that: In step 1, the data set D={D1, D2, ...D x ,…,D n }, where the data subsets D1, D2, …D x ,…,D n They come from different collection environments, the target categories contained in the data subsets are the same, there are distribution differences between the data subsets, and the intersection of any two data subsets is empty.

3. The cross-domain image classification method according to claim 1, characterized in that: In step 2, the training set D train ={D1,D2,…,D n }, validation set D val ={D x }, D = D train ∪D val , 4. The cross-domain image classification method according to claim 1, characterized in that: The step 3 comprises the following steps: Step 3.1: Set the training set D train Mixed scrambled for D mix , and for D mix The sample data in is randomly cropped and output at a fixed size, and then randomly flipped horizontally with a probability of 50% to obtain the processed sample number; Step 3.2: Use a convolutional neural network to construct a feature extractor G, which is mathematically expressed as a feature extraction function G(·). Select a CNN-based image recognition model ResNet-50 as the feature extractor G and assume that it is part of the ideal function. Step 3.3, add classifier C after G to form an image classification model; Step 3.4: Initialize the feature extractor G and classifier C using the Xavier method; Step 3.5: Use the supervised learning method to use the data processed in step 3.1 for the joint training of the feature extractor G and the classifier C, using the cross entropy loss function as a constraint.

5. The cross-domain image classification method according to claim 1, characterized in that: The step 4 comprises the following steps: Step 4.1, build the mirror network M: construct the feature extractor G', make it the same as the feature extractor G network structure and the classifier C', make it the same as the feature extractor G and classifier C network structure respectively, and then use the weight parameters of the feature extractor G and classifier C in step 3 to initialize G' and C' respectively; Step 4.2, construct feature extractor G': The final convolutional layer of the CNN-based feature extractor can be interpreted as similar object features, so the output of the final convolutional layer can be regarded as object features. Further object feature representation is obtained through the feature reconstructor H. ResNet-50 is selected as the feature extractor G', and the final object feature representation is f = R(x i )=H(G'(x i )); Step 4.3, construct feature reconstructor H: add feature reconstructor H after feature representation G to reconstruct features. In form, the feature representation is dimensioned up and then downgraded to achieve the purpose of feature reconstruction. Use 3-layer MLP to construct function H(·), use ReLU as activation function, place normalization (BatchNormalization) layer between linear layer and ReLU, normalize the data to (0, 1), accelerate model convergence speed, and improve model effect; Step 4.4, set constraints: feature reconstructor H produces 4 outputs h u ,h u' ,h v ,h v' , the output results of each two reconstruction heads use the loss function 1. To avoid the situation where the difference is too large and the learning is difficult, cross-domain and enhancement will not appear in the same comparison pair at the same time. Finally, 4 comparison pairs are constructed, (h p′ ,h q ),(h p, h q′ ),(h p ,h p′ )(h q ,h q′ ), as shown in Formula 2; Among them, B represents the size of batch size, MSE ij Indicates b p The i-th sample in b q is the mean square error between the jth samples in , λ is the weight parameter in the loss function, τ is the temperature coefficient, ∈ is a small constant used to prevent division by zero. Step 4.5, set up classifier C': use a 1-layer fully connected network as the classifier, and set constraints to satisfy: Among them, l h is the cross entropy loss function.

6. The cross-domain image classification method according to claim 1, characterized in that: The step 5 includes the loss function LOSS as: LOSS=w1L ctr +w2L mse +w3L cls +γL2 norm (H) (5) Among them, w1, w2, w3, γ are weight parameters, L2 norm (H) represents the L2 regularization of the feature reconstruction model H (parameters).

7. The cross-domain image classification method according to claim 1, characterized in that: The sample alignment technology is used in step 6, which ensures that the sample data can be enhanced and processed in pairs when the number of cross-domain samples is unbalanced. The specific algorithm is as follows: Input: Datasets source domain pairs Dp and Dq; Output: Sample aligned source domain pair D′ p , D′ q ; Function sample_align : and The lengths are l p , l q if l p >l q : Towards Randomly add l p -l q From own samples if l p <l q : Towards Randomly add l q -l p From own samples return Function datasets_pairs(D p ,D q ): Initialization sequence D′ p , D′ q for l in L: #L is D p , D q Collection of tags From D p Select the sample with label , to generate a set From D q Select the sample with label , to generate a set Bundle Append to D′ p and D′ q back return D′ p ,D′ q 。 8. The cross-domain image classification method according to claim 1, characterized in that: The step 7 comprises the following steps: Step 7.1, the second stage of data enhancement, introduces three types of data enhancement methods: 1) crop: RandomResizedCrop; 2) flip: RandomHorizontalFlip; 3) transformation: ColorJitter, RandomGrayscale, GaussianBlur. This process includes two steps: pre-processing: processing all data; post-processing: processing only one branch; Step 7.2: Initialize the feature reconstructor H using the Xavier method; Step 7.3: Set the number of network layers, number of neurons, learning rate, number of network iterations and SGD parameters of H to verify the model M in D val The performance on the above example saves the weight parameters corresponding to the result with the highest recognition accuracy of model M.

Images