A cross-domain image classification method based on inter-domain contrast adversarial learning

Abstract

The application discloses a cross-domain image classification method based on inter-domain contrastive adversarial learning. The application comprises the following steps: 1) on the basis of a domain adversarial network, a mapping network H is used to map the features extracted by a backbone network to a high-dimensional hidden space; 2) a contrastive learning loss is used to replace the classification loss of a domain classifier in the hidden space, and contrastive learning is performed. Compared with the prior art, the application combines contrastive learning, reduces the mismatch phenomenon in the traditional domain-invariant representation learning method, effectively migrates the image classification neural network model trained on the labeled source domain image to the unlabeled target domain, and improves the effect of the model in classifying the target domain image after migration.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image classification, and particularly relates to application of contrast learning and adversarial learning to cross-domain image classification tasks. BACKGROUND

[0002] Unsupervised domain adaptation (UDA) has become an active research field for solving the generalization problem when the training data and test data distribution are not subject to the independent and identically distributed constraint. Based on the covariate shift assumption, early domain adaptation research focuses on invariant representation learning to minimize the domain distance, including from statistical matching to adversarial learning.

[0003] Image classification is a typical application scenario of unsupervised domain adaptation technology. Image datasets often come from multiple sources, such as different websites, digital cameras, etc. Unsupervised domain adaptation image classification assumes that source domain images are labeled, target domain images are not labeled, source domain and target domain data distributions are different, but the source domain and the target domain have the same label set. Previous adversarial learning-based domain-invariant representation learning methods only coarsely align the data distributions of the source domain and the target domain, without considering the internal structure of the same domain data. Therefore, in the field of image classification, how to consider the content structure of the same domain data in cross-domain adversarial learning is a technical problem to be solved. SUMMARY

[0004] The application aims to solve the problems in the prior art and provide a cross-domain image classification method based on inter-domain contrast adversarial learning.

[0005] Inspired by contrast learning, the application proposes an inter-domain contrast adversarial learning method for unsupervised domain adaptation. The advantage of this is to further align the data distribution at the fine-grained classification category level on the basis of coarse-grained source domain and target domain data distribution alignment.

[0006] The technical solution adopted by the application is as follows:

[0007] A cross-domain image classification method based on inter-domain contrast adversarial learning, the steps of which are as follows:

[0008] S1: Obtain a labeled source domain image dataset and an unlabeled target domain image dataset, wherein the source domain image dataset and the target domain image dataset have different distributions but contain the same set of image categories;

[0009] S2: constructing a cross-domain image classification model based on a Min-Max adversarial learning framework, including a feature extraction network F, an image classification network C and a feature mapping network H, so as to convert image data into a high-dimensional hidden space; in the hidden space, positive and negative sample pairs are constructed according to whether the samples come from the source domain or the target domain, and a domain contrast loss L cts is calculated; based on the Min-Max adversarial learning framework, a total loss target function is obtained by combining the domain contrast loss L cts and the classification loss L cls of the image classification network C; the cross-domain image classification model is jointly optimized;

[0010] S3: taking the feature extraction network F and the image classification network C obtained by the joint optimization in S2 as the final image classification model, and performing image classification prediction on the input target domain image.

[0011] As a preferred embodiment, the specific implementation steps of S2 are as follows:

[0012] S21: constructing a cross-domain image classification model based on a Min-Max adversarial learning framework; in the constructed cross-domain image classification model, image features are first extracted by a feature extraction network F, then the output of the feature extraction network F is predicted by an image classification network C to predict the category of the image, and the output of the image classification network C is mapped to a high-dimensional hidden space by a mapping network H;

[0013] S22: in the high-dimensional hidden space, image samples sampled from the same domain are taken as positive samples, and image samples sampled from different domains are taken as negative samples, and a domain contrast loss L cts is calculated; then based on the Min-Max adversarial learning framework, a total loss target function is obtained by combining the domain contrast loss L cts and the classification loss L cls ; based on the total loss target function, the cross-domain image classification model is optimized and solved on the source domain image dataset and the target domain image dataset, and the optimized feature extraction network F, image classification network C and feature mapping network H are obtained.

[0014] As a preferred embodiment, the feature extraction network F adopts a pre-trained ResNet-50 network.

[0015] As a preferred embodiment, the image classification network C adopts a multi-layer perceptron model and uses a ReLU activation function.

[0016] As a preferred embodiment, the feature mapping network H adopts a multi-layer perceptron model and uses a ReLU activation function.

[0017] As a preferred embodiment, in S22, the domain contrast loss L cts is calculated according to the following formula:

[0018]

[0019]

[0020] wherein: l i represents the loss value of the i th image sample; 1 [·] represents an indicator function, the value of the indicator function is 1 when the condition in [·] is true, otherwise the value is 0; d i represents an indicator function, the value of the indicator function is 1 when the condition in [·] is true, otherwise the value is 0; d j represents the domain label of the i th image sample and the j th image sample respectively; n s and n t respectively represent the data amount of a batch of data of the source domain image dataset and the target domain image dataset; sim(·,·) represents a cosine similarity calculation function, v i and v j respectively represent the output of the i th image sample and the j th image sample in the feature mapping network H respectively; τ is a temperature parameter.

[0021] As preferred, the formula of the total loss target function is as follows:

[0022]

[0023] wherein: L cls is the classification loss of the image classification network C on the source domain, λ is a weight hyperparameter of the balanced domain contrast loss L cts and the classification loss L cls .

[0024] As preferred, the classification loss L cls adopts a cross-entropy loss.

[0025] Compared with the prior art, the present application has the following beneficial effects:

[0026] Compared with the existing domain classification loss, the domain contrast loss proposed by the present application realizes additional domain clustering, that is, gathering instances with the same domain. At the same time, the present application not only realizes the alignment of the "domain" level that can be achieved by the traditional domain adversarial training method, but also retains the internal structure within each domain, realizing the alignment of the "category" level between domains while forming a good clustering of different categories within each domain, which effectively improves the classification accuracy of the model when migrating to the target dataset. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a step schematic diagram of a cross-domain image classification method based on inter-domain contrast adversarial learning;

[0028] Figure 2A training flowchart of the cross-domain image classification model of the present application. DETAILED DESCRIPTION

[0029] The present application will be further described and explained with reference to the accompanying drawings and specific embodiments.

[0030] As shown in Figure 1 , in a preferred embodiment of the present application, a cross-domain image classification method based on inter-domain contrastive adversarial learning is provided, and the steps are shown as S1-S3:

[0031] S1: Obtain a labeled source domain image dataset and an unlabeled target domain image dataset, wherein the source domain image dataset and the target domain image dataset have different distributions but contain the same set of image classes.

[0032] S2: Construct a cross-domain image classification model based on a Min-Max adversarial learning framework, including a feature extraction network F, an image classification network C, and a feature mapping network H, so as to convert image data into a high-dimensional hidden space; in the hidden space, positive and negative sample pairs are constructed according to whether the samples come from the source domain or the target domain, and a domain contrast loss L cts is calculated; based on the Min-Max adversarial learning framework, the domain contrast loss L cts and the classification loss L cls of the image classification network C are combined to jointly optimize the cross-domain image classification model.

[0033] In this embodiment, the specific implementation steps of S2 are as follows:

[0034] S21: Construct a cross-domain image classification model based on a Min-Max adversarial learning framework; in the constructed cross-domain image classification model, first extract image features through a feature extraction network F, then predict the class of the image by outputting the feature extraction network F through an image classification network C, and finally map the output of the image classification network C to a high-dimensional hidden space through a mapping network H, and subsequent contrastive learning will be performed in this hidden space.

[0035] In this embodiment, the feature extraction network F adopts a pre-trained ResNet-50 network.

[0036] In this embodiment, the image classification network C adopts a multi-layer perceptron model and uses a ReLU activation function.

[0037] In this embodiment, the feature mapping network H adopts a multi-layer perceptron model and uses a ReLU activation function.

[0038] S22: In the high-dimensional hidden space, image samples sampled from the same domain are taken as positive sample pairs, image samples sampled from different domains are taken as negative sample pairs, and domain contrast loss L is calculated cts ; then, based on a Min-Max adversarial learning framework, domain contrast loss L cts and classification loss L cls are combined to obtain a total loss objective function, and the cross-domain image classification model is optimized and solved based on the total loss objective function on the source domain image dataset and the target domain image dataset to obtain an optimized feature extraction network F, image classification network C, and feature mapping network H.

[0039] In the embodiment, in S22 described above, domain contrast loss L cts is calculated according to the following formula:

[0040]

[0041]

[0042] wherein: l i represents the loss value of the i th image sample; 1 [·] ∈ {0, 1} represents an indicator function, and the value of the indicator function is 1 when the condition in [·] is true, and otherwise the value is 0; d i ∈ {0, 1} and d j ∈ {0, 1} represent the domain labels of the i th image sample and the j th image sample, respectively; n s and n t are the data amounts of one batch of data of the source domain image dataset and the target domain image dataset, respectively; sim(·, ·) represents a cosine similarity calculation function, v i and v j represent the outputs of the i th image sample and the j th image sample in the feature mapping network H, respectively; and τ is a temperature parameter.

[0043] In the embodiment, the formula of the total loss objective function is represented as follows:

[0044]

[0045] wherein: L cls is the classification loss of the image classification network C on the source domain, and in the embodiment, the classification loss L cls adopts a cross-entropy loss; λ is a weight hyperparameter of balancing domain contrast loss L cts and classification loss L cls .

[0046] S3: Taking the feature extraction network F and the image classification network C obtained by joint optimization in S2 as the final image classification model, the target domain image input is subjected to image classification prediction.

[0047] The cross-domain image classification method based on inter-domain contrastive adversarial learning shown in S1-S3 in the above embodiments will be applied to a specific data set below to demonstrate its technical effects.

[0048] Embodiments

[0049] This embodiment uses Office-31 and other image classification data sets to evaluate the performance of the image classification model (named DCAN) trained by the cross-domain image classification method based on inter-domain contrastive adversarial learning shown in S1-S3 in the above embodiments on this task. Office-31 is a general data set for evaluating unsupervised domain adaptation methods, which consists of 4110 pictures of 31 office environment categories, which come from three different domains: Amazon website (A), digital SLR camera (D), and network camera (W). The data set is unbalanced between different domains, with 2817 pictures in the A domain, 498 pictures in the D domain, and 795 pictures in the W domain.

[0050] As shown in Figure 1 , this embodiment uses all labeled source domain data and all unlabeled target domain samples according to the processes described in S1-S3 above, and compares the classification accuracy. The cross-domain image classification model uses ResNet-50 pre-trained on ImageNet as the feature extraction network, the image classification network uses a three-layer MLP network, the feature mapping network also uses a three-layer MLP network with a hidden layer dimension of 1024, and the activation function used is ReLU. Both MLP networks are trained from scratch. As shown in Figure 2 , the training process of the cross-domain image classification model of the present application, since the classification loss needs to be minimized in the objective function, and the domain contrastive loss needs to be maximized, therefore, referring to the practice in the domain adaptive neural network DANN, a gradient reversal layer (GRL) is added between the image classification network and the feature mapping network, and the output of the classification network is reversed after the gradient is reversed. The feature mapping is carried out in order to realize the model training. In this embodiment, the weight hyperparameter λ of the balanced domain contrastive loss L cts and the classification loss L cls in the objective function is 1.

[0051] The classification accuracy experimental results of the final image classification model DCAN and other contrastive models on the data set are shown in Table 1.

[0052] Table 1 Experimental results

[0053]

[0054] The above comparative model is specifically referred to in the following prior art documents:

[0055] 1. ResNet-50

[0056] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). 770-778.

[0057] 2. DANN

[0058] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Laviolette, Mario Marchand, and Victor S. Lempitsky. 2015. Domain-Adversarial Training of Neural Networks. JMLR (2015).

[0059] 3. CDAN

[0060] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. 2018. Conditional Adversarial Domain Adaptation. In Proceedings of Neural Information Processing Systems (NeurIPS), Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 1647-1657.

[0061] 4. MDD

[0062] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael I. Jordan. 2019. Bridging Theory and Algorithm for Domain Adaptation. In Proceedings of the International Conference on Machine Learning (Proceedings of Machine Learning Research), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). 7404-7413.

[0063] The ablation experiment is also performed in the embodiment, and the ablation experiment result is shown in Table 2, which shows the influence of the number of layers, the output dimension and the temperature parameter τ of the mapping network H on the model performance.

[0064] Table 2 Ablation experiment result

[0065]

[0066] In Table 2, the first table (the first two rows) is the performance of the final image classification model when the MLP network as the feature mapping network H takes different numbers of layers (Depth); the second table (the middle two rows) is the performance of the final image classification model when the output dimension (Output dimension) of the feature mapping network H takes different dimensions; and the third table (the last two rows) is the performance of the final image classification model when the temperature parameter τ in the loss takes different values. cts In Table 2, the first table (the first two rows) is the performance of the final image classification model when the MLP network as the feature mapping network H takes different numbers of layers (Depth); the second table (the middle two rows) is the performance of the final image classification model when the output dimension (Output dimension) of the feature mapping network H takes different dimensions; and the third table (the last two rows) is the performance of the final image classification model when the temperature parameter τ in the loss takes different values.

[0067] In summary, the present application combines contrastive learning, reduces the mismatch phenomenon in the traditional domain-invariant representation learning method, and effectively migrates the image classification neural network model trained on the labeled source domain image to the unlabeled target domain, thereby improving the effect of the model in classifying the target domain image after migration.

[0068] The above-described embodiment is only a preferred scheme of the present application, and is not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical scheme obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.

Claims

1. A cross-domain image classification method based on inter-domain contrastive adversarial learning, characterized in that, The steps are as follows: S1: Obtain a labeled source domain image dataset and an unlabeled target domain image dataset, wherein the source domain image dataset and the target domain image dataset have different distributions, but contain the same set of image categories; S2: A cross-domain image classification model built on the Min-Max adversarial learning framework, including a feature extraction network. An image classification network and a feature mapping network Image classification network The output is generated by the feature mapping network. Mapping to a high-dimensional latent space; constructing positive and negative sample pairs in this latent space based on whether the sample comes from the source or target domain, and calculating the domain contrast loss. ; Based on the Min-Max adversarial learning framework, combined with domain contrastive loss Image classification networks Classification loss The cross-domain image classification model is jointly optimized; S3: Feature extraction network obtained through joint optimization in S2 Image classification networks As the final image classification model, it performs image classification prediction on the input target domain image.

2. The cross-domain image classification method based on inter-domain contrastive adversarial learning as described in claim 1, characterized in that, The specific implementation steps of S2 are as follows: S21: Construct a cross-domain image classification model based on the Min-Max adversarial learning framework; in the constructed cross-domain image classification model, a feature extraction network is first used. Extract image features, and then use the feature extraction network. The output is processed by an image classification network. Image classification network for predicting image categories The output is then processed by a mapping network. Mapped to a higher-dimensional latent space; S22: In the high-dimensional latent space, image samples from the same domain are used as positive sample pairs, and image samples from different domains are used as negative sample pairs. The domain contrast loss is then calculated. Then, based on the Min-Max adversarial learning framework, combined with domain contrastive loss... and classification loss The total loss objective function is obtained. Based on this total loss objective function, the cross-domain image classification model is optimized and solved on both the source and target domain image datasets to obtain the optimized feature extraction network. Image classification network and feature mapping networks .

3. The cross-domain image classification method based on inter-domain contrastive adversarial learning as described in claim 1, characterized in that, The feature extraction network A pre-trained ResNet-50 network is used.

4. The cross-domain image classification method based on inter-domain contrastive adversarial learning as described in claim 1, characterized in that, The image classification network A multilayer perceptron model is used, and the ReLU activation function is employed.

5. The cross-domain image classification method based on inter-domain contrastive adversarial learning as described in claim 1, characterized in that, The feature mapping network A multilayer perceptron model is used, and the ReLU activation function is employed.

6. The cross-domain image classification method based on inter-domain contrastive adversarial learning as described in claim 2, characterized in that, In S22, the domain contrast loss The calculation formula is as follows: in: This represents the loss value for the i-th image sample; Indicates an indicator function, when The value of the indicator function is 1 when the condition in the condition is true, and 0 otherwise. and Let represent the domain labels of the i-th image sample and the j-th image sample, respectively; and These represent the data volume of a batch of the source domain image dataset and the target domain image dataset, respectively. This represents the function for calculating cosine similarity. These represent the i-th and j-th image samples respectively in the feature mapping network. The output in; It is a temperature parameter.

7. The cross-domain image classification method based on inter-domain contrastive adversarial learning as described in claim 2, characterized in that, The formula for the total loss objective function is expressed as follows: in: Image classification network Classification loss in the source domain, For the equalization domain contrast loss and classification loss The weight hyperparameter.

8. The cross-domain image classification method based on inter-domain contrastive adversarial learning as described in claim 1, characterized in that, The classification loss Cross-entropy loss is used.

Images