A cross-domain image classification method based on few-shot unsupervised domain adaptation

Abstract

A cross-domain image classification method based on few-shot unsupervised domain adaptation includes obtaining a training feature data set; constructing a feature extractor and using a pre-trained model; building a model based on a sample selection strategy, adversarial domain adaptation and prototype learning; obtaining a total loss function, and iterating to optimize the model; and using the completed domain self-adaptive classification model to perform cross-domain image classification. The application uses a sample selection strategy to expand the source domain samples without additional unlabeled source domain data. In addition, by constructing a source domain class prototype, the target domain samples are directly aligned with the source domain prototype, and fine cross-domain feature alignment is achieved. The application combines local class prototype alignment and global adversarial alignment, not only enhances the model's ability in feature alignment, but also verifies its excellent classification performance on public data sets, especially in the case of extremely limited labeled source domain samples, which shows significant robustness.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of cross-domain image classification, and particularly relates to a cross-domain image classification method based on few-shot unsupervised domain adaptation. BACKGROUND

[0002] Domain adaptation has shown great potential in solving the distribution shift problem between source and target domains. In particular, in the deep learning framework, by aligning the feature distributions between different domains, domain adaptation significantly improves the generalization performance of the model on the target domain. Among them, unsupervised domain adaptation has become a key direction in cross-domain problem research because it does not require labeled data in the target domain. Traditional unsupervised domain adaptation methods mainly complete cross-domain transfer between labeled source domains and unlabeled target domains through implicit or explicit distribution alignment. However, these methods usually rely on rich labeled data in the source domain to enable the model to generalize to the unlabeled data in the target domain. However, in practical applications, the acquisition of large-scale labeled data often requires high costs, and the labeled data in the source domain may even be extremely scarce. In addition, classical unsupervised domain adaptation methods only focus on global distribution alignment, ignoring the local feature alignment between classes. This problem is particularly prominent in the unsupervised domain adaptation scenario where the source domain sample is limited, and the sparsity of the labeled data in the source domain further exacerbates the challenge of feature alignment.

[0003] To address this problem, researchers have proposed few-shot unsupervised domain adaptation, aiming to transfer the domain-invariant features learned from a source domain containing only a few labeled samples to a target domain with a large number of unlabeled samples. In addition, there are a large number of unlabeled samples in the source domain of few-shot unsupervised domain adaptation, which increases the difficulty of learning domain-invariant features. With the development of self-supervised learning, contrastive learning, and meta-learning methods, the solution to the few-shot unsupervised domain adaptation problem has made some progress. However, existing few-shot unsupervised domain adaptation methods all rely on unlabeled samples in the source domain, and do not concern the correlation between samples and classes, which limits the solution to cross-domain distribution differences.

[0004] The technical differences compared with the prior art are as follows:

[0005] Comparison with the technology of patent CN113298189B "A cross-domain image classification method based on unsupervised domain adaptation"

[0006] The patent CN113298189B improves the accuracy and model generalization of image cross-domain classification by constructing source domain and target domain image sample data, normalization processing, using domain adaptive network training and iterative updating parameters, and realizes the goal of effective transfer learning between different distribution data sets, but this invention needs to use a large amount of labeled source domain data for training. The present application is a method for learning discriminative knowledge and domain invariant features from limited labeled source domain samples for insufficient source domain labeled samples. Effective classification is realized on the target domain. Specifically, the present application uses a composite model based on sample selection strategy, adversarial domain adaptation and prototype learning to perform cross-domain image classification. Among them, the sample selection strategy aims to expand the source domain sample set without relying on additional unlabeled source domain data. In addition, the present application directly aligns the target domain samples with the source domain prototypes by constructing the source domain class prototypes, realizing more fine cross-domain feature alignment. By combining local class prototype alignment and global adversarial alignment, the present application not only improves the model's ability in feature alignment, but also performs well in classification performance on public data sets, especially in the case of extremely limited labeled samples in the source domain, showing excellent robustness. These innovations make the present application have significant technical advantages and application potential in the field of cross-domain image classification. SUMMARY

[0007] To solve the above problems, the present application provides a cross-domain image classification method based on few-shot unsupervised domain adaptation, which learns discriminative features and domain invariant features only through a small amount of labeled samples in the source domain, in order to achieve good cross-domain classification effect in the unlabeled target domain.

[0008] To achieve the above purpose, the technical scheme adopted by the present application is:

[0009] A cross-domain image classification method based on few-shot unsupervised domain adaptation, characterized by the following specific steps:

[0010] S1: Obtain labeled source domain image sample data and unlabeled target domain image sample data as training data set;

[0011] S2: Construct a feature extractor ViT and use a pre-trained model;

[0012] S3: Build a model based on sample selection strategy, adversarial domain adaptation and prototype learning;

[0013] S4: Get the total loss function and iterate to optimize the model;

[0014] S5: Use the trained domain adaptive classification model to perform cross-domain image classification.

[0015] Further, the source domain and target domain sample data sets in step S1 are represented as Dsl and D tu where the corresponding samples are denoted as (x sl ,y sl ) and (x tu ) respectively.

[0016] Further, the feature extractor in step S2 is denoted as F, which takes the source domain samples and target domain samples in step S1 as input to extract features using a pre-trained ViT-B / 16 model on ImageNet-21K.

[0017] Further, the sample selection strategy in step S3 is based on the confidence and similarity of samples to augment source domain samples, making the model unnecessary to contain extra unlabeled source domain data, including the following sub-steps:

[0018] S31: Calculate the score matrix S = {s ij} in each training epoch, where s ij represents the quantified score of a pair of samples , defined as the Euclidean distance between the corresponding features:

[0019] where F(·) represents the sample features extracted by the feature extractor;

[0020] S32: Find the closest source sample for each target sample by finding the column index j1 of the minimum value s ij of each row:

[0021]

[0022] S33: The most similar labeled source sample of the unlabeled target sample can be found by the index j1, forming a high-similarity and interference-free source dataset, denoted as

[0023]

[0024] where D tu←sl represents the set of the most similar labeled source domains to each unlabeled sample in the target domain, both represent the most similar labeled source domain samples to each unlabeled sample in the target domain, but contains the exact index, denotes the corresponding source domain true label, N tu denotes the number of target domain samples;

[0025] S34: Set a fixed ratio of high-confidence samples and calculate the information entropy of each sample to exclude unreliable target samples and focus on learning high-confidence target samples. Specifically, the ratio of high-confidence target samples r is defined h , and select the one with the lowest entropy from the target sample The corresponding index of the target sample is expressed as:

[0026]

[0027] The above argΤοрK(·,·) means return The largest element index, represents the information entropy of each unlabeled target domain sample, and its calculation formula is:

[0028]

[0029] Among them, p i It is an array of probability distribution, containing the probability value of a sample in each category.

[0030] S35: High confidence target domain subset and the most similar source domain subset that matches it It can be expressed as:

[0031]

[0032]

[0033] Furthermore, the adversarial domain adaptation described in step S3 is used to learn domain-invariant features, which includes the following sub-steps:

[0034] S36: The adversarial domain adaptation network includes a feature extractor F based on ViT, which converts the input sample x from the input space X to the shared latent space F(x), a classifier C that predicts the extracted features and adds corresponding labels, converts the feature vector in the shared latent space to the label space Y, and a domain discriminator D that distinguishes whether the sample features come from the source domain (domain index 0) or the target domain (domain index 1). Through adversarial training, the feature extractor is optimized to deceive the domain discriminator and obtain the common features between the source domain and the target domain. At the same time, F and C are also trained to improve the ability to distinguish between categories;

[0035] S37: The loss in the process of training the feature extractor F and the classifier C by minimizing the classification error, and the source domain sample subset in substep S35 The feature extractor is passed in, so the classification loss expression is:

[0036]

[0037] where represents the standard cross-entropy loss function;

[0038] S38: In the training process of the adversarial network using the rich discriminative information provided by the classifier prediction, the conditional domain discriminator D is conditioned, and the domain discriminative loss is maximized by the domain discriminator D to distinguish between the source domain and the target domain, and the expression is:

[0039]

[0040] S39: The total loss expression of the adversarial domain adaptation is:

[0041]

[0042] where λ d is a trade-off hyperparameter.

[0043] Further, the prototype learning in step S3 is local alignment, which is used to assist global adversarial alignment to solve the cross-domain distribution difference problem, including the following sub-steps:

[0044] S310: Based on the source domain subset obtained in sub-step S35, the original features of the source domain subset are extracted by the feature extractor in step S2 to obtain F(x sl );

[0045] S311: Define a prototype layer for K-class classification task, and the prototype layer is represented as P = {p1, p2,..., p K}, and the mean of the sample features belonging to the class K is calculated as the class prototype of K class owned by the source domain subset The calculation expression is:

[0046]

[0047] where F(·) represents the sample feature vector extracted by the feature extractor, is the number of samples of class K;

[0048] S312: By minimizing the intra-class compactness loss, the compactness of the source domain sample features is enhanced, and the intra-class consistency of the source domain is promoted. The intra-class compactness loss is defined on the source domain samples by using distance measurement, and the expression is:

[0049]

[0050] S313: After the source domain prototype is constructed, the high-confidence sample features of the target domain are aligned, so as to realize cross-domain feature alignment, and the loss between the target domain sample and the source domain prototype is dynamically evaluated according to the target domain predicted class probability as a weight, which is used to measure the matching degree of the target sample and the class prototype, and the loss expression between the high-confidence target domain sample and the corresponding source domain prototype is:

[0051]

[0052] Wherein, represents the probability that the target sample belongs to the class K, and by minimizing the loss term, the feature representation of the target domain sample and the source domain class prototype can be directly aligned, the accurate classification of the target domain sample in the feature space is realized, and the uncertainty and noise interference brought by the pseudo label are reduced;

[0053] S314: A unified comprehensive prototype alignment loss is constructed by combining the intra-class compactness loss and the cross-domain prototype loss, so as to improve the separation and compactness of the class cluster distribution in the cross-domain task, and the expression is:

[0054] L proto =L cross +λ s L intra ,

[0055] Wherein λ p is a hyperparameter used to control the prototype loss.

[0056] Further, the total loss function described in step S4 is:

[0057] L=L adv +λ p L proto ,

[0058] Wherein λ p is a hyperparameter used to control the prototype loss.

[0059] The beneficial effects of the present application are:

[0060] The present application expands the rare source domain samples through the sample selection strategy, filters the high-quality source domain and target domain samples for domain-invariant feature learning, and combines local class prototype alignment and global adversarial alignment, so as to realize sufficient cross-domain feature alignment. Especially in the case that the labeled sample of the source domain is extremely limited, the present application exhibits excellent performance in the unlabeled target domain classification task. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a flowchart of the few-shot unsupervised domain adaptation method of the present application;

[0062] Figure 2 is a structural diagram of the global confrontation alignment part in the application;

[0063] Figure 3 is a structural diagram of the local prototype alignment part in the application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical scheme and advantages of the application more clear, the application will be described in detail below in combination with the drawings and specific embodiments.

[0065] Embodiment 1:

[0066] As shown in the accompanying drawings, Figure 1 a cross-domain image classification method based on few-shot unsupervised domain adaptation includes the following steps:

[0067] S1: Obtain labeled source domain image sample data and unlabeled target domain image sample data as training data sets;

[0068] S2: Construct a feature extractor ViT using a pre-trained model;

[0069] S3: Build a model based on a sample selection strategy, confrontation domain adaptation and prototype learning;

[0070] S4: Obtain the total loss function and iterate to optimize the model;

[0071] S5: Use the trained domain adaptive classification model to perform cross-domain image classification.

[0072] In the step S1, the source domain and target domain sample data sets are represented as D sl and D tu , respectively, wherein the corresponding samples are represented as (x sl , y sl ) and (x tu ), respectively.

[0073] In the step S2, the feature extractor is represented by F, and the ViT-B / 16 model pre-trained on ImageNet-21K is used to input the source domain sample and target domain sample in step S1 to extract features.

[0074] In the step S3, the sample selection strategy is based on the confidence and similarity of the samples, and is used to expand the source domain samples, so that the model does not need additional unlabeled source domain data, and includes the following sub-steps:

[0075] S31: Calculate the score matrix S={s ij} at each training period, wherein s ij represents the quantified score of a pair of samples , defined as the Euclidean distance between the corresponding features:

[0076] where F(·) represents the sample features extracted by the feature extractor.

[0077] S32: Find the closest source sample for each target sample by finding the minimum value s ij of each row and the column index j1:

[0078]

[0079] S33: The most similar labeled source samples of the unlabeled target samples can be found by the index j1, forming a high-similarity and interference-free source dataset, denoted as

[0080]

[0081] where D tu←sl represents the set of the most similar labeled source domains to each unlabeled sample in the target domain, all represent the most similar labeled source domain samples to each unlabeled sample in the target domain, but contains the exact index, represents the corresponding source domain true label, and N tu represents the number of target domain samples.

[0082] S34: Set a fixed proportion of high-confidence samples and calculate the information entropy of each sample to exclude unreliable target samples and focus on learning high-confidence target samples. Specifically, define the proportion of high-confidence target samples r h , and select the corresponding index of the target samples with the lowest entropy from the target samples, denoted as:

[0083]

[0084] The above argΤορK(·,·) represents returning the index of the element with the maximum value, represents the information entropy of each unlabeled target domain sample, and its calculation formula is:

[0085]

[0086] where p i is an array of probability distribution, containing the probability value of a sample in each class.

[0087] S35: High-confidence target domain subset and the most similar source domain subset matched which can be expressed as:

[0088]

[0089] In step S3, the adversarial domain adaptation is applied to learn the domain-invariant features, as shown in the following equation: Figure 2 which contains the following sub-steps:

[0090] S36: The adversarial domain adaptation network includes a ViT-based feature extractor F that converts the input sample x from the input space X to the shared latent space F(x), a classifier C that predicts and adds the corresponding label to the extracted features, which converts the feature vector in the shared latent space to the label space Y, and a domain discriminator D that distinguishes whether the sample features come from the source domain (with a domain index of 0) or the target domain (with a domain index of 1). The feature extractor is optimized to deceive the domain discriminator through adversarial training, obtaining the common features between the source domain and the target domain. At the same time, F and C are also trained to improve the inter-class discrimination ability.

[0091] S37: The loss in the process of training the feature extractor F and the classifier C is minimized by minimizing the classification error, and the source domain sample subset is passed into the feature extractor, so the classification loss expression is:

[0092]

[0093] where represents the standard cross-entropy loss function.

[0094] S38: During the training process of the adversarial network using the rich discriminative information provided by the classifier, the conditional domain discriminator D is conditioned, and the domain discrimination loss is maximized by the domain discriminator D to distinguish between the source domain and the target domain. Its expression is:

[0095]

[0096] S39: The total loss expression of the adversarial domain adaptation is:

[0097]

[0098] where λ d is a trade-off hyperparameter.

[0099] In step S3, the prototype learning is a local alignment to assist the global adversarial alignment in solving the cross-domain distribution difference problem, as shown in the following equation: Figure 3 which contains the following sub-steps:

[0100] S310: Based on the source domain subset The original features of the sample are F(x sl ) after passing through the feature extractor in step S2.

[0101] S311: Defining a prototype layer for the K-class classification task, the prototype layer is represented as P={p1, p2,..., p K} and calculating the mean value of the sample features belonging to the class K as the class prototype of the source domain subset with K classes The calculation expression is:

[0102]

[0103] Where F(·) represents the sample feature vector extracted by the feature extractor, is the number of samples of class K.

[0104] S312: By minimizing the intra-class compactness loss, the compactness of the source domain sample features is enhanced, and the intra-class consistency of the source domain is promoted. The intra-class compactness loss is defined on the source domain samples using distance measurement, and the expression is:

[0105]

[0106] S313: After constructing the source domain prototype, the high-confidence sample features of the target domain are aligned, thereby realizing cross-domain feature alignment. According to the predicted class probability of the target domain as a weight to dynamically evaluate the loss between the target domain sample and the source domain prototype, which is used to measure the matching degree of the target sample and each class prototype. The loss expression between the high-confidence target domain sample and its corresponding source domain prototype is:

[0107]

[0108] Where, represents the probability that the target sample belongs to class K. By minimizing this loss term, the feature representation of the target domain sample and the source domain class prototype can be directly aligned, realizing accurate classification of the target domain sample in the feature space, while reducing the uncertainty and noise interference brought by the pseudo label.

[0109] S314: Combining the intra-class compactness loss and the cross-domain prototype loss, a unified comprehensive prototype alignment loss is constructed, thereby improving the separation and compactness of the class cluster distribution in the cross-domain task, and the expression is:

[0110] L proto =L cross +λ s L intra ,

[0111] Where λ p is a hyperparameter used to control the prototype loss.

[0112] The total loss function in step S4 is:

[0113] L = L adv + λ p L proto ,

[0114] where λ p is a hyperparameter used to control the prototype loss.

[0115] In the specific implementation process, the batch size is set to 32, and the SGD optimizer with a momentum of 0.9 is used for 300 training cycles. The initial learning rate is set to 0.01, the weight decay is set to 0.0005 to prevent overfitting, and the Nesterov momentum is enabled to improve the convergence speed. In addition, we use the cosine annealing learning rate scheduling strategy to optimize the training process of the model. The strategy includes a warm-up phase and an annealing phase. The initial learning rate is maintained for the first 200 training cycles, and the learning rate is gradually reduced according to the cosine function for the last 100 cycles.

[0116] Experiments are performed using the Office-31 dataset, which consists of three different domains: Amazon (A), Webcam (W), and DSLR (D), each representing unique image capture conditions and data distribution. The dataset contains 31 classes, a total of 4652 images, which mainly involve common office items such as books, keyboards, and scissors. Among them, the images of Amazon come from online stores and usually have a clean white background; the images of Webcam are low-resolution photos; and the images of DSLR are high-resolution photos. There are significant distribution differences between these domains. In the experiment of few-shot unsupervised domain adaptation, six typical transfer tasks are set: A→W, A→D, W→A, W→D, D→A, and D→W, and the source domain is set to 3 labeled samples per class.

[0117] Further, the method of the present application is compared with other methods: CDAN, CDS, PCS, PAC, C-VisDiT, EUDA, among which the first five methods also include unlabeled samples in the source domain, and EUDA and the present application are set the same, with only 3 labeled samples per class in the source domain. The comparison results are shown in Table 1 below.

[0118] Table 1 Comparison of adaptation accuracy on the Office-31 dataset with 3 labeled samples per class in the source domain.

[0119]

[0120] As can be seen from Table 1, the classification effect of the present application is better than that of other comparison methods, especially when the source domain has only limited labeled samples. The effectiveness of the present application can be verified through the above experiments.

[0121] The above descriptions are only the preferred embodiments of the present application, not intended to limit the present application in any manner. Any modification or equivalent variation made according to the technical essence of the present application shall still fall within the scope of the present application.

Claims

1. A cross-domain image classification method based on few-shot unsupervised domain adaptation, characterized by: The specific steps are as follows: S1: Obtain labeled source domain image sample data and unlabeled target domain image sample data as training dataset; S2: Construct a feature extractor and use the pre-trained model to input the source domain samples and target domain samples in step S1 to extract features; S3: Build a model based on sample selection strategy, adversarial domain adaptation and prototype learning; The sample selection strategy is based on the confidence and similarity of the samples and is used to expand the source domain samples so that the model does not require additional unlabeled source domain data. It includes the following sub-steps: S31: Calculate the score matrix at each training epoch ,in Represents a pair of samples The quantitative score of is defined as the Euclidean distance between corresponding features: ,in Represents the sample features extracted by the feature extractor; S32: By finding the minimum value of each row Column index of , find the closest source sample for each target sample: ， S33: By index The most similar labeled source sample of the unlabeled target sample can be found to form a source dataset with high similarity and no interference, which is expressed as : ， in represents the set of labeled source domains that are most similar to each unlabeled sample in the target domain, , Both represent the labeled source domain sample that is most similar to each unlabeled sample in the target domain, but Contains the exact index, , It represents the corresponding source domain true label, It represents the number of target domain samples; S34: Set a fixed ratio of high-confidence samples and calculate the information entropy of each sample to exclude unreliable target samples and focus on learning high-confidence target samples. Specifically, the ratio of high-confidence target samples is defined. , and select the one with the lowest entropy from the target sample The corresponding index of the target sample is expressed as: ， above Indicates return The largest element index, represents the information entropy of each unlabeled target domain sample, and its calculation formula is: ， in, It is an array of probability distribution, containing the probability value of a sample in each category; S35: High confidence target domain subset and the most similar source domain subset that matches it It can be expressed as: ； ; The adversarial domain adaptation described above is used to learn domain-invariant features and includes the following sub-steps: S36: Adversarial Domain Adaptation Network including ViT-based Feature Extractor , the input sample From the input space Transforming to a shared latent space , a classifier that predicts the extracted features and adds corresponding labels , transforming the feature vector in the shared latent space into the label space , a domain discriminator that distinguishes whether the sample features come from the source domain with a domain index of 0 or the target domain with a domain index of 1 , through adversarial training, the feature extractor is optimized to deceive the domain discriminator and obtain the common features between the source domain and the target domain. At the same time, and It is also trained to improve the ability to distinguish between categories; S37: Train the feature extractor by minimizing the classification error and classifier The loss in the process of The feature extractor is passed in, so the classification loss expression is: ， in represents the standard cross entropy loss function; S38: In the process of training the adversarial network using the discriminant information provided by the classifier prediction, the conditional domain discriminator is conditioned, and the domain discrimination loss is obtained through the domain discriminator To maximize the discrimination between the source domain and the target domain, its expression is: ， S39: The total loss expression of adversarial domain adaptation is: ， in is a trade-off hyperparameter; The prototype learning mentioned above is a local alignment, which is used to assist global adversarial alignment to solve the problem of cross-domain distribution differences. It includes the following sub-steps: S310: Source domain subset obtained based on sub-step S35 , the original features after the feature extractor in step S2 are ; S311: Define a prototype layer for For the classification task of the class, the prototype layer is represented as , calculate the category The mean of the sample features of the source domain subset has Class prototype , its calculation expression is: ， in Represents the sample feature vector extracted by the feature extractor, is a category The number of samples; S312: By minimizing the intra-category compactness loss, the compactness of the source domain sample features is enhanced, and the consistency within the source domain class is promoted. The intra-category compactness loss is defined on the source domain samples using distance measurement. The expression is: ； S313: After constructing the source domain prototype, the high-confidence sample features of the target domain are aligned to achieve cross-domain feature alignment. The loss between the target domain sample and the source domain prototype is dynamically evaluated based on the target domain predicted category probability as a weight to measure the degree of match between the target sample and each category prototype. The loss expression between a high-confidence target domain sample and its corresponding source domain prototype is: ， in, Indicates that the target sample belongs to the category By minimizing this loss term, the feature representation of the target domain sample and the source domain category prototype can be directly aligned; S314: Combining the intra-category compactness loss and the cross-domain prototype loss, a unified comprehensive prototype alignment loss is constructed, which is expressed as: ， in is a hyperparameter used to control prototype loss; S4: Get the total loss function and iterate to optimize the model; The total loss function described is: ， in is a hyperparameter used to control prototype loss; S5: Use the trained domain adaptation classification model to perform cross-domain image classification.

2. The cross-domain image classification method based on few-shot unsupervised domain adaptation according to claim 1, characterized in that: The source domain and target domain sample datasets described in step S1 are respectively represented as and , where the corresponding samples are expressed as and .

3. The cross-domain image classification method based on few-sample unsupervised domain adaptation according to claim 1, characterized in that: The feature extractor described in step S2 is used To represent, the ViT-B / 16 model pre-trained on ImageNet-21K is used as the pre-training model.

Images