Cross-domain image classification method based on few-sample unsupervised domain adaptation

Abstract

The invention discloses a cross-domain image classification method based on few-sample unsupervised domain adaptation. The method comprises the following steps: acquiring a training feature data set; constructing a feature extractor, and using a pre-training model; building a model based on a sample selection strategy, adversarial domain adaptation and prototype learning; a total loss function is obtained, and iteration is carried out to optimize the model; and performing cross-domain image classification by using the trained domain adaptive classification model. According to the invention, the source domain sample is expanded by using the sample selection strategy without additional unmarked source domain data. Besides, by constructing a source domain category prototype, a target domain sample is directly aligned with the source domain prototype, and refined cross-domain feature alignment is realized. According to the method, local category prototype alignment and global adversarial alignment are combined, the capability of the model in the aspect of feature alignment is enhanced, the excellent classification performance of the model is verified on a public data set, and particularly, remarkable robustness is shown under the condition that source domain labeled samples are extremely limited.

Description

Technical Field

[0001] The present invention belongs to the field of cross-domain image classification, and specifically is a cross-domain image classification method based on few-sample unsupervised domain adaptation. Background Art

[0002] Domain adaptation has shown great potential in solving the problem of distribution shift between source and target domains. In particular, in deep learning frameworks, domain adaptation significantly improves the generalization performance of the model in the target domain by aligning the feature distributions between different domains. Among them, unsupervised domain adaptation has become a key direction for cross-domain problem research because it does not require labeled data in the target domain. Traditional unsupervised domain adaptation methods mainly complete cross-domain migration through implicit or explicit distribution alignment between labeled source domains and unlabeled target domains. However, these methods usually rely on abundant labeled data in the source domain to enable the model to generalize to unlabeled data in the target domain. But in practical applications, the acquisition of large-scale labeled data often requires high costs, and source domain labeled data may even be extremely scarce. In addition, classic unsupervised domain adaptation methods only focus on global distribution alignment, ignoring local feature alignment between categories between samples. This problem is particularly prominent in unsupervised domain adaptation scenarios with limited source domain samples, and the sparsity of source domain labeled data further exacerbates the challenge of feature alignment.

[0003] To address this problem, researchers proposed few-shot unsupervised domain adaptation, which aims to transfer domain-invariant features learned from a source domain with only a few labeled samples to a target domain with a large number of unlabeled samples. In addition, there are a large number of unlabeled samples in the source domain of few-shot unsupervised domain adaptation, which increases the difficulty of learning domain-invariant features. With the development of methods such as self-supervised learning, contrastive learning, and meta-learning, some progress has been made in solving the problem of few-shot unsupervised domain adaptation. However, existing few-shot unsupervised domain adaptation methods all have the problem of relying on unlabeled samples in the source domain, and do not care about the association between samples and between categories, which brings limitations to solving cross-domain distribution differences.

[0004] Compared with the existing technology, the technical differences are as follows:

[0005] Technical comparison with patent CN113298189B "A cross-domain image classification method based on unsupervised domain adaptation"

[0006] Patent CN113298189B improves the accuracy and model generalization of cross-domain image classification by constructing source domain and target domain image sample data, normalizing, using domain adaptive network training and iteratively updating parameters, and achieves the goal of effective transfer learning between datasets with different distributions. However, this invention requires a large amount of labeled source domain data for training. The present invention proposes a method for learning discriminative knowledge and domain invariant features from a source domain with limited labeled samples to achieve effective classification in the target domain in response to the shortage of labeled samples in the source domain. Specifically, the present invention adopts a composite model based on sample selection strategy, adversarial domain adaptation and prototype learning to perform cross-domain image classification. Among them, the sample selection strategy aims to expand the source domain sample set without relying on additional unlabeled source domain data. In addition, the present invention directly aligns the target domain sample with the source domain prototype by constructing a source domain category prototype, thereby achieving more refined cross-domain feature alignment. By combining local category prototype alignment and global adversarial alignment, the present invention not only improves the model's ability in feature alignment, but also performs well in classification performance on public datasets, especially when the source domain labeled samples are extremely limited, showing excellent robustness. These innovations give the present invention significant technical advantages and application potential in the field of cross-domain image classification. Summary of the invention

[0007] To address the above problems, the present invention proposes a cross-domain image classification method based on few-sample unsupervised domain adaptation, which learns discriminative features and domain-invariant features through only a small number of labeled samples in the source domain, in the hope of achieving good cross-domain classification effect in the unlabeled target domain.

[0008] To achieve the above object, the technical solution adopted by the present invention is:

[0009] A cross-domain image classification method based on few-sample unsupervised domain adaptation, characterized in that the specific steps are as follows:

[0010] S1: Obtain labeled source domain image sample data and unlabeled target domain image sample data as training data sets;

[0011] S2: Construct feature extractor ViT and use the pre-trained model;

[0012] S3: Build a model based on sample selection strategy, adversarial domain adaptation and prototype learning;

[0013] S4: Get the total loss function and iterate to optimize the model;

[0014] S5: Use the trained domain adaptation classification model to perform cross-domain image classification.

[0015] Furthermore, the source domain and target domain sample datasets described in step S1 are represented as Dsl and D tu , where the corresponding samples are represented as (x sl ,y sl ) and (x tu ).

[0016] Furthermore, the feature extractor described in step S2 is represented by F, and uses the ViT-B / 16 model pre-trained on ImageNet-21K, and inputs the source domain samples and the target domain samples in step S1 to extract features.

[0017] Furthermore, the sample selection strategy described in step S3 is based on the confidence and similarity of the samples, and is used to expand the source domain samples so that the model does not need additional unlabeled source domain data, and includes the following sub-steps:

[0018] S31: Calculate the score matrix S = {s ij}, where s ij Represents a pair of samples The quantitative score of is defined as the Euclidean distance between corresponding features:

[0019] Where F(·) represents the sample features extracted by the feature extractor;

[0020] S32: By finding the minimum value s of each row ij , find the closest source sample for each target sample:

[0021]

[0022] S33: The most similar labeled source sample of the unlabeled target sample can be found through index j1, forming a source data set with high similarity and no interference, which is expressed as

[0023]

[0024] Where D tu←sl represents the set of labeled source domains that are most similar to each unlabeled sample in the target domain, Both represent the labeled source domain samples that are most similar to each unlabeled sample in the target domain, but Contains the exact index, It represents the corresponding source domain true label, N tu It represents the number of samples in the target domain;

[0025] S34: Set a fixed ratio of high-confidence samples and calculate the information entropy of each sample to exclude unreliable target samples and focus on learning high-confidence target samples. Specifically, the ratio of high-confidence target samples r is defined. h , and select the one with the lowest entropy from the target sample The corresponding index of the target samples is expressed as:

[0026]

[0027] The above argΤοрK(·,·) means return The largest element index, represents the information entropy of each unlabeled target domain sample, and its calculation formula is:

[0028]

[0029] Among them, p i It is an array of probability distribution, containing the probability value of a sample in each category.

[0030] S35: High confidence target domain subset and the most similar source domain subset that matches it It can be expressed as:

[0031]

[0032]

[0033] Furthermore, the adversarial domain adaptation described in step S3 is used to learn domain-invariant features, and comprises the following sub-steps:

[0034] S36: The adversarial domain adaptation network includes a feature extractor F based on ViT, which converts the input sample x from the input space X to the shared latent space F(x), a classifier C that predicts the extracted features and adds corresponding labels, converts the feature vector in the shared latent space to the label space Y, and a domain discriminator D that distinguishes whether the sample features come from the source domain whose domain index is 0 or the target domain (domain index is 1). The feature extractor is optimized through adversarial training to deceive the domain discriminator and obtain the common features between the source domain and the target domain. At the same time, F and C are also trained to improve the ability to distinguish between categories;

[0035] S37: The loss in the process of training the feature extractor F and the classifier C by minimizing the classification error, and the source domain sample subset in substep S35 The feature extractor is passed in, so the classification loss expression is:

[0036]

[0037] in represents the standard cross entropy loss function;

[0038] S38: In the process of training the adversarial network using the rich discriminant information provided by the classifier prediction, the conditional domain discriminator D is conditioned, and the domain discrimination loss is to maximize the distinction between the source domain and the target domain through the domain discriminator D, and its expression is:

[0039]

[0040] S39: The total loss expression of adversarial domain adaptation is:

[0041]

[0042] where λ d is a trade-off hyperparameter.

[0043] Furthermore, the prototype learning described in step S3 is a local alignment to assist the global adversarial alignment in solving the problem of cross-domain distribution differences, and includes the following sub-steps:

[0044] S310: Source domain subset obtained based on sub-step S35 The original feature after the feature extractor in step S2 is F(x sl );

[0045] S311: Define a prototype layer for the K-class classification task. The prototype layer is represented by P = {p1, p2, ..., p K}, calculate the mean of the sample features belonging to category K as the category prototype of the source domain subset with K categories Its calculation expression is:

[0046]

[0047] Where F(·) represents the sample feature vector extracted by the feature extractor, is the number of samples of category K;

[0048] S312: By minimizing the intra-category compactness loss, the compactness of the source domain sample features is enhanced, the consistency within the source domain class is promoted, and the intra-category compactness loss is defined on the source domain samples using distance measurement. The expression is:

[0049]

[0050] S313: After constructing the source domain prototype, align the high-confidence sample features of the target domain to achieve cross-domain feature alignment. Use the target domain predicted category probability as a weight to dynamically evaluate the loss between the target domain sample and the source domain prototype to measure the matching degree between the target sample and each category prototype. The loss expression between the high-confidence target domain sample and its corresponding source domain prototype is:

[0051]

[0052] in, Represents the probability that the target sample belongs to category K. By minimizing this loss term, the feature representation of the target domain sample and the source domain category prototype can be directly aligned to achieve accurate classification of the target domain sample in the feature space, while reducing the uncertainty and noise interference caused by pseudo labels;

[0053] S314: Combining the intra-category compactness loss and the cross-domain prototype loss, a unified comprehensive prototype alignment loss is constructed to improve the separation and compactness of cluster distribution in cross-domain tasks. Its expression is:

[0054] L proto =L cross +λ s L intra ,

[0055] where λ p is a hyperparameter used to control the prototype loss.

[0056] Furthermore, the total loss function described in step S4 is:

[0057] L=L adv +λ p L proto ,

[0058] where λ p is a hyperparameter used to control the prototype loss.

[0059] The beneficial effects of the present invention are:

[0060] The present invention expands the scarce source domain samples through a sample selection strategy, and screens high-quality source domain and target domain samples to learn domain-invariant features. It also combines local category prototype alignment with global adversarial alignment to achieve sufficient cross-domain feature alignment. Especially when the source domain labeled samples are extremely limited, the present invention demonstrates excellent performance in unlabeled target domain classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of the few-sample unsupervised domain adaptation method of the present invention;

[0062] Figure 2 It is a structural diagram of the global adversarial alignment part in the present invention;

[0063] Figure 3 It is a structural diagram of the local prototype alignment part in the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] Embodiment 1:

[0066] As attached Figure 1 As shown, a cross-domain image classification method based on few-sample unsupervised domain adaptation includes the following steps:

[0067] S1: Obtain labeled source domain image sample data and unlabeled target domain image sample data as training data sets;

[0068] S2: Construct feature extractor ViT, using pre-trained model;

[0069] S3: Build a model based on sample selection strategy, adversarial domain adaptation and prototype learning;

[0070] S4: Get the total loss function and iterate to optimize the model;

[0071] S5: Use the trained domain adaptation classification model to perform cross-domain image classification.

[0072] In step S1, the source domain and target domain sample datasets are represented as D sl and D tu , where the corresponding samples are represented as (x sl ,y sl ) and (x tu ).

[0073] In step S2, the feature extractor is represented by F, and the ViT-B / 16 model pre-trained on ImageNet-21K is used to input the source domain samples and the target domain samples in step S1 to extract features.

[0074] In step S3, the sample selection strategy is based on the confidence and similarity of the samples, and is used to expand the source domain samples so that the model does not need additional unlabeled source domain data, and includes the following sub-steps:

[0075] S31: Calculate the score matrix S = {s ij}, where s ij Represents a pair of samples The quantitative score of is defined as the Euclidean distance between corresponding features:

[0076] Where F(·) represents the sample features extracted by the feature extractor.

[0077] S32: By finding the minimum value s of each row ij , find the closest source sample for each target sample:

[0078]

[0079] S33: The most similar labeled source sample of the unlabeled target sample can be found through index j1, forming a source data set with high similarity and no interference, which is expressed as

[0080]

[0081] Where D tu←sl represents the set of labeled source domains that are most similar to each unlabeled sample in the target domain, Both represent the labeled source domain sample that is most similar to each unlabeled sample in the target domain, but Contains the exact index, It represents the corresponding source domain true label, N tu It represents the number of samples in the target domain.

[0082] S34: Set a fixed ratio of high-confidence samples and calculate the information entropy of each sample to exclude unreliable target samples and focus on learning high-confidence target samples. Specifically, the ratio of high-confidence target samples r is defined h , and select the one with the lowest entropy from the target sample The corresponding index of the target samples is expressed as:

[0083]

[0084] The above argΤοрK(·,·) means return The largest element index, represents the information entropy of each unlabeled target domain sample, and its calculation formula is:

[0085]

[0086] Among them, p i It is an array of probability distribution, containing the probability value of a sample in each category.

[0087] S35: High confidence target domain subset and the most similar source domain subset that matches it It can be expressed as:

[0088]

[0089] In step S3, adversarial domain adaptation is used to learn domain-invariant features, such as Figure 2 As shown, it includes the following sub-steps:

[0090] S36: The adversarial domain adaptation network includes a feature extractor F based on ViT, which converts the input sample x from the input space X to the shared latent space F(x), a classifier C that predicts the extracted features and adds corresponding labels, which converts the feature vector in the shared latent space to the label space Y, and a domain discriminator D that distinguishes whether the sample features come from the source domain (domain index is 0) or the target domain (domain index is 1). Through adversarial training, the feature extractor is optimized to deceive the domain discriminator and obtain the common features between the source domain and the target domain. At the same time, F and C are also trained to improve the ability to distinguish between categories.

[0091] S37: The loss in the process of training the feature extractor F and the classifier C by minimizing the classification error, and the source domain sample subset in substep S35 The feature extractor is passed in, so the classification loss expression is:

[0092]

[0093] in represents the standard cross entropy loss function.

[0094] S38: In the process of training the adversarial network using the rich discriminative information provided by the classifier prediction, the conditional domain discriminator D is conditioned, and the domain discriminant loss is to maximize the distinction between the source domain and the target domain through the domain discriminator D. Its expression is:

[0095]

[0096] S39: The total loss expression of adversarial domain adaptation is:

[0097]

[0098] where λ d is a trade-off hyperparameter.

[0099] In step S3, prototype learning is local alignment, which is used to assist global adversarial alignment to solve the problem of cross-domain distribution differences. Figure 3 As shown, it includes the following sub-steps:

[0100] S310: Source domain subset obtained based on sub-step S35 The original feature after the feature extractor in step S2 is F(x sl ).

[0101] S311: Define a prototype layer for the K-class classification task. The prototype layer is represented by P = {p1, p2, ..., p K}, calculate the mean of the sample features belonging to category K as the category prototype of the source domain subset with K categories Its calculation expression is:

[0102]

[0103] Where F(·) represents the sample feature vector extracted by the feature extractor, is the number of samples of category K.

[0104] S312: By minimizing the intra-category compactness loss, the compactness of the source domain sample features is enhanced and the intra-category consistency of the source domain is promoted. The intra-category compactness loss is defined on the source domain samples using distance measurement, and the expression is:

[0105]

[0106] S313: After constructing the source domain prototype, align the high-confidence sample features of the target domain to achieve cross-domain feature alignment. The loss between the target domain sample and the source domain prototype is dynamically evaluated based on the target domain predicted category probability as a weight to measure the matching degree between the target sample and each category prototype. The loss expression between the high-confidence target domain sample and its corresponding source domain prototype is:

[0107]

[0108] in, represents the probability that the target sample belongs to category K. By minimizing this loss term, the feature representation of the target domain sample and the source domain category prototype can be directly aligned to achieve accurate classification of the target domain sample in the feature space, while reducing the uncertainty and noise interference caused by pseudo labels.

[0109] S314: Combining the intra-category compactness loss and the cross-domain prototype loss, a unified comprehensive prototype alignment loss is constructed to improve the separation and compactness of cluster distribution in cross-domain tasks. Its expression is:

[0110] L proto =L cross +λ s L intra ,

[0111] where λ p is a hyperparameter used to control the prototype loss.

[0112] In step S4, the total loss function is:

[0113] L=L adv +λ p L proto ,

[0114] where λ p is a hyperparameter used to control the prototype loss.

[0115] In the specific implementation, the batch size was set to 32, and the SGD optimizer with a momentum of 0.9 was used for 300 training cycles. The initial learning rate was set to 0.01, the weight decay was set to 0.0005 to prevent overfitting, and the Nesterov momentum was enabled to increase the convergence speed. In addition, we adopted a cosine annealing learning rate scheduling strategy to optimize the training process of the model. This strategy includes a warm-up phase and an annealing phase. The initial learning rate is maintained for the first 200 training cycles, and the learning rate is gradually reduced according to the cosine function in the last 100 cycles.

[0116] The Office-31 dataset is used for experiments. It consists of three different domains: Amazon (A), Webcam (W), and DSLR (D). Each domain represents a unique image capture condition and data distribution. The dataset contains 31 categories and a total of 4652 images. These categories mainly involve common office items (such as books, keyboards, and scissors). Among them, Amazon's images come from online stores and usually have a clean white background; Webcam images are low-resolution photos; and DSLR images are high-resolution photos. There are significant distribution differences between these domains. In the experiment of few-sample unsupervised domain adaptation, six typical migration tasks are set: A→W, A→D, W→A, W→D, D→A, and D→W, and the source domain sets 3 labeled samples per category.

[0117] Furthermore, the method of the present invention is compared with other methods: CDAN, CDS, PCS, PAC, C-VisDiT, EUDA, where the source domain settings of the first five methods also include unlabeled samples, while EUDA has the same settings as the present invention, and the source domain only has 3 labeled samples of each class. The comparison results are shown in Table 1 below.

[0118] Table 1 Comparison of adaptation accuracy for 3 labeled samples per category in the source domain on the Office-31 dataset.

[0119]

[0120] It can be seen from Table 1 that the classification effect of the present invention is better than other comparison methods, especially when there are only limited labeled samples in the source domain. The effectiveness of the present invention can be verified through the above experiments.

[0121] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A cross-domain image classification method based on few-sample unsupervised domain adaptation, characterized in that: The specific steps are as follows: S1: Obtain labeled source domain image sample data and unlabeled target domain image sample data as training data sets; S2: Construct feature extractor ViT and use the pre-trained model; S3: Build a model based on sample selection strategy, adversarial domain adaptation and prototype learning; S4: Get the total loss function and iterate to optimize the model; S5: Use the trained domain adaptation classification model to perform cross-domain image classification.

2. The cross-domain image classification method based on few-sample unsupervised domain adaptation according to claim 1, characterized in that: The source domain and target domain sample datasets described in step S1 are denoted as D sl and D tu , where the corresponding samples are represented as (x sl ,y sl ) and (x tu ).

3. The cross-domain image classification method based on few-sample unsupervised domain adaptation according to claim 1, characterized in that: The feature extractor described in step S2 is represented by F, and uses the ViT-B / 16 model pre-trained on ImageNet-21K to input the source domain samples and target domain samples in step S1 to extract features.

4. The cross-domain image classification method based on few-sample unsupervised domain adaptation according to claim 1, characterized in that: The sample selection strategy described in step S3 is based on the confidence and similarity of the samples, and is used to expand the source domain samples so that the model does not require additional unlabeled source domain data. It includes the following sub-steps: S31: Calculate the score matrix S = {s ij }, where s ij Represents a pair of samples The quantitative score of is defined as the Euclidean distance between corresponding features: Where F(·) represents the sample features extracted by the feature extractor; S32: By finding the minimum value s of each row ij , find the closest source sample for each target sample: S33: The most similar labeled source sample of the unlabeled target sample can be found through index j1, forming a source data set with high similarity and no interference, which is expressed as Where D tu←sl represents the set of labeled source domains that are most similar to each unlabeled sample in the target domain, Both represent the labeled source domain sample that is most similar to each unlabeled sample in the target domain, but Contains the exact index, It represents the corresponding source domain true label, N tu It represents the number of samples in the target domain; S34: Set a fixed ratio of high-confidence samples and calculate the information entropy of each sample to exclude unreliable target samples and focus on learning high-confidence target samples. Specifically, the ratio of high-confidence target samples r is defined. h , and select the one with the lowest entropy from the target sample The corresponding index of the target samples is expressed as: The above argΤοрK(·,·) means return The largest element index, represents the information entropy of each unlabeled target domain sample, and its calculation formula is: Among them, p i It is an array of probability distribution, containing the probability value of a sample in each category. S35: High confidence target domain subset and the most similar source domain subset that matches it It can be expressed as:

5. The cross-domain image classification method based on few-sample unsupervised domain adaptation according to claim 1, characterized in that: The adversarial domain adaptation described in step S3 is used to learn domain-invariant features and includes the following sub-steps: S36: The adversarial domain adaptation network includes a feature extractor F based on ViT, which converts the input sample x from the input space X to the shared latent space F(x), a classifier C that predicts the extracted features and adds corresponding labels, converts the feature vector in the shared latent space to the label space Y, and distinguishes whether the sample features come from the source domain with a domain index of 0 or the target domain with a domain index of 1. The feature extractor is optimized through adversarial training to deceive the domain discriminator and obtain the common features between the source domain and the target domain. At the same time, F and C are also trained to improve the ability to distinguish between categories; S37: The loss in the process of training the feature extractor F and the classifier C by minimizing the classification error, and the source domain sample subset in substep S35 The feature extractor is passed in, so the classification loss expression is: Where l(·,·) represents the standard cross entropy loss function; S38: In the process of training the adversarial network using the rich discriminant information provided by the classifier prediction, the conditional domain discriminator D is conditioned, and the domain discrimination loss is to maximize the distinction between the source domain and the target domain through the domain discriminator D, and its expression is: S39: The total loss expression of adversarial domain adaptation is: where λ d is a trade-off hyperparameter.

6. The cross-domain image classification method based on few-sample unsupervised domain adaptation according to claim 1, characterized in that: The prototype learning described in step S3 is a local alignment, which is used to assist the global adversarial alignment to solve the problem of cross-domain distribution differences. It includes the following sub-steps: S310: Source domain subset obtained based on sub-step S35 The original feature after the feature extractor in step S2 is F(x sl ); S311: Define a prototype layer for the K-class classification task. The prototype layer is represented by P = {p1, p2, ..., p K }, calculate the mean of the sample features belonging to category K as the category prototype of the source domain subset with K categories Its calculation expression is: Where F(·) represents the sample feature vector extracted by the feature extractor, is the number of samples of category K; S312: By minimizing the intra-category compactness loss, the compactness of the source domain sample features is enhanced, the consistency within the source domain class is promoted, and the intra-category compactness loss is defined on the source domain samples using distance measurement. The expression is: S313: After constructing the source domain prototype, align the high-confidence sample features of the target domain to achieve cross-domain feature alignment. Use the target domain predicted category probability as a weight to dynamically evaluate the loss between the target domain sample and the source domain prototype to measure the matching degree between the target sample and each category prototype. The loss expression between the high-confidence target domain sample and its corresponding source domain prototype is: in, Represents the probability that the target sample belongs to category K. By minimizing this loss term, the feature representation of the target domain sample and the source domain category prototype can be directly aligned to achieve accurate classification of the target domain sample in the feature space, while reducing the uncertainty and noise interference caused by pseudo labels; S314: Combining the intra-category compactness loss and the cross-domain prototype loss, a unified comprehensive prototype alignment loss is constructed to improve the separation and compactness of cluster distribution in cross-domain tasks. Its expression is: L proto =L cross +λ s L intra , where λ p is a hyperparameter used to control the prototype loss.

7. The cross-domain image classification method based on few-sample unsupervised domain adaptation according to claim 1, characterized in that: The total loss function described in step S4 is: L=L adv +λ p L proto , where λ p is a hyperparameter used to control the prototype loss.

Images