Domain Adaptation Image Classification Method Based on Dynamic Alignment of Feature and Label Space

Abstract

The present invention discloses a domain adaptation image classification method based on dynamic alignment of feature and label space, comprising the following steps: (1) inputting source domain images and target domain images into an adversarial unsupervised domain adaptation model, extracting high-dimensional features using a feature extractor, and training a classifier using source domain images and corresponding labels; (2) subdomain alignment of the feature space of high-dimensional features, and domain alignment through the label space; (3) inputting the high-dimensional features after domain alignment into a discriminator for domain discrimination; (4) inputting the discriminated image into a classifier, measuring the subdomain distribution distance between the source domain and the target domain, and dynamically adjusting the relationship weight between domain adaptation and domain discrimination. The present invention uses the subdomain distribution distance between the source domain and the target domain for the first time, dynamically adjusting the relationship weight between domain adaptation and domain discrimination, so that the domain adaptation ability and the domain discrimination ability are balanced, so as to achieve accurate labeling of target domain samples and make the model more generalizable.

Description

Technical Field

[0001] The present invention relates to the technical field of transfer learning, and in particular to a domain adaptation image classification method based on dynamic alignment of feature and label space. Background Art

[0002] Deep learning achieves better performance than traditional machine learning methods in most problem scenarios by learning task-specific nonlinear feature representations from massive data. However, in real-world applications, collecting labeled data is usually costly. In order to reduce the need for labeled data, a new machine learning paradigm, unsupervised domain adaptation, is proposed. It is mainly used to solve similar task decision problems with different distributions between the source domain and the target domain. It uses source domain supervision knowledge to assist in target domain model training and solves target domain tasks by transferring source domain knowledge to the target domain. At present, domain adaptation has been successfully applied in many fields, such as semantic segmentation, object tracking, image classification, pedestrian re-identification, autonomous driving, and machine translation.

[0003] Most existing unsupervised domain adaptation methods only focus on feature space alignment and often ignore the label space information of the data. In addition, existing methods rarely pay attention to the dynamic relationship adjustment between domain alignment and domain discrimination, which leads to an imbalance between domain alignment ability and domain discrimination ability, thus causing negative transfer. Summary of the invention

[0004] Purpose of the invention: In view of the above problems, the purpose of the present invention is to provide a domain adaptation image classification method based on dynamic alignment of feature and label space, which measures the subdomain distribution distance between the source domain and the target domain, adaptively learns the dynamic coefficient between domain adaptation and domain discrimination, and improves the classification accuracy of the target domain.

[0005] Technical solution: The present invention discloses a domain adaptation image classification method based on dynamic alignment of feature and label space, which uses an adversarial unsupervised domain adaptation model to classify the target domain image. The adversarial unsupervised domain adaptation model includes a feature extractor, a discriminator and three classifiers. The image classification method includes the following steps:

[0006] (1) Input the source domain image and the target domain image into the adversarial unsupervised domain adaptation model, use the feature extractor to extract the high-dimensional features of the source domain image and the target domain image, and use the source domain image and the corresponding labels to train the classifier;

[0007] (2) Subdomain alignment is performed on the feature space of high-dimensional features of the source domain and the target domain, and domain alignment is performed through the label space;

[0008] (3) The high-dimensional features of the source domain and the target domain after domain alignment are input into the discriminator, and the discriminator is used to perform domain discrimination on the input image;

[0009] (4) The discriminated image is input into the classifier, the subdomain distribution distance between the source domain and the target domain is measured, and the relationship weight between domain adaptation and domain discrimination is dynamically adjusted to achieve the classification of the target domain samples.

[0010] Furthermore, the step of training the classifier using the source domain images and corresponding labels in step (1) includes:

[0011] In unsupervised domain adaptation, we first assume that (x s ,y s ) is a labeled sample in the source domain, x t is an unlabeled sample in the target domain, and the source domain sample set Target domain sample set n s is the number of source domain samples, n t is the number of samples in the target domain, x s Obey the source domain sample distribution P(x s ), x t Obey the target domain sample distribution P(x t ), and P(x s )≠P(x t );

[0012] The classifier is initialized using supervised data from the source domain, and the cross entropy loss is used to enable the classifier to classify the source domain samples. The classification loss expression generated by the classifier is:

[0013]

[0014] In the formula, C m (.) represents the output result of the mth classifier, J(.) is the cross entropy function, M=3 represents the number of classifiers, is the i-th source domain sample, is the true label of the i-th source domain sample.

[0015] Furthermore, the step of performing subdomain alignment on the feature space of the high-dimensional features of the source domain and the target domain in step (2) includes: aligning the subdomain distribution distance between the source domain and the target domain using a subdomain distribution alignment method, and the resulting feature space alignment loss expression is:

[0016]

[0017] In the formula, K represents the total number of categories, and Respectively represent and The weight belonging to the kth class, where the weight is calculated as follows:

[0018]

[0019]

[0020] The role of Φ(.) is to map the original variable to the reproducing kernel Hilbert space. The specific expansion of the above formula is as follows:

[0021]

[0022] Among them, r(x s ,x t )=<Φ(x s ),Φ(x t )>,<.,.> represents the vector inner product.

[0023] Furthermore, the step of domain alignment of high-dimensional features through label space in step (2) includes: maximizing the label space information of the source domain and the target domain, and the information maximization loss expression is:

[0024]

[0025]

[0026] In the formula, q k (.) represents the value of the kth dimension, Represents the average output of the k-th target domain samples.

[0027] Furthermore, the domain discrimination loss expression generated when the discriminator performs domain discrimination on the input image in step (3) is:

[0028]

[0029] Where G(x s ) represents the source domain samples after G, G(x t ) represents the target domain samples passed through G.

[0030] Furthermore, the step of dynamically adjusting the relationship weight between domain adaptation and domain discrimination in step (4) includes:

[0031] Construct a dynamic balance factor to balance the network's generation ability and discrimination ability. The expression of discrimination ability is:

[0032]

[0033] Where W is the source domain and target domain features, S b is the inter-class scatter matrix, S w is the intra-class scatter matrix, h(W) represents the domain discrimination ability, the larger the h(W) value, the stronger the domain discrimination ability; tr(.) represents the trace of the matrix;

[0034] The domain discrimination ability and domain alignment ability of the adversarial unsupervised domain adaptation model are normalized, and the expressions are:

[0035]

[0036]

[0037] The weight coefficient is calculated using the normalized domain discrimination ability and domain alignment ability, and the expression is:

[0038]

[0039] Furthermore, the image classification method also includes subdomain discrimination during discrimination, and the resulting subdomain discrimination loss expression is:

[0040]

[0041] The objective function for classifying samples in the target domain is established using the weight coefficient, and the expression is:

[0042]

[0043] Among them, γ1,γ2,γ3,γ4 are the hyper parameters corresponding to the losses of each part, L MI =L MI1 +L MI2 ,The network is constrained by the overall objective function and continuously iterates to finally achieve the purpose of classifying the target domain samples.

[0044] Where M represents the number of classifiers.

[0045] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0046] 1. The present invention proposes a new adversarial unsupervised domain adaptation model, which aligns the subdomains of the feature spaces of the source domain and the target domain, significantly reduces the subdomain distribution distance between the source domain and the target domain, and improves the image classification accuracy of the model;

[0047] 2. For the first time, the label space alignment is combined with the feature space alignment. The overall domain distribution distance between the source domain and the target domain is implicitly reduced by maximizing the label space information, making the domain alignment effect of the model more significant.

[0048] 3. Combine domain discrimination and subdomain discrimination to update model parameters alternately in an adversarial manner;

[0049] 4. For the first time, the subdomain distribution distance between the source domain and the target domain is used to dynamically adjust the relationship weight between domain adaptation and domain discrimination, so as to achieve a balance between domain adaptation ability and domain discrimination ability, so as to achieve accurate labeling of target domain samples and make the model more generalizable. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a framework diagram of the present invention;

[0051] Figure 2 Schematic diagram of the structure of the adversarial unsupervised domain adaptation model;

[0052] Figure 3 Friedman test result diagram of the method of the present invention on D->A. DETAILED DESCRIPTION

[0053] The domain adaptation image classification method based on dynamic alignment of feature and label space described in this embodiment uses an adversarial unsupervised domain adaptation model to classify the target domain image. The adversarial unsupervised domain adaptation model includes a feature extractor G, a discriminator D and three classifiers CL, such as Figure 1 , 2 As shown, the image classification method includes the following steps:

[0054] (1) Input the source domain image and the target domain image into the adversarial unsupervised domain adaptation model, use the feature extractor to extract the high-dimensional features of the source domain image and the target domain image, and use the source domain image and the corresponding label to train the classifier:

[0055] In unsupervised domain adaptation, we first assume that (x s ,y s ) is a labeled sample in the source domain, x t is an unlabeled sample in the target domain, and the source domain sample set Target domain sample set n s is the number of source domain samples, n t is the number of samples in the target domain, x s Obey the source domain sample distribution P(x s ), x t Obey the target domain sample distribution P(x t ), and P(x s )≠P(x t );

[0056] The classifier is initialized using supervised data from the source domain, and the cross entropy loss is used to enable the classifier to classify the source domain samples. The classification loss expression generated by the classifier is:

[0057]

[0058] In the formula, C m (.) represents the output result of the mth classifier, J(.) is the cross entropy function, M=3 represents the number of classifiers, is the i-th source domain sample, is the true label of the i-th source domain sample.

[0059] (2) Subdomain alignment is achieved by aligning the high-dimensional features of the source domain and the target domain through feature space alignment, and domain alignment is achieved through label space alignment.

[0060] The subdomain distribution alignment method is used to align the subdomain distribution distance between the source domain and the target domain. The resulting feature space alignment loss expression is:

[0061]

[0062] In the formula, K represents the total number of categories, and Respectively represent and The weight belonging to the kth class, where the weight is calculated as follows:

[0063]

[0064]

[0065] The role of Φ(.) is to map the original variable to the reproducing kernel Hilbert space. The specific expansion of the above formula is as follows:

[0066]

[0067] Among them, r(x s ,x t )=<Φ(x s ),Φ(x t )>,<.,.> represents the vector inner product.

[0068] Maximize the label space information of the source domain and the target domain. The information maximization loss expression is:

[0069]

[0070]

[0071] In the formula, q k (.) represents the value of the kth dimension, Represents the average output of the k-th target domain samples.

[0072] (3) The high-dimensional features of the source domain and the target domain after domain alignment are input into the discriminator, and the discriminator is used to perform domain discrimination on the input image;

[0073] The domain discrimination loss expression generated when performing domain discrimination is:

[0074]

[0075] Where G(x s ) represents the source domain samples after G, G(x t ) represents the target domain samples passed through G.

[0076] When performing discrimination, subdomain discrimination is also included, and the resulting subdomain discrimination loss expression is:

[0077]

[0078] Where M represents the number of classifiers.

[0079] (4) The discriminated image is input into the classifier, the subdomain distribution distance between the source domain and the target domain is measured, and the relationship weight between domain adaptation and domain discrimination is dynamically adjusted to achieve the classification of the target domain samples.

[0080] Construct a dynamic balance factor to balance the network's generation ability and discrimination ability. The expression of discrimination ability is:

[0081]

[0082] Where W is the source domain and target domain features, S b is the inter-class scatter matrix, S w is the intra-class scatter matrix, h(W) represents the domain discrimination ability, the larger the h(W) value, the stronger the domain discrimination ability; tr(.) represents the trace of the matrix;

[0083] The domain discrimination ability and domain alignment ability of the adversarial unsupervised domain adaptation model are normalized, and the expressions are:

[0084]

[0085]

[0086] The weight coefficient is calculated using the normalized domain discrimination ability and domain alignment ability, and the expression is:

[0087]

[0088] The objective function for classifying samples in the target domain is established using the weight coefficient, and the expression is:

[0089]

[0090] Among them, γ1,γ2,γ3,γ4 are the hyper parameters corresponding to the losses of each part, LMI =L MI1 +L MI2 ,The network is constrained by the overall objective function and continuously iterates to finally achieve the purpose of classifying the target domain samples.

[0091] To verify the effectiveness of the image classification method of the present invention, experiments were conducted on the Digits digital dataset and the Office-31 dataset, respectively. The Digits digital dataset includes MNIST, USPS and SVHN, where MNIST is a handwritten 0-9 digital dataset, which contains 60,000 example training sets and 10,000 example test sets. The numbers in the dataset have been standardized and preprocessed, and the size of each image is 28*28. USPS is also a handwritten 0-9 digital dataset, which contains 7291 example training sets and 2007 example test sets. After preprocessing, the size of each image in this dataset is 16*16. The SVHN dataset comes from the house numbers in Google Street View images. Its style is similar to that of the MNIST dataset, with a total of 0-9 numbers, and it contains 73,257 example training sets and 26,032 example test sets. The present invention evaluates the effectiveness of the method proposed in this article in three domain adaptation tasks: MNIST->USPS, USPS->MNIST, and SVHN->MNIST. The Office-31 dataset contains 31 categories and 4652 images from three visual fields: Amazon (e-commerce images), Webcam (images taken by webcams), and DSLR (images taken by single-lens reflex cameras). The effectiveness of the method is evaluated in six domain adaptation tasks: A->W, D->W, W->D, A->D, D->A, and W->A.

[0092] The experimental results are shown in Table 1 and Table 2, where DAFL is the method of the present invention, which is in bold. The experiment was then subjected to a Friedman test, and the test results are shown in Table 1. Figure 3 shown.

[0093] Table 1: Verification of the effectiveness of the proposed method on the Digits dataset

[0094] Method M->U U->M S->M Average DAN 80.3±0.4 77.8±0.4 73.5±0.4 77.2 DRCN 91.8±0.3 73.7±0.2 82.0±0.6 82.5 ADDA 89.4±0.1 90.1±0.4 76.0±0.3 85.2 CyCADA 95.6±0.5 96.5±0.4 90.4±0.5 94.2 CDAN 93.9±0.4 96.9±0.3 88.5±0.1 93.1 MCD 94.2±0.5 94.1±0.5 96.2±0.4 94.8 CAT 90.6±0.4 80.9±0.4 98.1±0.2 89.9 TPN 92.1±0.6 94.1±0.2 93.0±0.5 93.1 LWC 95.6±0.3 97.1±0.3 97.1±0.3 96.6 DDAN 89.6±0.2 76.8±0.3 - 83.2 ETD 96.4±0.2 96.3±0.1 97.9±0.1 96.9 DSAN 96.9±0.2 95.3±0.1 90.1±0.4 94.1 DAFL 97.0±0.3 97.6±0.1 97.9±0.1 97.5

[0095] As shown in Table 1, the average classification accuracy of the proposed method DAFL reaches 97.5%, among which the classification accuracy of the M->U task reaches 97.0%, the classification accuracy of the U->M task reaches 97.6%, and the accuracy of the S->M task reaches 97.9%. Whether it is the classification accuracy of a single task or the average classification accuracy, the DAFL method is significantly better than other comparison methods.

[0096] Table 2: Effectiveness verification of the proposed method on the Office-31 dataset

[0097] Method A->W D->W W->D A->D D->A W->A Average ResNet50 68.4±0.3 96.7±0.3 99.3±0.4 68.9±0.4 62.5±0.2 60.7±0.3 76.1 DAN 80.5±0.2 97.1±0.2 99.6±0.1 78.6±0.3 63.6±0.3 62.8±0.4 80.4 DANN 82.6±0.2 96.9±0.3 99.3±0.1 81.5±0.1 68.4±0.5 67.5±0.2 82.7 ADDA 86.2±0.3 96.2±0.2 98.4±0.3 77.8±0.4 69.5±0.1 68.9±0.1 82.9 JAN 85.4±0.3 97.4±0.2 99.8±0.2 84.7±0.3 68.6±0.3 70.0±0.4 84.3 GTA 89.5±0.3 97.9±0.1 99.8±0.3 87.7±0.2 72.8±0.5 71.4±0.2 86.5 CAT 91.1±0.2 98.6±0.4 99.6±0.2 90.6±0.4 70.4±0.3 66.5±0.3 86.1 DDAN 86.0±0.2 97.19±0.1 99.2±0.3 86.3±0.4 72.1±0.2 73.2±0.3 85.7 ETD 92.1±0.3 100.0±0.0 100.0±0.0 88.0±0.2 71.0±0.4 67.8±0.1 86.2 DAFL 89.3±0.5 99.9±0.1 99.9±0.1 87.4±0.2 74.2±0.3 70.1±0.4 86.8

[0098] As shown in Table 2, DAFL's average classification accuracy reached 86.8%, which is significantly better than ETD's accuracy of 86.2%. The DAFL method even achieved 100% classification accuracy in the two tasks D->W and W->D. In addition, in the two tasks D->A and W->A, DAFL achieved classification accuracy of 74.2% and 70.1% respectively, which is significantly better than ETD's accuracy of 71.0% and 67.8% in the two tasks.

[0099] In order to verify that the proposed method DAFL is significantly better than other comparison methods, a Friedman test is added to the D->A task, where the numerical ranking of each method indicates the superiority of its performance, and a smaller numerical value represents a better performance of the method. Figure 3 As shown in the figure, it can be seen that the Friedman test value of this method is 1.0333, which is much smaller than the value of 1.9667 of the second-ranked GTA method, indicating that DAFL is better than other methods in terms of results and has better performance significance.

Claims

1. A domain adaptation image classification method based on dynamic alignment of feature and label space, characterized in that: The target domain image is classified using an adversarial unsupervised domain adaptation model, wherein the adversarial unsupervised domain adaptation model includes a feature extractor, a discriminator and three classifiers. The image classification method includes the following steps: (1) Input the source domain image and the target domain image into the adversarial unsupervised domain adaptation model, use the feature extractor to extract the high-dimensional features of the source domain image and the target domain image, and use the source domain image and the corresponding labels to train the classifier; (2) Subdomain alignment is performed on the feature space of high-dimensional features of the source domain and the target domain, and domain alignment is performed through the label space; (3) The high-dimensional features of the source domain and the target domain after domain alignment are input into the discriminator, and the discriminator is used to perform domain discrimination on the input image; (4) Input the discriminated image into the classifier, measure the subdomain distribution distance between the source domain and the target domain, dynamically adjust the relationship weight between domain adaptation and domain discrimination, and realize the classification of the target domain samples; The step of dynamically adjusting the relationship weight between domain adaptation and domain discrimination in step (4) includes: Construct a dynamic balance factor to balance the network's generation ability and discrimination ability. The expression of discrimination ability is: In the formula, W is the source domain and target domain features, S b is the inter-class scatter matrix, S w is the intra-class scatter matrix, h(W) represents the domain discrimination ability, the larger the h(W) value, the stronger the domain discrimination ability; tr(.) represents the trace of the matrix; The domain discrimination ability and domain alignment ability of the adversarial unsupervised domain adaptation model are normalized, and the expressions are: The weight coefficient is calculated using the normalized domain discrimination ability and domain alignment ability, and the expression is: The image classification method also includes subdomain discrimination when performing discrimination, and the resulting subdomain discrimination loss expression is: Where M represents the number of classifiers; Finally, the weight coefficient is used to establish the objective function for classifying samples in the target domain, which is expressed as: Among them, γ1, γ2, γ3, γ4 are the hyper parameters corresponding to the losses of each part, L MI =L MI1 +L MI2 ,The network is constrained by the overall objective function and continuously iterates to finally achieve the purpose of classifying the target domain samples.

2. The image classification method according to claim 1, characterized in that: The steps of training the classifier using the source domain images and corresponding labels in step (1) include: First, assume that in unsupervised domain adaptation, (x s ,y s ) is a labeled sample in the source domain, x t is an unlabeled sample in the target domain, and the source domain sample set Target domain sample set n s is the number of source domain samples, n t is the number of samples in the target domain, x s Obey the source domain sample distribution P(x s ), x t Obey the target domain sample distribution P(x t ), and P(x s )≠P(x t ); Then, the classifier is initialized using the supervised data from the source domain, and the cross entropy loss is used to enable the classifier to classify the source domain samples. The classification loss L generated by the classifier is cl The expression is: In the formula, C m (.) represents the output result of the mth classifier, J(.) is the cross entropy function, M represents the number of classifiers, M=3, is the i-th source domain sample, is the true label of the i-th source domain sample.

3. The image classification method according to claim 2, characterized in that: The subdomain alignment step of the feature space of the source domain and the target domain high-dimensional features in step (2) includes: The subdomain distribution alignment method is used to align the subdomain distribution distance between the source domain and the target domain. The resulting feature space alignment loss expression is: In the formula, K represents the number of categories contained in the source domain and the target domain. and Respectively represent and The weights belonging to the kth class, where the weights are calculated as follows: The role of Φ(.) is to map the original variable to the reproducing kernel Hilbert space. The specific expansion of the above formula is as follows: Among them, r(x s ,x t )=<Φ(x s ),Φ(x t )>,<.,.> represents the vector inner product.

4. The image classification method according to claim 3, characterized in that: In step (2), the domain alignment step of the high-dimensional features through the label space includes: Maximize the label space information of the source domain and the target domain. The information maximization loss expression is: where q k (.) represents the value of the kth dimension, Represents the average output of the k-th target domain samples.

5. The image classification method according to claim 4, characterized in that: The domain discrimination loss expression generated when the discriminator performs domain discrimination on the input image in step (3) is: Where G(x s ) represents the source domain samples after G, G(x t ) represents the target domain samples passed through G.

Images