Deep Domain Adaptive Image Classification Method Based on Prototype Network

Abstract

The present invention discloses a deep domain adaptive image classification method based on a prototype network, comprising: a pre-trained feature extractor is used to extract the expanded D S and D T Extract features to get F(D S ) and F(D T ); F(D S )、F(D T ) is input into the first nonlinear projector and the first classifier in turn to obtain the classification loss function L d2 , respectively input into the embedding module to obtain F′(D S ) and F′(D T ) and the domain confusion loss function L d1 ; Define the prototype network, and transform F′(D S ) and F′(D T ) is input into the prototype network to the prototype loss function L P1 , L P2 , through L d1 , L d2 , L P1 , L P2 The total loss function is obtained for reverse training; the trained prototype network is used for classification. The present invention solves the problem in the prior art that the source domain and the unlabeled target domain are inconsistent and the classification accuracy of the trained classifier is low when the number of given samples is small.

Description

Technical Field

[0001] This invention belongs to the field of transfer learning technology and relates to a deep domain adaptive image classification method based on prototype networks. Background Technology

[0002] Machine learning, especially deep learning, has driven the rapid development of computer vision. Its algorithms have been widely applied to solve various practical engineering problems, greatly improving the performance of computer vision tasks including image classification, object detection, object tracking, and semantic segmentation. The quality of a deep network model often depends heavily on the labeled training dataset relevant to the task. However, obtaining large-scale labeled data requires unimaginable human and material resources, and deep network models trained on labeled datasets exhibit weak robustness and extremely low generalization ability. Furthermore, machine learning requires training and testing data to be independent and identically distributed, but this constraint is difficult to guarantee in real-world applications. A deep network model trained for a specific image classification task may have strong correct classification ability for that task, but when the application scenario changes and differs from the training task, its performance may drop significantly, or even become completely unusable.

[0003] Domain adaptation relaxes two fundamental constraints of traditional machine learning: sufficient training samples and independent and identically distributed training and testing data. It leverages the correlation between the distributions of source and target domain data, aiming to solve unlabeled target domain classification tasks by utilizing knowledge from the labeled source domain. Domain adaptation typically assumes that the source and target domain data belong to the same category and continuously narrows the distribution distance between them in a high-dimensional shared space. This allows models trained on labeled source domain data to be applied to the unlabeled target domain, ensuring that performance does not significantly degrade when solving target domain classification problems.

[0004] Slightly different from the domain adaptation described above, the source and target domain data are obtained through different channels, and the target domain data is completely unlabeled. Therefore, it cannot be definitively stated whether the target domain data shares all categories with the source domain data. The target domain data may contain samples with completely different categories than those in the source domain data. Thus, in practical applications, it is difficult to ensure that the categories of the source and target domain data are completely consistent. Therefore, achieving global domain distribution alignment while maintaining the distinguishability of source and target domain categories is particularly important.

[0005] Traditional classification problems typically assume a relatively balanced distribution of samples across all classes, meaning that the number of samples in each class is roughly equal. However, often certain classes in the source domain have fewer samples, resulting in an imbalanced distribution that can lead to incorrect mappings and misclassification of samples in the target domain. Furthermore, when the target domain data size and sample labels are significantly reduced, deep network models are less likely to correctly predict the target domain category based solely on existing knowledge of the source domain. Summary of the Invention

[0006] The purpose of this invention is to provide a deep domain adaptive image classification method based on prototype networks, which solves the problems in the prior art where the source domain and unlabeled target domain categories are inconsistent and the number of given samples is small, which affects the performance of deep network models and leads to low classification accuracy of the classifier after training.

[0007] The technical solution adopted in this invention is:

[0008] A deep domain-adaptive image classification method based on prototype networks includes the following steps:

[0009] Step 1, including source domain data D S and target domain data D T The dataset was expanded;

[0010] Step 2: Establish a shared feature extractor F(·), and use the pre-trained shared feature extractor F(·) to extract features from the augmented source domain data D. S and target domain data D T Extract features to obtain feature vector F(D) S ) and F(D T );

[0011] Step 3: Construct the first nonlinear projector and the first classifier, and input the feature vector F(D) into the first nonlinear projector. S ), F(D T The input is fed into the first nonlinear projector and then into the first classifier to obtain the classification loss function L. d2 ;

[0012] Step 4: Construct and train the embedding module, and input the feature vector F(D) S ) and F(D T The embedded features F′(D) are respectively input into the embedding module to obtain the embedded features. S ) and F′(D T Loss function L for domain confusion d1 ;

[0013] Step 5 defines the prototype network, and obtains F′(D) from the embedded modules. S ) and F′(D TThe prototype network is input into the prototype network, and the prototype loss function L is obtained by training the prototype network. P1 L P2 For the loss function L d1 L d2 L P1 L P2 The weighted summation yields the total loss function, which is then used to back-train the network model.

[0014] Step 6: Use the trained prototype network to classify the images to be classified.

[0015] The invention is further characterized by:

[0016] Step 1 specifically involves inputting the source domain and target domain datasets into a random data augmentation network in batches. The random data augmentation network rotates, crops, and adds Gaussian white noise to the original source domain and target domain dataset samples to restore them to their original input size, forming new samples that are added to the original dataset, thereby expanding the dataset.

[0017] In step 2, the shared feature extractor F(·) uses a pre-trained VGG16 deep convolutional network.

[0018] Step 3 includes:

[0019] Step 3.1, the first nonlinear projector f(·) includes a fully connected layer, a ReLU activation function layer, and another fully connected layer connected in sequence, which projects the feature vector F(D) S ) and F(D T The eigenvector f(D) is obtained by inputting it into the first nonlinear projector f(·). S ) and f(D T );

[0020] Step 3.2, the feature vector f(D) S ) and f(D T The results are fed into the first classifier G to obtain the classification predictions g(D) for the source and target domains, respectively. S ) and g(D T );

[0021] Step 3.3, define the domain discriminator D. The domain discriminator D consists of three fully connected layers connected in sequence, defined with respect to the source domain data distribution P. S (x) Target domain data distribution P T The classification loss E on the neighborhood discriminator D, feature extractor F, and first classifier G of (x) is formally defined as:

[0022]

[0023] In the above formula, f iS , Representing the i-th sample in the source domain respectively The feature vector f obtained after the first nonlinear projection and classifier is i T , They represent the i-th sample in the target domain, respectively. The feature vector obtained after the first nonlinear projection and classifier is defined as h(·)=(f,g). In the formula Let f represent the outer product operation of f and g, used to fuse the outputs of the first nonlinear projection layer and the first classifier layer. Then, the loss function for learning domain-specific feature representations is further defined as:

[0024]

[0025] Using loss function L d2 The shared feature extractor, the first nonlinear projector, and the first classifier are trained in reverse.

[0026] Step 4 includes:

[0027] Step 4.1, the embedded module includes an autoencoder, the autoencoder using F(D) S ), F(D T The input is ) and the output is the corresponding dimensionality reduction and reconstruction vector. An attention mechanism is added to the autoencoder as a mandatory constraint. The attention mechanism includes a fully connected layer, where the attention score Sigmoid (FC(F)) is used to remove any domain-specific information.

[0028] The eigenvector F(D) S ), F(D T The input embedding module ultimately outputs the embedding vector F′(D). S ) and F′(D T );

[0029] Step 4.2, embed the vector F′(D) S ) and F′(D T The input is fed into the second nonlinear projector to obtain f′(D). S ) and f′(D T Then, the classification prediction results g′(D) for the source domain and target domain are obtained through the second classifier G′. S ) and g′(D T );

[0030] Step 4.3: Define the data distribution P in the source domain. S (x) Target domain data distribution P TThe domain adversarial loss E on the embedding module F′ of (x) and the second classifier G′ is formally defined as:

[0031]

[0032] In the above formula, f i S′ , Representing the i-th sample in the source domain respectively The feature vector f obtained after the second nonlinear projection and classifier is i T′ , These represent the i-th sample in the target domain after passing through... The feature vector obtained after the second nonlinear projection and classifier, let h′(·)=(f′,g′), is defined as follows. In the formula If we still represent the outer product operation of f′ and g′, then the domain adversarial loss E is equivalent to:

[0033]

[0034] Step 4.4: Although the source and target domain data are obfuscated as much as possible after the embedding module, the difficulty of obfuscating different samples varies. The network model should maintain consistency for different samples, therefore, the generalized entropy index is introduced:

[0035]

[0036] In the above formula, M represents the number of samples in each training batch, and L... k This represents the loss function value corresponding to the k-th sample. This represents the average loss function value of a batch of samples. The larger the generalized entropy exponent, the greater the sensitivity to differences at the top of the distribution. Let α = 2, calculate the generalized entropy exponent for each batch of source and target domain data and denote it as w. S and w T The loss function that ultimately achieves domain confusion through domain adversarial tactics is defined as:

[0037]

[0038] Using loss function L d1 The embedded module, the second nonlinear projector, and the second classifier are trained in reverse.

[0039] Step 5 includes:

[0040] Step 5.1, for the embedding vector F′(D) S ), F′(D T For each category, the mean of its respective feature vectors is taken as the category prototype C. k,

[0041]

[0042] In the above formula, S k This represents the set of samples of category k, and all category prototypes form a prototype network.

[0043] Step 5.2: Calculate the test sample using the distance function d(·). The squared Euclidean distance between the corresponding feature vector and each class prototype is used, and the Softmax function is used to convert it into a probability value to predict the class probability of the sample:

[0044]

[0045] Step 5.3: Using the category prediction rule from Step 5.2, for the support set samples S in the source domain data... S The corresponding true class label is c. Based on the negative log-likelihood, the prototype loss function for each round in the source domain is defined as:

[0046]

[0047] Similarly, for the support set samples S in the target domain data... T The corresponding true class label is c. Based on the negative log-likelihood, the prototype loss function for each round in the target domain is defined as:

[0048]

[0049] Step 5.4: Use the domain confusion loss function L obtained in steps 3.3 and 4.4. d1 L d1 and the prototype loss function L obtained in step 5.4 P1 L P2 Define the total loss function. By training the network model parameters in reverse, the total loss is defined as:

[0050] L total =(L P1 +L P2 )+λ(L d1 +L d1 (11)

[0051] In the above formula, λ represents the learning rate hyperparameter set to 5 to 10.

[0052] The beneficial effects of this invention are:

[0053] This paper proposes a deep domain adaptive image classification method based on a prototype network. While maintaining global domain alignment, it also preserves the domain-specific distinguishing information between the source and target domains. After domain adversarial training, domain confusion training, and prototype network training, the network model achieves better classification performance in the target domain by calculating the similarity between query set images and category prototypes. Attached Figure Description

[0054] Figure 1 This is a network structure diagram of the deep domain adaptive image classification method based on a prototype network according to the present invention;

[0055] Figure 2 This is a schematic diagram of the depth autoencoder structure in the embedding module of step 4 of the present invention. Detailed Implementation

[0056] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0057] This invention presents a deep domain adaptive image classification method based on prototype networks, such as... Figure 1 Follow these steps:

[0058] Step 1, including source domain data D S and target domain data D T The dataset was expanded;

[0059] Step 2: Establish a shared feature extractor F(·). The pre-trained shared feature extractor F(·) is used to extract the augmented source domain data D. S and target domain data D T Extract features to obtain feature vector F(D) S ) and F(D T );

[0060] Step 3: Construct the first nonlinear projector and the first classifier, and combine the feature vectors F(D) of the source domain and the target domain. S ), F(D T The input is fed into the first nonlinear projector and then into the first classifier to obtain the predicted classification result g(D). S ), g(D T This leads to the classification loss function L. d2 This is used to train the shared feature extractor, the first nonlinear projector, and the first classifier through reverse training.

[0061] Step 4: Construct and train the embedding module, which takes the feature vector F(D) from the source domain as an example. S ) and the feature vector F(D) of the target domain T The inputs are respectively fed into the embedded module to obtain F′(D) S ) and F′(DT );

[0062] Embedding vector F′(D S ), F′(D T The input is projected onto the second nonlinear projection to obtain f′(D). S ) and f′(D T Then, the second classifier G′ produces the classification prediction results g′(D) for the source and target domains, respectively. S ) and g′(D T This leads to the loss function L for domain confusion. d1 This is used to perform reverse training on the embedded module, the second nonlinear projector, and the second classifier.

[0063] Step 5: Obtain the prototype network, and then embed the modules to obtain F′(D). S ) and F′(D T The prototype network is input into the prototype network, and the prototype loss function L is obtained by training the prototype network. P1 L P2 For the loss function L d1 L d2 L P1 L P2 The weighted summation yields the total loss function, which is then used to back-train the network model.

[0064] Step 6: Use the trained prototype network to classify the images to be classified.

[0065] The invention is further characterized by:

[0066] Step 1 specifically involves the source domain dataset D. S Follows a certain distribution P S (x), with category label C S ,Right now Target domain dataset D T Follows a certain distribution P T (x), with category label C T ,Right now The source and target domain datasets are input into the random data augmentation network in batches. The random data augmentation network rotates, crops, and adds Gaussian white noise to the original source and target domain dataset samples to restore them to the original input size, forming new samples that are added to the original dataset, thereby achieving the purpose of dataset expansion.

[0067] Step 2 specifically involves using a shared feature extractor F(·) to extract features from the source and target domains. The feature extractor employs a pre-trained VGG16 deep convolutional network to obtain the feature vector F(D) of the source domain. S ) and the feature vector F(D) of the target domain TThen the obtained feature vectors are input to downstream branches such as the first nonlinear projector and the embedding module.

[0068] Step 3 is implemented in the following steps:

[0069] Step 3.1: Combine the feature vectors F(D) of the source domain and the target domain. S ) and F(D T The input is fed into the first nonlinear projector f(·), which consists of three layers: a fully connected layer, a ReLU activation function layer, and another fully connected layer. This structure aims to expand the output space and retain more feature information. After the first nonlinear projection, f(D) is obtained. S ) and f(D T );

[0070] Step 3.2, Feature vectors F(D) of the source and target domains S ) and F(D T After the first nonlinear projection, f(D) is obtained. S ) and f(D T The data is then input into the first classifier G to obtain the classification prediction results g(D) for the source domain and the target domain, respectively. S ) and g(D T );

[0071] Step 3.3: Since the data distribution of each class in the source domain and the target domain is not completely identical, a domain discriminator D is introduced. This domain discriminator D consists of three fully connected layers and aims to preserve the domain-specific information in each domain. It is defined on the data distribution P of the source domain. S (x) Target domain data distribution P T The classification loss E on the neighborhood discriminator D, feature extractor F, and first classifier G of (x) is formally defined as:

[0072]

[0073] In the above formula, f i S , Representing the i-th sample in the source domain respectively The feature vector f obtained after the first nonlinear projection and classifier is i T , They represent the i-th sample in the target domain, respectively. The feature vector obtained after the first nonlinear projection and classifier is defined as h(·)=(f,g). In the formula Let f represent the outer product operation of f and g, used to fuse the outputs of the first nonlinear projection layer and the first classifier layer. Then, the loss function for learning domain-specific feature representations is further defined as:

[0074]

[0075] Using loss function L d2 The shared feature extractor, the first nonlinear projector, and the first classifier are trained in reverse.

[0076] like Figure 2 Step 4 is implemented in the following steps:

[0077] Step 4.1: Extract the feature vectors F(D) from the source and target domains obtained by the feature extractor. S ), F(D T The input is fed into the embedding module F′. The embedding module consists of an autoencoder and an attention mechanism. The autoencoder uses F(D) as input. S ), F(D T Given the input F′(D), the output is the corresponding dimensionality-reconstructed vector. To maintain global domain alignment, i.e., to achieve domain confusion between the source and target domains as much as possible, a mandatory constraint needs to be added. This constraint is an attention mechanism FC(·) composed of fully connected layers, and the attention score Sigmoid(FC(F)) is used to remove any domain-specific information. Combining the autoencoder and the attention submodule, the final output of the embedding module is F′(D). S ) and F′(D T );

[0078] Step 4.2: After passing through the embedding module, the domain confusion embedding vector F′(D) is obtained. S ), F′(D T The input is projected onto the second nonlinear projection to obtain f′(D). S ) and f′(D T Then, the second classifier G′ produces the classification prediction results g′(D) for the source and target domains, respectively. S ) and g′(D T );

[0079] Step 4.3: Define the data distribution P in the source domain. S (x) Target domain data distribution P T The domain adversarial loss E on the embedding module F′ of (x) and the second classifier G′ is formally defined as:

[0080]

[0081] In the above formula, f i S′ , Representing the i-th sample in the source domain respectively The feature vector f obtained after the second nonlinear projection and classifier is i T′ , These represent the i-th sample in the target domain after passing through... The feature vector obtained after the second nonlinear projection and classifier, let h′(·)=(f′,g′), is defined as follows. In the formula If we still represent the outer product operation of f′ and g′, then the domain adversarial loss E is equivalent to:

[0082]

[0083] Step 4.4: Although the source and target domain data are obfuscated as much as possible after the embedding module, the difficulty of obfuscating different samples varies. The network model should maintain consistency for different samples, therefore, the generalized entropy index is introduced:

[0084]

[0085] In the above formula, M represents the number of samples in each training batch, and L... k This represents the loss function value corresponding to the k-th sample. This represents the average loss function value of a batch of samples. The larger the generalized entropy exponent, the greater the sensitivity to differences at the top of the distribution. Let α = 2, calculate the generalized entropy exponent for each batch of source and target domain data and denote it as w. S and w T The loss function that ultimately achieves domain confusion through domain adversarial tactics is defined as:

[0086]

[0087] Using loss function L d1 The embedded module, the second nonlinear projector, and the second classifier are trained in reverse.

[0088] Step 5 is implemented in the following steps:

[0089] Step 5.1: After the embedding module, the domain confusion embedding vector F′(D) of the source and target domains is obtained. S ), F′(D T For each category, the mean of its respective feature vectors is taken as the category prototype C. k The category prototype is then defined as:

[0090]

[0091] In the above formula, S k This represents the set of samples of category k, and all category prototypes form a prototype network.

[0092] Step 5.2: Calculate the test sample using the distance function d(·). The squared Euclidean distance between the corresponding feature vector and each class prototype is used, and the Softmax function is used to convert it into a probability value to predict the class probability of the sample:

[0093]

[0094] Step 5.3: Using the category prediction rule from Step 5.2, for the support set samples S in the source domain data... S The corresponding true class label is c. Based on the negative log-likelihood, the prototype loss function for each round in the source domain is defined as:

[0095]

[0096] Similarly, for the support set samples S in the target domain data... T The corresponding true class label is c. Based on the negative log-likelihood, the prototype loss function for each round in the target domain is defined as:

[0097]

[0098] Step 5.4: Use the domain confusion loss function L obtained in steps 3.3 and 4.4. d1 L d1 and the prototype loss function L obtained in step 5.4 P1 L P2 Define the total loss function. By training the network model parameters in reverse, the total loss is defined as:

[0099] L total =(L P1 +L P2 )+λ(L d1 +L d1 (11)

[0100] In the above formula, λ represents the learning rate hyperparameter, which is set to 5 based on the network convergence. ～ 10.

[0101] Step 6 is implemented in the following steps:

[0102] During the testing phase, given the target domain test task The query set image S that needs to be classified Q For N-way and K-shot tasks, a pre-trained prototype network is used to process the query set image S. Q Classification is performed by comparing the query set images S. Q Predict the corresponding category by calculating the squared Euclidean distance between the prototype and each category.

Claims

1. A deep domain adaptive image classification method based on prototype network, characterized in that: The following steps are involved: Step 1: Include source domain data and target domain data Expand the data set; Step 2: Create a shared feature extractor , using a pre-trained shared feature extractor The expanded source domain data and target domain data Extract features and get feature vectors and ; Step 3: Construct the first nonlinear projector and the first classifier, and transform the feature vector , After being input into the first nonlinear projector and then into the first classifier, the classification loss function is obtained. ; Step 4: Build and train the embedding module to transform the feature vector and Input into the embedding module to get the embedding features and Loss function for domain confusion ; Step 5 defines the prototype network and embeds the module to obtain and Input into the prototype network and train the prototype network to obtain the prototype loss function , , for the loss function , , Perform weighted superposition to obtain the total loss function for reverse training of the network model; Step 6, using the trained prototype network to classify the image to be classified; The step 3 comprises: Step 3.1, first nonlinear projector It includes a fully connected layer, a ReLU activation function layer, and a fully connected layer connected in sequence. The feature vector and Input to the first nonlinear projector Get the feature vector and ; Step 3.2: transform the feature vector and Enter the first classifier The classification prediction results of the source domain and the target domain are obtained and ; Step 3.3, define the domain discriminator , domain discriminator It includes three fully connected layers connected in sequence, defined on the source domain data distribution Target domain data distribution Domain discriminator , Feature Extractor and the first classifier The classification loss on , the loss function is formally defined as: （1） In the above formula, , Represent the source domain Samples The feature vector obtained after the first nonlinear projection and classifier is , Represent the target domain Samples The feature vector obtained after the first nonlinear projection and classifier is set ,definition , where express and The outer product operation is used to fuse the outputs of the first nonlinear projection layer and the first classifier layer. The loss function of learning domain-specific feature representation is further defined as: （2） Using loss function , reversely training the shared feature extractor, the first nonlinear projector, and the first classifier; The step 4 comprises: Step 4.1, the embedding module includes an autoencoder, which is , As input, the corresponding dimension reduction reconstruction vector is output. The attention mechanism is added as a mandatory constraint in the autoencoder. The attention mechanism includes a fully connected layer, in which the attention mechanism uses the attention score To remove any domain specific information; The feature vector , The input embedding module finally outputs the embedding vector and ; Step 4.2: embed the vector and Input to the second nonlinear projector to obtain and ; Then pass through the second classifier The classification prediction results of the source domain and the target domain are obtained and ; Step 4.3: Define the data distribution in the source domain Target domain data distribution Embedded Module and the second classifier Domain adversarial loss on , the loss function is formally defined as: （3） In the above formula, , Represent the source domain Samples The feature vector obtained after the second nonlinear projection and classifier is , Represent the target domain Samples passed The feature vector obtained after the second nonlinear projection and classifier is set ,definition , where Still said and The outer product operation of the domain adversarial loss is equivalent to: （4） Step 4.4: Although the source domain and target domain data are as confused as possible after the embedding module, the difficulty of domain confusion varies for different samples. The network model should remain consistent for different samples, so the generalized entropy index is introduced: （5） In the above formula, Represents the number of samples in each training batch, Represents the The corresponding loss function value of samples is, Represents the average loss function value of a batch of samples, let , calculate the generalized entropy index corresponding to each batch of source domain and target domain data and record it as and , then the loss function for achieving domain confusion through domain adversarial is defined as: （6） Using loss function The embedding module, the second nonlinear projector, and the second classifier are trained backwards.

2. The deep domain adaptive image classification method based on prototype network according to claim 1, characterized in that: The step 1 specifically includes inputting the source domain and target domain data sets into the random data augmentation network in batches. The random data augmentation network rotates, crops, and adds Gaussian white noise to the original source domain and target domain data set samples and restores them to the original input size to form new samples and add them to the original data set, thereby expanding the data set.

3. The deep domain adaptive image classification method based on prototype network according to claim 1, characterized in that: The feature extractor shared in step 2 Use the pre-trained VGG16 deep convolutional network.

4. The deep domain adaptive image classification method based on prototype network according to claim 1, characterized in that: The step 5 comprises: Step 5.1, embedding vector , , for each category, take the mean of its respective feature vector as the category prototype , In the above formula, Indicates the category A sample set, all category prototypes form a prototype network; Step 5.2: Using distance function Calculate the test sample The squared Euclidean distance between the corresponding feature vector and each category prototype is converted into a probability value using the Softmax function to predict the category probability of the sample: （8） Step 5.3: Using the category prediction rule in step 5.2, for the support set samples in the source domain data , and its corresponding true category label is , based on the negative log-likelihood, the prototype loss function of the source domain in each round is defined as: （9） Similarly, for the support set samples in the target domain data , and its corresponding true category label is , based on the negative log-likelihood, the prototype loss function of the target domain in each round is defined as: （10） Step 5.4: Use the domain confusion loss function obtained in steps 3.3 and 4.4 , And the prototype loss function from step 5.4 , , define the total loss function, and by reverse training the network model parameters, the total loss is defined as: （11） In the above formula Represents the learning rate hyperparameter, set to 5~10.

Images