Unsupervised domain adaptive image recognition method based on prototype network contrast learning

Abstract

The application discloses an unsupervised domain adaptive image recognition method based on a prototype network contrast learning, and steps of the method comprise the following steps: 1, defining a domain adaptive image dataset with c categories; 2, constructing a contrast learning domain adaptive network model based on a prototype network; 3, training the constructed network model by using a domain bias prototype modeling strategy and a prototype level contrast learning strategy; 4, constructing an overall loss function optimization model weight parameter to obtain an optimal target domain image classification model. The application can realize accurate identification of the target domain image categories by exploring the semantic structure of the image features between the domains to construct the domain bias prototype, and then adaptively aligning the prototype to reduce the difference between the domains.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision transfer learning domain adaptation image classification, and particularly relates to an unsupervised domain adaptation image recognition method based on prototype network contrastive learning. BACKGROUND

[0002] In recent years, with the development of deep convolutional neural networks (CNN), many computer vision models based on rich labeled data have achieved excellent performance. However, the performance of these models is often affected by the distribution difference between different data sets. For example, sketches usually lack detailed color information, while real-world photos show rich colors. Due to domain bias, a network trained on sketch images performs poorly on real photos. An effective way to reduce the domain bias problem is to obtain a large amount of labeled data in the new domain. However, this method is often impractical due to high cost. Therefore, it is very important to reduce the domain difference during the training phase. In recent years, unsupervised domain adaptation (UDA) has become a mainstream method to solve the problem of domain gap, which aims to transfer the knowledge of the labeled source domain to the unlabeled target domain.

[0003] In unsupervised domain adaptation, labeled images in the source domain and unlabeled images in the target domain are used to train the model. By learning the shared representation between the source domain and the target domain, the model can make good predictions on the target domain. This usually involves designing an adaptive loss function to minimize the difference between the source domain and the target domain.

[0004] However, in the case of large domain difference, it becomes challenging to align the features due to the collapse of the distribution structure between domains. This is because previous methods ignore the semantic relationship between features, resulting in suboptimal models that produce negative transfer, making it difficult for the model to distinguish the classes to which the target domain images belong. In addition, when there are similar feature classes in the domain, the model will have difficulty distinguishing them, for example, "tiger" is obviously more similar to "lion" than to "key". Therefore, the model is more likely to misclassify "lion" as "tiger" rather than "key". Two classes with similar features can be considered "difficult pairs", and vice versa "easy pairs". At present, most domain adaptation methods do not properly handle "difficult pairs" and "easy pairs". SUMMARY

[0005] The present application is to solve the above-mentioned deficiencies in the prior art, and proposes an unsupervised domain adaptation image recognition method based on prototype network contrastive learning, so as to express the domain semantic structure by constructing cross-domain class prototypes based on the semantic information of image features, and to handle "difficult pairs" on the basis of prototypes when reducing the difference between domains, thereby improving the accuracy of target domain image classification.

[0006] The application achieves the above-mentioned purposes by adopting the following technical scheme.

[0007] The unsupervised domain adaptive image recognition method based on the prototype network contrast learning has the following steps.

[0008] Step 1: defining the domain adaptive image dataset as D={D s , D t}, D s represents a source domain image set containing a class label, D t represents a target domain image set without a class label, wherein x s represents any labeled source domain image in the source domain image set, y s represents the class label of x s , y s ∈{1, 2,..., c}; c represents the total number of classes; (x s , y s )∈D s , n s represents the total number of source domain images; x t represents an unlabeled target domain image, x t ∈D t , n t represents the total number of target domain images; and the source domain and the target domain belong to the same label space.

[0009] Step 2: constructing a prototype-based contrast learning domain adaptive network model for image classification of the target domain image x t .

[0010] The contrast learning domain adaptive network model comprises a feature extraction network E, a task-specific classifier C, a graph convolution affinity matrix generation network G A , a graph convolution node aggregation network G N and a graph convolution classifier G c .

[0011] The convolution layer and the fully connected layer in the contrast learning domain adaptive network model are initialized by xavier initialization.

[0012] Step 3: the training strategy of the prototype network-based contrast learning domain adaptive network model comprises a domain bias prototype modeling strategy and a prototype-level contrast learning strategy.

[0013] Step 3.1: domain bias prototype modeling strategy:

[0014] Step 3.1.1: the source domain image x s and the target domain image x tThe input is processed in the feature extraction network E, and the source domain features f are obtained accordingly. s and target domain features f t ;

[0015] The source domain feature f s The input is processed by the task-specific classifier C to obtain the source domain image x. s Output category probability prediction distribution p(y′) s |C(E(x s )));wherein, y′ s x represents s Predicted category labels;

[0016] The cross-entropy loss function L of the task-specific classifier C is established using equation (1). ce :

[0017]

[0018] The target domain feature f t The input is processed by the task-specific classifier C to obtain the target domain image x. t Normalized probability output for the k-th class Then the target domain dataset D t Average normalized probability output in the k-th category

[0019] Using equation (2), the mutual information loss L in the target domain is established. MI :

[0020]

[0021] Step 3.1.2: From the dataset D = {D} s D t Take a batch of data B = (B) from the data. s B t ), where B s B represents a batch of source domain images containing category labels. t This represents a batch of target domain images that do not contain category labels;

[0022] Construct a fully connected graph structure G = (V, A) for all images in B, where V represents the set of nodes and A represents the affinity matrix;

[0023] Use the target domain features or source domain features as the initial features of any node in V;

[0024] Let the i-th node v i and the j-th node v j Node pairs (v)i v j Input the graph convolutional affinity matrix generator network G. A The process is performed to obtain node pairs (v) i v j Similarity score between And as an unnormalized affinity matrix The element in the i-th row and j-th column of the array, where σ(·) is the activation function;

[0025] right After normalization, the normalized similarity score a is obtained. i，j And use it as the element in the i-th row and j-th column of A, thus obtaining the normalized affinity matrix. D is The degree matrix, where I is the identity matrix;

[0026] Step 3.1.3: Based on the label information, use equation (3) to obtain the element t in the i-th row and j-th column of the real matrix T. i，j :

[0027]

[0028] In equation (3), y i and y j For the i-th node v i and the j-th node v j The category label of the image; where, if the i-th node v i Or the j-th node v j The image belongs to B s Source domain image in Then y i or y j For source domain image Real category labels If the i-th node v i Or the j-th node v j The image belongs to B t Target domain image Then y i or y j For target domain image The pseudo-labels obtained after inputting into the classifier C for the specific task This indicates that the task-specific classifier C predicts the target domain image. The probability of the k-th category;

[0029] Set a probability threshold τ, if Then delete and The corresponding target domain image edges connected to the node, so that the affinity matrix A in the affinity matrix A and the similarity score of the combined node pair is set to zero.

[0030] The binary cross-entropy loss L of the graph convolution node aggregation network G A is established by using formula (4): bce

[0031]

[0032] Step 3.1.4: input all nodes in V into the graph convolution node aggregation network G N , and update the node features in V using formula (5):

[0033]

[0034] In formula (5), f i represents the initial feature of the i-th node v i , f j represents the initial feature of the j-th node v j , and f i represents the aggregated feature of the i-th node v c .

[0035] The cross-entropy loss function L of the graph convolution classifier G c is established by using formula (6):

[0036]

[0037] In formula (6), f represents the aggregated feature of the node to which the source domain image belongs;

[0038] Step 3.1.5: the source domain bias prototype B and the target domain bias prototype B of the k-th category are calculated by using formula (7) and formula (8) respectively:

[0039]

[0040]

[0041] In formula (7) and formula (8), and respectively represent all image sets of the k-th category in Bs and Bt, respectively represent the aggregated feature of the node to which the target domain image belongs;

[0042] ​Step 3.1.6: Update the global bias prototype of the k-th class using the exponential moving average strategy shown in equations (9) and (10):

[0043]

[0044]

[0045] In equations (9) and (10), ← indicates assignment, and ρ represents the balance parameter. and Let represent the source domain global bias prototype and the target domain global bias prototype of the k-th category in the (I-1)th iteration, respectively; and Let the source domain bias prototype and target domain bias prototype of the k-th class in the i-th iteration be respectively. When i = 1, let

[0046] Step 3.2: Construct the loss function L of the prototype-level contrastive learning strategy using equation (12). proNCE :

[0047]

[0048] In equation (12), u represents the bias prototype with category label k, u + Let u be a unique positive sample, and let k be a unique bias prototype in another domain that is of the same class as u. - A negative sample represents a bias prototype of other categories in two domains; N is the set of all bias prototypes. - Let w(·,·) represent the set of all negative samples relative to u, where w(·,·) is the weight function and φ(·,·) is the distance metric function.

[0049] Step 4: Model Training

[0050] Step 4.1: Construct the overall loss function L using equation (13) overall :

[0051]

[0052] In equation (13), λ1, λ2 and λ3 are three weight parameters, and γ is an adaptive parameter that increases with the number of iterations;

[0053] Step 4.2: Define the E-parameters of the feature extraction network as θ. e The task-specific classifier C has parameters θ. c Graph convolutional parent matrix generator network G A Graph convolutional node aggregation network G N and graph volume integrator G c The parameter is θg , thereby constructing the parameter updating strategy by using formula (14):

[0054]

[0055] In formula (14), η represents a learning rate;

[0056] Step 4.3: based on the domain adaptation image dataset D={D s , D t}, the contrastive learning domain adaptation network model is trained by using the random gradient descent method, and the model parameters are updated by using formula (14) until the overall loss function L overall converges, thereby obtaining the optimal contrastive learning domain adaptation network model for realizing the classification of the target domain image.

[0057] The unsupervised domain adaptation image recognition method based on the prototype network contrastive learning has the characteristics that the feature extraction network E in the step 2 is based on the ResNet-50 structure and sequentially comprises an input convolutional layer, a maximum pooling layer, M down-sampling convolutional layers, Q residual convolutional blocks and an average pooling layer; and a batch normalization layer and a ReLU activation function are connected after each convolutional layer.

[0058] The task-specific classifier C is composed of H layers of full connection networks, and a batch normalization layer and a ReLU activation function are connected after the first layer of full connection networks.

[0059] The graph convolution affinity matrix generation network G A and the graph convolution node aggregation network G N are both composed of N layers of convolutional layers with a convolution kernel of 1*1, and a batch normalization layer and a LeakyRuLe activation function are connected after each convolutional layer.

[0060] The graph convolution classifier G c is composed of one layer of convolutional layers with a convolution kernel of 1*1 and a batch normalization layer.

[0061] The electronic device comprises a memory and a processor, and the memory is used for storing a program supporting the processor to execute the unsupervised domain adaptation image recognition method, and the processor is configured to execute the program stored in the memory.

[0062] The computer readable storage medium stores a computer program, and the computer program is executed by the processor to execute the steps of the unsupervised domain adaptation image recognition method.

[0063] Compared with the prior art, the present application has the beneficial effects that:

[0064] 1. In order to solve the problem of negative transfer caused by excessive domain gap, this invention introduces semantic information modeling into graph convolutional networks to aggregate image features from different domains and generate a domain bias prototype. This prototype combines features of the same category from different domains, and the intermediate domain gap it represents is small, so it is easier to transfer. This enables the model to learn semantic knowledge of the target domain from the semantic distribution of the source domain, thereby improving the model's discrimination ability on target domain images.

[0065] 2. This invention introduces contrast loss to enhance semantic discriminativeness, making prototypes of different categories farther apart and prototypes of the same category closer together; by adaptively setting weights for "difficult pairs" and "easy pairs", the model has the ability to distinguish "difficult pairs", thereby improving the model's ability to distinguish target domain images. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating the training process of the method of the present invention;

[0067] Figure 2 This is a schematic diagram of the overall model of the present invention;

[0068] Figure 3 This is a schematic diagram illustrating the feature aggregation of the present invention. Detailed Implementation

[0069] In this embodiment, as Figure 1 As shown, an unsupervised domain adaptive image recognition method based on prototype network contrastive learning proceeds according to the following steps:

[0070] Step 1: Define the domain-adaptive image dataset as D = {D} s D t}, D s D represents the source domain image set containing category labels. t Let x represent the target domain image set without category labels, where x s Let y represent any labeled source image in the source image set. s x represents s Category labels, y s ∈{1, 2, ..., c}; c represents the total number of categories; (x s y s )∈D s Let n s Let x represent the total number of source domain images; t Represents the unlabeled target domain image, x t ∈D t Let n tdenotes the total number of target domain images; and the source domain and the target domain belong to the same label space; in this embodiment, all the training images are randomly cropped to have a pixel size of 224x224 after normalization processing, and the test images are center-cropped to have a pixel size of 224x224;

[0071] In this embodiment, the method is evaluated on five public datasets, namely Office-31, OfficeHome, ImageCLEF-DA, VisDA-2017 and DomainNet, respectively.

[0072] Office-31 is a commonly used benchmark dataset and is often used for DA tasks. It consists of a total of 4110 images, divided into 31 different categories, and contains three different domains: Amazon (A), DSLR (D) and Webcam (W); one of the domains is used as the source domain and another domain is used as the target domain, and there are 6 different combinations of transfer tasks;

[0073] ImageCLEF-DA is a benchmark dataset for ImageCLEF-DA 2014 domain adaptation challenge; it includes three domains: Caltech-256 (C), ImageNet ILSVRC 2012 (I) and Pascal VOC 2012 (P); each domain consists of 12 categories, and each category has 50 images;

[0074] Office-Home consists of 15500 images distributed in four domains, each containing 65 different categories; the four domains in this dataset are art, clip art, product and real world, and there are a total of 12 transfer tasks;

[0075] VisDA-2017 is a large-scale DA benchmark dataset, which includes synthetic image domains and real image domains; it has a total of 12 categories; the synthetic image domain contains a huge set of 152,409 images, while the real image domain contains 55400 images; when evaluating, the synthetic image is used as the source domain and the real image is used as the target domain to train the model;

[0076] DomainNet is one of the largest datasets in DA, containing 345 categories and about 600,000 images; DomainNet consists of six domains with significant differences: Clipart (clp), Infograph (inf), Painting (pnt), Quickdraw (qdr), Real (rel) and Sketch (skt); due to the large number of domains involved, there are a total of 30 transfer tasks on this dataset;

[0077] Step 2: as Figure 2As shown, a prototype-based contrastive learning domain adaptation network model is constructed for image classification of target domain images x t ;

[0078] The contrastive learning domain adaptation network model comprises a feature extraction network E, a task-specific classifier C, a graph convolution affinity matrix generation network G A , a graph convolution node aggregation network G N , and a graph convolution classifier G c ;

[0079] The feature extraction network E is based on the ResNet-50 structure and sequentially comprises an input convolutional layer, a max-pooling layer, M down-sampling convolutional layers, Q residual convolutional blocks, and an average-pooling layer; and a batch normalization layer and a ReLU activation function are connected after each convolutional layer; the output feature dimension of the feature extraction network E is 256, and the image input feature extraction network E outputs the features used for training;

[0080] The task-specific classifier C is composed of H fully connected networks, and a batch normalization layer and a ReLU activation function are connected after the first fully connected network; in order to match the classification task, the output dimension of the last fully connected layer is specific to the total number of dataset categories;

[0081] The graph convolution affinity matrix generation network G A and the graph convolution node aggregation network G N are both composed of N convolutional layers with a 1×1 convolution kernel (the convolution kernel size is 1×1 and the step is 1; the input channel of G A is 256 and the output channel is 1, and the input channel of G N is 512 and the output channel is 256), and a batch normalization layer and a LeakyRuLe activation function are connected after each convolutional layer;

[0082] The graph convolution classifier G c is composed of a convolutional layer with a 1×1 convolution kernel (the convolution kernel size is 1×1 and the step is 1; the input channel is 256 and the output channel is specific to the total number of dataset categories) and a batch normalization layer;

[0083] The convolutional layers and fully connected layers in the prototype-based contrastive learning domain adaptation network model are all initialized with xavier initialization;

[0084] Step 3: As shown in Figure 2 , the training strategy of the prototype network-based contrastive learning domain adaptation network model comprises a domain bias prototype modeling strategy and a prototype-level contrastive learning strategy;

[0085] Step 3.1: Domain bias prototype modeling strategy:

[0086] Step 3.1.1: Source domain image x s and target domain image x t The input features are processed in the feature extraction network E, and the source domain features f are obtained accordingly. s and target domain features f t Both the source and target domains have a feature dimension of 256.

[0087] Source domain features f s The input is processed by a task-specific classifier C to obtain the source domain image x. s Output category probability prediction distribution p(y′) s |C(E(x s )));wherein, y′ s x represents s Predicted category labels;

[0088] The cross-entropy loss function L of the task-specific classifier C is established using equation (1). ce :

[0089]

[0090] By using source domain images and labels to train the classifier's discriminative ability, it can also perceive the semantic features of the target domain.

[0091] The target domain features f t The input is processed by a task-specific classifier C to obtain the target domain image x. t Normalized probability output for the k-th class Then the target domain dataset D t Average normalized probability output in the k-th category

[0092] Using equation (2), the mutual information loss L in the target domain is established. MI :

[0093]

[0094] In information theory, mutual information measures the correlation between two random variables. In practice, a strong correlation between the target feature and the prediction will be beneficial for semantic enhancement because the extracted features will provide more information and contain more important predictive semantics, ignoring trivial semantics. Therefore, this method uses mutual information maximization on the target data to learn more important predictive semantic information.

[0095] Step 3.1.2: From the dataset D = {D} s D t Take a batch of data B = (B) from the data. s B t), where B s B represents a batch of source domain images containing category labels. t This represents a batch of target domain images without category labels; the training process uses a batch of images to construct the graph, otherwise the constructed graph would be too large to train.

[0096] Construct a fully connected graph structure G = (V, A) for all images in B, where V represents the set of nodes and A represents the affinity matrix;

[0097] In graph G, any node is connected to all other nodes.

[0098] Use either the source domain features or the target domain features as the initial features of any node in V;

[0099] Let the i-th node v i and the j-th node v j Node pairs (v) i v j Input graph convolutional affinity matrix generator network G A The process is performed to obtain node pairs (v) i v j Similarity score between And as an unnormalized affinity matrix The element in the i-th row and j-th column of node v, where σ(·) is the activation function; i and node v j Both feature dimensions are 256. After calculating the absolute value of both, the feature dimensions are transformed into channel dimensions. Input G A The resulting 1D channel value is for node v. i and node v j Unnormalized similarity scores between

[0100] right After normalization, the normalized similarity score a is obtained. i，j And use it as the element in the i-th row and j-th column of A, thus obtaining the normalized affinity matrix. D is The degree matrix, where I is the identity matrix;

[0101] Step 3.1.3: Based on the label information, use equation (3) to obtain the element t in the i-th row and j-th column of the real matrix T. i，j :

[0102]

[0103] In equation (3), y i and y j For the i-th node v i and the j-th node vj the class label of the image to which the i-th node v i or the j-th node v j belongs; wherein, if the image to which the i-th node v s or the j-th node v belongs is a source domain image in B i , then y j or y is the true class label of the source domain image if the image to which the i-th node v i or the j-th node v j belongs is a target domain image in B t , then y i or y j is the true class label of the target domain image the pseudo label obtained after inputting the classifier C specific to the task; denotes the probability that the classifier C specific to the task predicts the target domain image to be of the k-th class;

[0104] a probability threshold τ is set, if then the target domain image corresponding to is deleted, and the edge connected to the node to which the image belongs is set to zero, so that the similarity score of the node pair combined with in the affinity matrix A is set to zero; the pseudo label of the image is considered to have low confidence, and the edge connected to the image in the graph G is deleted;

[0105] The binary cross entropy loss L A of the graph convolutional node aggregation network G bce is established by using formula (4):

[0106]

[0107] The network is constrained by the binary cross entropy loss, and it is hoped that the unnormalized similarity score A output by the graph convolutional node aggregation network G is closer to the element t i，j corresponding to the real matrix T; if t i，j = 1, it indicates that the nodes v i and v j are of the same class, the value of which is closer to 1, and vice versa, the value of which is closer to 0; since t i，j is constructed from label information, the normalized similarity score a i，j can express the semantic relationship between the nodes v i and v j .​

[0108] Step 3.1.4: As Figure 3 As shown, all nodes in V are input into the graph convolutional node aggregation network G. N In the process, the node features in V are updated using equation (5):

[0109]

[0110] In equation (5), fi represents the i-th node v i The initial features, f j Represents the j-th node v j initial characteristics, Represents the i-th node v i Aggregation characteristics;

[0111] If the similarity score is a i，j The higher the value, the higher the value at the i-th node v. i aggregation features It aggregates more of the j-th node v j initial features f j Image features are aggregated based on similarity scores, and these aggregated features describe the semantic relationships between images. Simultaneously, feature aggregation is performed interactively between the source and target domains, enabling the transfer of semantic feature information between different domains and improving the generalization of the target domain.

[0112] The graph volume integrator G is established using equation (6). c Cross-entropy loss function

[0113]

[0114] In equation (6), Represents the source domain image The aggregation characteristics of the node to which it belongs;

[0115] Due to source domain aggregation characteristics It also aggregates similar target domain image features, so the image convolution classifier G is trained using cross-entropy loss. c It can improve its ability to identify target domain images;

[0116] Step 3.1.5: Calculate the source domain bias prototype of the k-th category using equations (7) and (8) respectively. and target domain bias prototype

[0117]

[0118]

[0119] In formula (7) and formula (8), and respectively represent B s and B t all image sets of the k-th category in the source domain, represent target domain images the aggregated features of the belonging nodes;

[0120] Since and the source domain bias prototype and the target domain bias prototype are calculated from the aggregated features of the source domain and the target domain images, and the target domain bias prototype contains the image features of another domain whose category label is k, so the difference between the intermediate domain feature space represented by the source domain bias prototype and the target domain bias prototype is much smaller than the difference between the source domain feature space and the target domain feature space, and the intermediate domain feature space is more conducive to feature migration, so that the generalization of the model on the target domain is improved;

[0121] Step 3.1.6: Update the global bias prototype of the k-th category by using the exponential moving strategy shown in formula (9) and formula (10):

[0122]

[0123]

[0124] In formula (9) and formula (10), <- represents assignment, and represents a balance parameter, and respectively represent the source domain global bias prototype and the target domain global bias prototype of the k-th category in the I-1-th iteration; and respectively represent the source domain bias prototype and the target domain bias prototype of the k-th category in the I-th iteration, and when I=1, let wherein, ρ is set to 0.7; wherein and are the source domain global bias prototype and the target domain global bias prototype of the k-th category in the I-th iteration; the feature space represented thereby represents the distribution of the overall data set of the source domain and the target domain, and then prototype-level contrastive learning training is performed to reduce the difference between the domains;

[0125] Step 3.2: After modeling the domain bias prototype, the domain difference is reduced to a certain extent, and the transferability is guaranteed to a certain extent. Since the downstream task of the model is the classification task of the target domain image, it is hoped that the model can learn semantic discriminativeness. Since the prototypes of different categories represent the feature distribution of different categories, the same category prototypes are aggregated and the different categories prototypes are far apart, which can well maintain the semantic structure between the domains. This allows the semantic information learned on the source domain dataset to be well transferred to the target domain dataset, improving the recognition ability of the target domain image. The loss function L of the prototype-level contrastive learning strategy is constructed using Equation (12). proNCE :

[0126]

[0127] In equation (12), u represents the bias prototype with category label k, u + Let u be a unique positive sample, and let k be a unique bias prototype in another domain that is of the same class as u. - A negative sample represents a bias prototype of other categories in two domains; N is the set of all bias prototypes. - Let L represent the set of all negative samples relative to u, where φ(·,·) is the distance metric function and w(·,·) is the weight function; where φ(·,·) is the cosine distance, and the smaller φ(·,·) is, the more similar the two elements are. In equation (12), L proNCE The optimization objective is to make φ(u, u) in the molecule - The larger the value, the more distinct the prototypes u and u' are in the category. - Further apart; at the same time, make φ(u, u) in the denominator + The smaller the value, the more likely the prototypes u and u of the same category are to be identical. + Closer; where τ is the temperature coefficient, set to 0.02; where w(·,·) is the cosine similarity, when w(u,u) - The larger the ) is, the larger the prototype u and u - The more similar, in L proNCE The larger the weight in the matrix, the better the relationship between u and u. - Pull further apart; when w(u, u) - The smaller the value of ) is, the less similar the prototypes u and u- are, in L proNCE The smaller the weight in the middle, the better the relationship between u and u. - The gaps are smaller; similar category prototype pairs are "difficult pairs", and vice versa, which is a proper way to handle "difficult pairs" and "easy pairs";

[0128] Step 4: Model Training

[0129] Step 4.1: Construct the overall loss function L using equation (13) overall ;

[0130]

[0131] In equation (13), λ1, λ2, and λ3 are three weight parameters, and γ is an adaptive parameter that increases with the number of iterations; where λ1 is set to 0.3, λ2 to 0.1, and λ3 to 1. α is set to 10, and p changes from 0 to 1 during training;

[0132] Step 4.2: Define the E-parameter of the feature extraction network as θ e The task-specific classifier C has parameters θ. c Graph convolutional parent matrix generator network G A Graph convolutional node aggregation network G N and graph volume integrator G c The parameter is θ g Thus, a parameter update strategy is constructed using equation (14):

[0133]

[0134] In equation (14), η represents the learning rate; the initial value of η is set to 0.01, and it is reduced to 0.001 during the training process using the annealing algorithm;

[0135] Step 4.3: Based on the domain-adaptive image dataset D = {D} s D t The contrastive learning domain adaptive network model is trained using stochastic gradient descent, and the model parameters are updated using Equation (14) until the overall loss function L is reached. overall The process continues until convergence, thus obtaining the optimal contrastive learning domain adaptive network model, which is used to classify images in the target domain.

[0136] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0137] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. An unsupervised domain adaptive image recognition method based on a prototype network contrastive learning, characterized in that, The method comprises the following steps: Step 1: Define the domain-adaptive image dataset as D = {D} s D t }, D s D represents the source domain image set containing category labels. t Let x represent the target domain image set without category labels, where x s Let y represent any labeled source image in the source image set. s x represents s Category labels, y s ∈{1, 2, ..., c}; c represents the total number of categories; (x s y s )∈D s Let n s Let x represent the total number of source domain images; t Represents the unlabeled target domain image, x t ∈D t Let n t This represents the total number of images in the target domain; and the source and target domains belong to the same label space. Step 2: Constructing a prototype-based contrastive learning domain adaptation network model for classifying target domain images x t performing image classification; The contrastive learning domain adaptation network model comprises: a feature extraction network E, a task-specific classifier C, a graph convolution affinity matrix generation network G A , a graph convolution node aggregation network G N , and a graph convolution classifier G c ; The convolutional layers and the fully connected layers in the contrast learning domain adaptation network model are initialized by xavier initialization; The training strategy of the contrast learning domain adaptation network model based on the prototype network comprises a domain bias prototype modeling strategy and a prototype level contrast learning strategy. Step 3.1: The domain bias prototype modeling strategy: Step 3.1.1: the source domain image x s and the target domain image x t are input into the feature extraction network E for processing, and the source domain feature f s and the target domain feature f t are obtained accordingly; The source domain feature f s is input into the task-specific classifier C for processing, and an output class probability prediction distribution p(y′ s | C(E(x s )) of the source domain image x s is obtained; wherein y′ s represents the predicted class label of x s . The cross-entropy loss function L for the task-specific classifier C is established using formula (1) ce : The target domain feature f t Inputting the task-specific classifier C for processing, and obtaining a target domain image x t The normalized probability output on the k-th category The target domain dataset D t The average normalized probability output on the k-th category The target domain mutual information loss L is established using formula (2) MI : Step 3.1.2: From the dataset D = {D} s D t Take a batch of data B = (B) from the data. s B t ), where B s B represents a batch of source domain images containing category labels. t This represents a batch of target domain images that do not contain category labels; A fully connected graph structure G=(V, A) of all images in B is constructed, wherein V represents a node set, and A represents an affinity matrix. The target domain feature or the source domain feature is used as the initial feature of any node in V. input the node pair (v i , v j ) between the i-th node v i and the j-th node v j into the graph convolution affinity matrix generation network G A for processing, and obtain the similarity score between the node pair (v i , v j ), and the element in the i-th row and the j-th column of the unnormalized affinity matrix , wherein σ(·) is an activation function; right After normalization, the normalized similarity score a is obtained. i，j And use it as the element in the i-th row and j-th column of A, thus obtaining the normalized affinity matrix. D is The degree matrix, where I is the identity matrix; Step 3.1.3: Based on the tag information, obtain the element t in the i-th row and j-th column of the real matrix T using formula (3) i，j : In equation (3), y i and y j For the i-th node v i and the j-th node v j The category label of the image; where, if the i-th node v i Or the j-th node v j The image belongs to B s Source domain image in Then y i or y j For source domain image Real category labels If the i-th node v i Or the j-th node v j The image belongs to B t Target domain image Then y i or y j For target domain image The pseudo-labels obtained after inputting into the task-specific classifier C. This indicates that the task-specific classifier C predicts the target domain image. The probability of the k-th category; Set a probability threshold τ, if Then delete and The corresponding target domain image The edges connecting the nodes make the affinity matrix A such that the nodes connected to the nodes are equal to the nodes in the affinity matrix A. The similarity score of the combined node pairs is set to zero; The graph convolution node aggregation network G is established using formula (4) A binary cross-entropy loss L of formula (3) bce : Step 3.1.4: Input all nodes in V into the graph convolution node aggregation network G N and update the node features in j using equation (5): In formula (5), f i represents the initial feature of the i-th node v i , f j represents the initial feature of the j-th node v j , and f represents the aggregated feature of the i-th node v i . establishing the graph convolution classifier G using formula (6) c cross-entropy loss function of In formula (6), representing source domain images aggregated features of the nodes belonging thereto; Step 3.1.5: Calculate the source domain bias prototype of the k-th category using formula (7) and formula (8) respectively and target domain bias prototype In formula (7) and formula (8), and respectively represent B s and B t all image sets of the k-th category, represent target domain images aggregated features of the belonging nodes; Step 3.1.6: The global bias prototype of the k-th category is updated by using the exponential moving average strategy shown in formula (9) and formula (10): In formula (9) and formula (10), <- represents assignment, and p represents a balance parameter. and respectively represent the source domain global bias prototype and the target domain global bias prototype of the k-th category in the I-1-th iteration; and respectively represent the source domain bias prototype and the target domain bias prototype of the k-th category in the I-th iteration, and when I = 1, let Step 3.2: Constructing the loss function L for the prototype-level contrastive learning strategy with formula (12) proNCE : In formula (12), u represents a bias prototype with a category label k, u + is a unique positive sample for u, represents a unique bias prototype with a category k in another domain, u - is a negative sample, represents a bias prototype of other categories in two domains; N is a set of all bias prototypes, N - represents a set of all negative samples relative to u, w(·, ·) is a weight function; φ(·, ·) is a distance metric function; Step 4: Model training: Step 4.1 : Constructing the overall loss function L with formula (13) overall : In formula (13), λ1, λ2 and λ3 are three weight parameters, and γ is an adaptive parameter that increases with the number of iterations. Step 4.2: Define the parameters of the feature extraction network E as θ e , the parameters of the task-specific classifier C as θ c , the graph convolutional affinity matrix generation network G A , the graph convolutional node aggregation network G N , and the graph convolutional classifier G c as θ g , so as to construct the parameter update strategy by using formula (14): In formula (14), η represents a learning rate. Step 4.3: training the contrastive learning domain adaptation network model based on the domain adaptive image dataset D = {D s , D t} using a random gradient descent manner and updating the model parameters using formula (14) until the overall loss function L overall converges, thereby obtaining an optimal contrastive learning domain adaptation network model for realizing classification of target domain images. Step 4.4: using the optimal contrastive learning domain adaptation network model to classify the target domain images.

2. The unsupervised domain adaptive image recognition method based on a prototype network contrast learning according to claim 1, characterized in that, The feature extraction network E in the step 2 is based on the ResNet-50 structure and comprises an input convolutional layer, a maximum pooling layer, M down-sampling convolutional layers, Q residual convolutional blocks and an average pooling layer in sequence; and a batch normalization layer and a ReLU activation function are connected after each convolutional layer. The task-specific classifier C is composed of H fully connected networks, and a batch normalization layer and a ReLU activation function are connected after the first fully connected network. The graph convolution affinity matrix generation network G A and the graph convolution node aggregation network G N Each of the N-layer convolution layers has a convolution kernel of 1*1, and is followed by a batch normalization layer and a LeakyRuLe activation function. The graph convolution classifier G c consists of a convolution layer with a 1x1 convolution kernel and a batch normalization layer.

3. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the unsupervised domain adaptation image recognition method of claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the unsupervised domain adaptation image recognition method of claim 1 or 2.

Images