A domain generalization method based on graph prototype network and instance contrast

Abstract

For the classification prediction problem of images distributed in unknown domains, the mismatched distribution of source data set and target data set will cause the performance of source model to decrease significantly in target domain, and the distribution alignment method for cross-domain visual representation proposed at present does not consider the difference of internal data structure of cross-domain. The application constructs a densely connected instance graph by using the CNN features of samples through the similarity of sample structure features. Each node corresponds to the CNN feature of the sample, and the feature is extracted by a standard convolutional network. Then, the graph convolution network is applied to the instance graph, and the graph structure information is propagated along the designed network to update the edge of the weighted graph. The application uses class mean value to construct class prototype for classification, and also considers the comparative supervision learning of instance nodes to learn the class semantic information on the instance nodes. In order to better learn and reduce the difference of class semantic information between domains, the application adopts soft label for knowledge distillation between domains.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of machine learning domain generalization, and particularly relates to a domain generalization method based on a graph prototype network and instance comparison. BACKGROUND

[0002] Generally, most machine learning models are first trained on a source domain dataset, and then the training results are predicted on a target domain dataset, where it is often implicitly assumed that the source domain dataset and the target domain dataset both follow the same distribution. However, such an assumption often cannot be established in the real world. For example, for multi-domain images collected under different angles, devices, environments, etc., the classifier obtained by training in one domain does not work well in other domains. Here, the process of transferring the knowledge of a certain domain to other unseen domains is called domain generalization. In transfer learning, the difficulty of the domain generalization problem mainly comes from two aspects, one is the distribution difference of different source datasets, and the other is the unknowability of the target domain. Domain generalization aims to directly extend the model trained on the source domain dataset to the unseen target domain with different distributions without retraining or fine-tuning on the target domain dataset. Domain generalization solves the classification prediction problem for images distributed in unknown domains. The mismatched distribution of the source dataset and the target dataset will lead to a significant decline in the performance of the source model in the target domain. The current distribution alignment methods for cross-domain visual representation do not consider the difference in the internal data structure of the cross-domain, and are subject to insufficient alignment of cross-domain representation. For example, the deep adversarial adaptive method only forces the alignment of the global distribution, but may lose the key semantic class label information of each class, and must be supervised learning using domain labels in training. Even with perfect confusion alignment, it cannot guarantee the adjacent mapping of non-same-domain samples with the same class label in the feature space. However, for traditional methods related to distribution alignment of data structure, although the distribution difference between the domains can be reduced and the original space attributes can be preserved, it is difficult to effectively model the data structure information and integrate it into the existing deep network. SUMMARY

[0003] The overall framework of the image classification method based on the image prototype network and the case comparison network is as follows: Figure 1As shown. In order to model the data structure under the deep network, the CNN features of the samples are used to construct a densely connected instance graph through the similarity of the sample structure features. Each node corresponds to the CNN feature of the sample, which is extracted by a standard convolutional network (such as ResNet). Then, a graph convolutional network (GCN) is applied to the instance graph, and the graph structure information is propagated along the designed network to update the edge of the weighted graph. On the one hand, the class mean value is used to construct the class prototype for classification; on the other hand, the comparative supervision learning of the instance node is also considered to learn the class semantic information on the instance node. At the same time, in order to better learn and reduce the difference of class semantic information between domains, soft labels (logit) are used for knowledge distillation between domains, that is, to narrow the Kullback-Leibler (KL) divergence. Knowledge distillation matches the prediction distribution set of data with the same class label but different domains with each prediction distribution, increases the entropy of the model prediction by using the set penalty of meaningful errors accumulated by multiple domains to predict the sample, and encourages the model to converge to a wide local minimum. The specific steps of the domain generalization method based on the graph prototype network and the instance comparison proposed by the present application are as follows:

[0004] Step 1: Obtain image samples and their labels, and construct an image feature extraction model;

[0005] Obtain image samples to construct an initial image dataset, divide the initial image dataset into a source domain dataset M = {M1,..., M i ,..., M m} and a target domain dataset T, wherein M m represents the mth domain dataset; the target domain dataset is inaccessible in the training process of the image feature extraction model;

[0006] The source domain dataset M is divided into a training set and a validation set, and the images in the source domain dataset M are data enhanced;

[0007] Obtain a pre-trained model, and construct the image feature extraction model based on the pre-trained model; extract the features in the source domain dataset M as graph input features X through the image feature extraction model;

[0008] Step 2: Establish a graph convolutional network and obtain a class prototype representation;

[0009] The graph structure information of the features extracted from the source domain dataset M is defined as G = <V, E, Z>, wherein V = {v1,..., v n} is a set of n nodes, is a node feature obtained by extracting two GCN layers, E = {e 11 ,..., e ij ,..., e nn} represents the distance between nodes; where cosine similarity is used. This represents the distance between node i and node j;

[0010] Construct an undirected graph adjacency matrix A containing n nodes using the distance E between nodes, and then transform the undirected graph adjacency matrix A. in, For degree matrix, j is the neighboring node number of node i;

[0011] Construct a normalized adjacency matrix based on the similarity between nodes. Where I is the identity matrix;

[0012] Given an adjacency matrix A∈R of an undirected graph containing n nodes n×n The linear transformation of graph convolution depends on the graph input features X∈R. k×n With filter W∈R k×d ;

[0013] Among them, the column vector X in the graph input feature X i ∈R k d represents the feature representation of the i-th node in the set of nodes V, and d represents the output feature dimension.

[0014] The embedded features are obtained by performing two layers of GCN processing as shown in the following formula.

[0015]

[0016] Where σ is the activation function, This represents the output of the i-th node at layer l. It is a graph convolution input;

[0017] Then, the embedded features generated by the graph convolutional network are used. Computation class prototype P∈R c×d The expression, This represents the output of the i-th node in the m-th source domain of the graph convolutional network;

[0018] The class prototype is defined as being tightly surrounded by nodes of the same class, so that nodes of the same class can represent their own class; the prototype of the c-th class in the m-th source domain. Calculated in the following way:

[0019]

[0020] Where PROTP is the way to represent the prototype P of the computed class. It is the representation of the i-th node in the m-th source domain, where m c For the c-th class of the m-th source domain, vi For the i-th node of the c-th class of the m-th domain, the specific formula is as follows:

[0021]

[0022] When the prototype distribution is not unimodal, the class representation is insufficient. At this time, each class can be represented by multiple prototypes, and the mean value (such as K-means clustering) is replaced by a permutation invariant function. For the sake of simplicity, the calculation is performed according to the average value.

[0023] The node is projected from the original embedding space to another distance space to learn a distance metric representation;

[0024] Step 3. Classification is performed by comparing the distance metric representation of the node with the distance metric representation of the class prototype;

[0025] The distance metric loss is calculated:

[0026] The embedding node learned by the graph convolution The distance metric representation of each node to each class prototype is calculated:

[0027]

[0028] Wherein, is the embedding difference between each node and each class prototype in the m-th source domain;

[0029] The node embedding difference is associated with all class prototypes, and a linear transformation f is applied to the different dimensions of the embedding difference to give different degrees of attention, while adaptively extracting embedding difference information, as shown in the following formula:

[0030]

[0031] The distance metric representation g represents the distance information of the node v to all class prototypes, which is used to define the relative position of the node and all class prototypes in the m-th source domain, and c C represents the c-th class; The distance metric representation is connected through the connection layer concat to calculate the distance metric representation of the class prototype and the node in all source domains M as follows:

[0032] G = concat(g 1 , …, g m ) (6)

[0033] Then the value of the softmax of the i-th node v i is calculated:

[0034]

[0035] where denotes the node v i distance metric for class c, P(y = c|v i ) gives the node v i predicted probability distribution for class c, the cross-entropy classification loss is constructed as follows:

[0036]

[0037] Step 4: Embedding space instance node supervised contrastive learning;

[0038] The domain supervision contrastive loss is constructed as follows:

[0039]

[0040] where I is the set of all mini-batch samples, i is an anchor point, p e P(i) is a positive sample in I that is the same as the i-th sample, |P(i)| is the number of samples in the set P(i), is a negative sample that is different from the i-th sample and is in the same domain as the i-th sample, denotes the features of the l2 regularization on the graph convolution output, τ denotes the temperature parameter, f denotes the similarity measure between different nodes, and samples with the same class label are represented and are close to each other, while samples with different class labels are represented and are far away from each other;

[0041] Step 5: Domain invariance knowledge distillation;

[0042] The complementary knowledge between the node information from different domains is learned through domain invariance knowledge distillation as follows, where X c denotes the set of all samples with the same class label c from each domain; the corresponding soft label value is obtained by taking the average of X c :

[0043]

[0044] where h is the learned node embedding from the last layer of the GNN encoder, denotes the GNN output of the i-th node;

[0045] The prediction distribution softmax function from X c is defined as:

[0046]

[0047] Computing inter-domain knowledge distillation, defining domain loss function as L kd : minimizing the difference in the semantic level between domains by KL divergence;

[0048]

[0049] Wherein M represents a set of domains, m represents the number of domains, represents the image classification probability of the i-th node in the m-th domain, τ represents a temperature parameter; D kl represents the KL divergence, which is used to calculate the difference between the output distribution of the node and the mean;

[0050] Step 6: define the total target loss function as follows:

[0051]

[0052] Wherein γ and are weight factors;

[0053] The graph convolutional network is trained by the total target loss function, and the field generalization is realized.

[0054] As preferred, in the step 1, the data enhancement mode includes cutting and reversing.

[0055] As preferred, in the step 1, the pre-training model is any one of ResNet or AlexNet model.

[0056] As preferred, in the step 2, the graph input feature X is obtained by a feature extractor ResNet, and the activation function of the GCN adopts ReLu.

[0057] Advantages of the present application

[0058] The present application proposes a domain generalization method based on graph prototype network and instance comparison. Compared with the prior art, the method not only emphasizes the whole global inter-domain prototype classification, but also combines the structured information of the internal sample, and models the data structure information. By constructing the domain-invariant class prototype classification and instance node supervised comparison, the margin between different classes is maximized, and the difference between different classes in the target domain is explicitly maximized. Not only can the difference between different domains be relieved, but also the class semantic information can be retained. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the drawings required to be used in the description of the specific embodiments or the prior art will be briefly introduced below, and some specific embodiments of the present application will be described in detail below with reference to the drawings in an exemplary but non-limiting manner. The same reference signs in the drawings indicate the same or similar components or parts. Those skilled in the art should understand that these drawings are not necessarily drawn to scale.

[0060] In the drawings:

[0061] Figure 1 is a schematic diagram of the principle of the image classification method of the present application based on a graph prototype network and an instance comparison network.

[0062] Figure 2 is a flowchart of class prototype classification by a graph prototype network. DETAILED DESCRIPTION

[0063] In order to make the purposes, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0064] Step 1: Obtain image samples and their labels, and construct an image feature extraction model;

[0065] As shown in Figure 2 , obtain image samples to construct an initial image dataset, divide the initial image dataset into a source domain dataset M = {M1,..., M i ,...,M m} and a target domain dataset T, where M m represents the mth domain dataset; the target domain dataset is inaccessible in the training process of the image feature extraction model;

[0066] The source domain dataset M is divided into a training set and a validation set, and the images in the source domain dataset M are data-augmented, and the data-augmentation methods include cropping and reversing;

[0067] Obtain a pre-trained model, and construct the image feature extraction model based on the pre-trained model; the pre-trained model is any one of ResNet or AlexNet model;

[0068] Extract the features in the source domain dataset M as graph input features X through the image feature extraction model;

[0069] Step 2: Establish a graph convolution network and obtain a class prototype representation;

[0070] The graph structure information of the extracted features from the source domain dataset M is defined as G =<V,E,Z> Where V = {v1, ..., v} n} is a set of n nodes. The node features are obtained through two GCN layers, E = {e 11 , ..., e ij , ..., e nn} represents the distance between nodes; where cosine similarity is used. This represents the distance between node i and node j;

[0071] Construct an undirected graph adjacency matrix A containing n nodes using the distance E between nodes, and then transform the undirected graph adjacency matrix A. in, For degree matrix, j is the neighboring node number of node i;

[0072] Construct a normalized adjacency matrix based on the similarity between nodes. Where I is the identity matrix;

[0073] Given an adjacency matrix A∈R of an undirected graph containing n nodes n×n The linear transformation of graph convolution depends on the graph input information X∈R. k×n With filter W∈R k×d ;

[0074] Among them, the column vector X in the graph input feature X i ∈R k d represents the feature representation of the i-th node in the set of nodes V, and d represents the output feature dimension.

[0075] The embedded features are obtained by performing two layers of GCN processing as shown in the following formula.

[0076]

[0077] Where σ is the activation function, This represents the output of the i-th node at layer l. X is the graph convolution input. Furthermore, the node features X are obtained by the feature extractor ResNet, and the activation function of the GCN is ReLU.

[0078] Then, the embedded features generated by the graph convolutional network are used. Computation class prototype P∈R C×d The expression, This represents the output of the i-th node in the m-th source domain of the graph convolutional network;

[0079] The definition of the class prototype is that it is tightly surrounded by the nodes of the same class, so that the nodes of the same class can represent their class. The prototype of the c-th class of the m-th source domain The distance metric loss is calculated as follows:

[0080]

[0081] where PROTP is the way to calculate the representation of the class prototype P, is the representation of the i-th node in the m-th source domain, m c is the c-th class of the m-th source domain, v i is the i-th node of the c-th class of the m-th domain, and the specific formula is as follows:

[0082]

[0083] In general, when calculating the class prototype, it is assumed that only one prototype is used to represent each class, but when the prototype distribution is not unimodal, such class representation is insufficient. At this time, each class can be represented by multiple prototypes, and the mean value is replaced by the permutation invariant function (such as K-means clustering). For the sake of simplicity, the calculation is carried out according to the average value;

[0084] Since the prototype is the representation of each class, the nodes can be classified by selecting the nearest prototype. However, directly embedding the average vector of the class into the node as the prototype may not provide the expected results. Therefore, instead of directly classifying the nodes based on their nearest prototypes, the nodes can be projected from the original embedding space to another distance space to learn a distance metric representation;

[0085] Step 3: Classification by comparing the distance metric representation of the nodes with the distance metric representation of the class prototypes;

[0086] The distance metric loss is calculated as follows:

[0087] The embedding nodes generated by the nodes learned by the graph convolution The distance metric representation of each node to each class prototype is calculated as follows:

[0088]

[0089] where, is the embedding difference between each node and each class prototype in the m-th source domain; the embedding difference is associated with all class prototypes, and a linear transformation f is applied to the embedding difference in different dimensions to give different degrees of attention to different dimensions, while adaptively extracting useful embedding difference information, as follows:

[0090]

[0091] The distance metric representation g represents the distance information from node v to all class prototypes, defining the relative positions of nodes in the m-th source domain with respect to all class prototypes, where c ∈ C represents the c-th class. The distance metric representations are concatenated using a concat layer as shown in the following formula to compute the distance metric representations between class prototypes and nodes in all source domains M:

[0092] G = concat(g) 1 , ..., g m (6)

[0093] Then calculate the i-th node v i The softmax value:

[0094]

[0095] in This indicates that node v is in the entire source domain. i For class c, the distance metric is P(y=c|v i The node v is given. i For the predicted probability distribution of class c, the cross-entropy classification loss is constructed as shown in the following formula:

[0096]

[0097] Step 4: Supervised comparative learning of embedded spatial instance nodes;

[0098] The previous classification loss prototype only considered the centroid of each class, ignoring variations within each class. In contrast, instance comparison between samples incorporates more fine-grained variations in instance node features within the embedded network. To ensure domain invariance while increasing the separability of class information, an instance-domain supervised contrastive loss was calculated. Traditional supervised contrastive loss only considers the positive and negative sets of class labels, but in domain generalization, it fails to account for the influence of multiple domains. Specifically, positive and negative samples from some domains may lead to incorrect judgments of positive samples from other domains. This could be because domain information in the samples carries a disproportionate weight compared to class information, thus excluding samples from different domains from the negative set.

[0099]

[0100] Where I is the set of all mini-batch samples, i is an anchor point, p∈P(i) is a positive sample in I that is the same as the i-th sample, and |P(i)| is the number of samples in set P(i). It is a negative sample that is different from the i-th sample and belongs to the same domain as the i-th sample. τ represents the feature of L2 regularization of the graph convolution output, τ represents the temperature parameter, and f represents the similarity measure of different nodes;

[0101] Sample representation of labels of the same category and Samples are close to each other, while those with different category labels are represented. and They are distancing themselves from each other. When contrastive loss supervision is directly used for domain generalization tasks, performance degrades. Specifically, the feature space becomes domain-discriminatory because positive samples from different domains are pushed away from the anchor point due to the influence of the domain. To make the feature space more suitable for domain generalization, a new contrastive loss supervision is proposed, which enables the feature extractor to not only generate features that are discriminative to the class label, but also to better extract domain-invariant features by attracting positive samples from different domains.

[0102] Step 5: Distillation of domain-invariant knowledge;

[0103] Since the feature extractors trained using class prototypes and instance node contrastive learning do not directly consider the gap between domains—class prototypes being averaged vectors from multi-domain samples, and instance node contrastive learning removing negative samples from outside the target domain—domain-invariant distillation is introduced to narrow the gap between the source and target domains. The purpose of domain-invariant distillation is to use node information from different domains to transfer complementary knowledge. In a mini-batch of samples, a set is generated by averaging multiple soft label values ​​with the same class label and relying on more complementary knowledge. Because different domains exhibit different inter-class relationships, the information provided by each sample, either itself or its specific domain, helps construct complementary knowledge, which can also be used for supervision to learn domain-invariant information. c This represents the set of all samples from various domains that have the same class label c. By analyzing X... c The corresponding soft tag value is obtained by averaging:

[0104]

[0105] Where h is the learned node embedding from the last layer of the GNN encoder. This represents the GNN output of the i-th node;

[0106] Definition from X c The softmax function for the predicted distribution is:

[0107]

[0108] Computational inter-domain knowledge distillation, defining the domain loss function as L kd Minimize semantic differences between domains using KL divergence;

[0109]

[0110] Where M represents the set of fields, and m represents the number of fields. D represents the image classification probability of the i-th node in the m-th domain, and τ represents the temperature parameter; kl This represents the KL divergence, used to calculate the difference between the output distribution of nodes and the mean.

[0111] Knowledge distillation was originally a model compression method that trains smaller "student" models to simulate a large "teacher" model by minimizing the classification loss of the output class probabilities (soft labels) of a large model. For domain generalization, it can be used to generate an ensemble teacher model by averaging samples with the same class labels, reducing inter-domain differences by minimizing the class probability distribution within a specific domain. Soft labels provide more semantic information about images. For example, given images in CIFAR-10, the probability that a dog image is a cat is much higher than the probability that it is a car. Therefore, soft labels provide the network with additional semantic information about the similarity between dog and cat images. Knowledge distillation also improves the loss condition and helps find the flat minimum in the network, thus improving generalization ability. Knowledge distillation has been shown to amplify regularization in Hilbert spaces, enhancing generalization capabilities.

[0112] Step 6: Define the overall objective loss function as follows;

[0113]

[0114] γ and γL is the weighting factor. The overall training is performed end-to-end, where the last two terms γL... kd and Calculated only during training. L class It is a prototype-supervised classification loss obtained through prototype learning of GNN. L kd This is domain-invariant knowledge distillation on the classifier, reflecting the differences in distribution between domains. L instance It is a domain-supervised contrastive loss that perceives instance category information through feature-level image instance node network structure analysis.

[0115] Example:

[0116] To model the data structure in deep networks, samples are first extracted using standard convolutional networks (e.g., ResNet). Then, graph convolutional networks (GCNs) are used to obtain node features in the embedding space. Subsequently, class prototypes are constructed using these node features, and classification is performed based on these prototypes. Simultaneously, contrastive supervised learning is used to extract category-semantic information from the node features in the embedding space. Furthermore, to reduce the differences in data distribution across multiple domains, soft labels are used to reduce KL divergence, and domain-invariant knowledge distillation is performed.

[0117] 1) Assume the source dataset includes three datasets: M1, M2, and M3, with each domain containing k images. Input image sample data from multiple domains. Constructing the graph structure information of samples from the source domain First, the adjacency matrix A of an undirected graph containing n nodes can be transformed into a more expressive form. For degree matrix, (where j is the index of all neighboring nodes of node i). Based on the similarity between nodes, construct a normalized adjacency matrix. (I is the identity matrix). Embedded features are obtained by performing a two-layer GCN transformation using formula (1). Then, the class prototypes of each domain are calculated using formula (2,3).

[0118] 2) Then embed node h∈R into the given node v. d The path from each node to each class prototype g∈R is calculated using formula (4). d Distance metric representation Formula (5) links the node embedding differences to all class prototypes, obtaining the distance information from the node to all class prototypes, thus precisely defining the relative position of the node to all class prototypes. Formulas (6,7,8) connect class prototypes through a connection layer and calculate the cross-entropy loss.

[0119] 3) Simultaneously, in the embedding space obtained through two layers of GCN, formula (9) calculates the instance node supervised contrast loss in the embedding space, making the features of samples with the same category label close to each other, while the features of different samples are far apart, alleviating the differences in distribution between different domains, and also preserving the semantic information of the category. The instance node supervised contrast loss considers the changes within the category on the one hand, and the influence of multiple domains on the other hand, where positive and negative samples in some domains may cause incorrect judgments of positive samples in other domains.

[0120] 4) In order to reduce the difference in domain distribution between domains, formulas (10,11,12) learn complementary knowledge between node information from different domains through domain-invariant knowledge distillation, thereby alleviating the difference in distribution between different domains.

[0121] 5) Finally, formula (13) defines the total objective loss function.

[0122] The above description is only a part of the specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A domain generalization method based on graph prototype networks and instance comparison, characterized in that, Includes the following steps: Step 1: Obtain image samples and their labels, and build an image feature extraction model; Obtain image samples to construct an initial image dataset, and divide the initial image dataset into source domain datasets M = {M1, ... M...} i M m } and target domain dataset T, where M m Let m represent the target domain dataset; the target domain dataset is inaccessible during the training of the image feature extraction model. The source domain dataset M is divided into a training set and a validation set, and the images in the source domain dataset M are augmented. Obtain a pre-trained model, and construct the image feature extraction model based on the pre-trained model; The image feature extraction model extracts features from the source domain dataset M, which are then used as graph input features X. Step 2: Build a graph convolutional network and obtain class prototype representations; The graph structure information of the extracted features from the source domain dataset M is defined as G =<V,E,Z> Where V = {v1, ..., v} n } is a set of n nodes. The node features are obtained through two GCN layers, E = {e 11 , ..., e ij , ..., e nn } represents the distance between nodes; where cosine similarity is used. This represents the distance between node i and node j; Construct an undirected graph adjacency matrix A containing n nodes using the distance E between nodes, and then transform the undirected graph adjacency matrix A. in, For degree matrix, j is the neighboring node number of node i; Construct a normalized adjacency matrix based on the similarity between nodes. Where I is the identity matrix; Given an adjacency matrix A∈R of an undirected graph containing n nodes n×n The linear transformation of graph convolution depends on the graph input features X∈R. k×n With filter W∈R k×d ; Among them, the column vector X in the graph input feature X i ∈R k d represents the feature representation of the i-th node in the set of nodes V, and d represents the output feature dimension. The embedded features are obtained by performing two layers of GCN processing as shown in the following formula. Where σ is the activation function, This represents the output of the i-th node at layer l. It is a graph convolution input; Then, the embedded features generated by the graph convolutional network are used. Computation class prototype P∈R c×d The expression, This represents the output of the i-th node in the m-th source domain of the graph convolutional network; The class prototype is defined as being tightly surrounded by nodes of the same class, so that nodes of the same class can represent their own class; the prototype of the c-th class in the m-th source domain. Calculated in the following way: Where PROTP is the way to represent the prototype P of the computed class. It is the representation of the i-th node in the m-th source domain, where m c For the c-th class of the m-th source domain, v i Let be the i-th node of class c in the m-th domain, and the specific formula is as follows: A distance metric representation is learned by projecting the nodes from the original embedding space to another distance space. Step 3: Classify by comparing the learned distance metric representation of nodes with the distance metric representation of class prototypes; Calculate the distance metric loss: Embedded nodes learned from graph convolution Calculate the relationship between each node and each class prototype. Distance metric representation: in, Embed the differences between each node and each class prototype in the m-th source domain; The node embedding differences are linked to all class prototypes, and a linear transformation f is applied to give different degrees of attention to different dimensions of the embedding differences in the manner shown in Equation (5), while the embedding difference information is adaptively extracted, as shown in the following equation: The distance metric representation g represents the distance information from node v to all class prototypes, defining the relative positions of nodes in the m-th source domain with all class prototypes, where c∈C represents the c-th class. The distance metric representations are concatenated through a concat layer as shown in the following formula to compute the distance metric representations of class prototypes and nodes in all source domains M: G=concat(g 1 ,…,g m ) (6) Then calculate the i-th node v i The softmax value: in This indicates that node v is in the entire source domain. i For class c, the distance metric is P(y=c|v i The node v is given. i For the predicted probability distribution of class c, the cross-entropy classification loss is constructed as shown in the following formula: Step 4: Supervised comparative learning of embedded spatial instance nodes; Construct the domain-supervised contrastive loss as shown in the following formula: Where I is the set of all mini-batch samples, i is an anchor point, p∈P(i) are positive samples in I that are identical to the i-th sample, and |P(i)| is the number of samples in set P(i). It is a negative sample that is different from the i-th sample and belongs to the same domain as the i-th sample. The features represented by L2 regularization of the graph convolution output are denoted by τ, where τ represents the temperature parameter, f represents the similarity measure between different nodes, and samples with the same class label are represented by τ. and Samples are close to each other, while those with different category labels are represented. and They are far apart; Step 5: Distillation of domain-invariant knowledge; As shown in the following formula, complementary knowledge between node information from different domains is learned through domain-invariant knowledge distillation, where X c This represents the set of all samples from various domains that have the same class label c; through X c The corresponding soft tag value is obtained by averaging: Where h is the learned node embedding from the last layer of the GNN encoder. This represents the GNN output of the i-th node; Definition from X c The softmax function for the predicted distribution is: Computational inter-domain knowledge distillation, defining the domain loss function as L kd Minimize semantic differences between domains using KL divergence; Where M represents the set of fields, and m represents the number of fields. D represents the image classification probability of the i-th node in the m-th domain, and τ represents the temperature parameter; kl This represents the KL divergence, used to calculate the difference between the output distribution of nodes and the mean. Step 6: Define the overall objective loss function as follows; Wherein, γ and As a weighting factor; By training the graph convolutional network using the overall objective loss function, domain generalization can be achieved.

2. The domain generalization method based on graph prototype networks and instance comparison as described in claim 1, characterized in that, In step 1, the data augmentation methods include cropping and reversing.

3. The domain generalization method based on graph prototype networks and instance comparison as described in claim 1, characterized in that, In step 1, the pre-trained model is either ResNet or AlexNet.

4. The domain generalization method based on graph prototype networks and instance comparison as described in claim 1, characterized in that, In step 2, the graph input feature X is obtained by the feature extractor ResNet, and the activation function of the GCN is ReLU.

Images