A cross-domain semantic segmentation method based on graph convolution and contrastive learning

By using graph convolution and contrast learning in semantic segmentation method, the dual-domain adjacency matrix and group comparison loss function are constructed, which solves the problem of category distribution imbalance in cross-domain transfer learning, improves the transfer effect, and achieves a better cross-border Domain semantic segmentation performance.

CN113903031BActive Publication Date: 2024-11-19KUNMING UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111069488.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-13
Publication Date
2024-11-19
Estimated Expiration
2041-09-13

AI Technical Summary

Technical Problem

The existing semantic segmentation method has the problem of imbalance in class distribution in cross-domain transfer learning, which makes it difficult to transfer distributions with small categories, and has poor cross-domain effects.

Method used

Using cross-domain semantic segmentation method based on graph convolution and contrast learning, graph convolution operations are carried out by constructing a two-domain adjacency matrix, the context relationship between domains is established, and the idea of ​​group comparison loss function and entropy value minimization is extracted. The characteristics of the change properties are constructed and the pseudo-labels are self-trained.

Benefits of technology

It effectively improves the class imbalance problem in the semantic segmentation dataset, extracts domain invariant information in the image, improves the effect of cross-domain transfer learning, and achieves better evaluation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113903031B_ABST
    Figure CN113903031B_ABST
Patent Text Reader

Abstract

The present invention provides a cross-domain semantic segmentation method based on graph convolution and contrastive learning, which belongs to the field of transfer learning and computer vision. The present invention designs a method for finding adjacency matrices in different domains, which provides a new idea for establishing long-distance contextual relationships between domains. In order to solve the problem of unbalanced distribution of different categories, a group contrastive learning loss is proposed. Secondly, in order to extract domain-invariant information, the present invention uses the newly proposed dual-domain adjacency matrix to perform graph convolution operations. In the process of performing graph convolution operations, the present invention constructs a graph structure on a feature graph. In order to establish long-distance contextual relationships between domains, the present invention creatively uses the proposed dual-domain adjacency matrix in the proposed graph structure to complete the graph convolution operation. The method proposed in the present invention establishes long-distance contextual relationships between domains, and can more effectively extract domain-invariant information, and obtains better evaluation results in subjective and objective evaluations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention designs a cross-domain semantic segmentation method based on graph convolution and contrastive learning, which belongs to the field of pattern recognition and computer vision. Background Art

[0002] The basic task of semantic segmentation is to correctly classify each pixel in the image. Since semantic segmentation is a pixel-level classification task, it is very helpful for scene understanding tasks. Therefore, semantic segmentation tasks have broad development prospects in tasks such as autonomous driving, medical image segmentation, and scene recognition.

[0003] Since the advent of the fully convolutional network, convolutional neural networks have made great progress in the field of semantic segmentation. However, since the semantic segmentation task requires pixel-level labels during training, and labeling pixel-level labels requires a lot of manpower and material resources, people consider using synthetic datasets to train semantic segmentation networks. However, due to the domain difference between synthetic datasets and real-world datasets, the performance of the network trained using synthetic datasets in real-world datasets is not satisfactory and cannot be directly applied to real-world scenarios. In order to solve this problem, unsupervised domain adaptation methods came into being.

[0004] Some recent methods use adversarial thinking and exploit adversarial learning between the discriminator and the generator to extract domain-invariant information. Although these works have made some progress, none of them consider how to exploit the contextual relationship between domains. Some other methods use the entropy value of the target domain prediction graph to propose a cross-domain method based on entropy minimization. However, there is a flaw in the entropy minimization method: categories with high predicted probability values ​​will have a larger gradient in the entropy loss function, so the network will be more inclined to migrate simple samples and ignore difficult samples. Since the distribution of classes in semantic segmentation datasets has a long tail effect, it is difficult to migrate distributions with a small proportion of categories (difficult samples), and the cross-domain effect is relatively poor. Summary of the invention

[0005] The present invention provides a cross-domain semantic segmentation method based on graph convolution and contrastive learning, which is used to solve the problems of unbalanced distribution of different categories, difficulty in migrating distribution of categories with a small proportion, and poor cross-domain effect in existing segmentation methods. The present invention achieves the goal of transfer learning.

[0006] The technical solution of the present invention is: a cross-domain semantic segmentation method based on graph convolution and contrastive learning, and the specific steps of the method are as follows:

[0007] S1: Select a network training dataset, wherein the network training dataset includes a source domain dataset and a target domain dataset, wherein the source domain dataset contains labels and the target domain dataset does not contain labels; the target domain dataset includes a training dataset and a test set;

[0008] S2: Construct a basic network model for semantic segmentation. First, use the source domain dataset to train the basic network model for semantic segmentation. Use the basic network model for semantic segmentation trained with the source domain dataset as the basic network. Send the source domain dataset and the target domain dataset together to the basic network for training.

[0009] S3: Use the feature graph output by the middle layer of the network to construct a dual-domain adjacency matrix, which is a position adjacency matrix and a channel adjacency matrix. Use the constructed source domain position adjacency matrix and source domain channel adjacency matrix to perform graph convolution operations on the graph constructed by the source domain feature graph. Use the constructed target domain position adjacency matrix and target domain channel adjacency matrix to perform graph convolution operations on the graph constructed by the target domain feature graph.

[0010] S4: Input the source domain feature map after the graph convolution operation into the source domain classifier to obtain the prediction probability map of the source domain image, and input the target domain feature map after the graph convolution operation into the target domain classifier to obtain the prediction probability map of the target domain image;

[0011] S5: Extract domain-invariant features using group contrastive learning: Use the group contrastive loss function to narrow the distance between the same classes in the domains and widen the distance between different classes in the domains, forcing the generator to extract features with domain-invariant properties.

[0012] S6: Use the idea of ​​entropy minimization to screen the prediction confidence of the target domain output prediction map, construct pseudo labels, and use the forged pseudo labels to self-train the network.

[0013] As a further solution of the present invention, the source domain dataset is a GTA5 dataset, the target domain dataset is a Cityscapes dataset, and the test set is a test set part of the target domain dataset.

[0014] As a further solution of the present invention, the source domain dataset is GTA5, which includes 24,966 pictures for training to synthesize street scene images and corresponding label images, and the target domain dataset is Cityscapes, which includes 5,000 pictures of 50 different city street scenes for training, real street scene pictures, and 500 real street scene pictures for testing.

[0015] As a further solution of the present invention, the implementation details of constructing the dual-domain adjacency matrix in S3 are as follows:

[0016] (1) First, we need to define a graph in the semantic segmentation task: Due to the characteristics of the graph structure itself, long-distance contextual relationships can be realized in a single-layer graph convolution operation. The graph consists of nodes and edges, and the graph convolution formula is defined as:

[0017]

[0018] In the formula, X is the graph before graph convolution, is the graph obtained after the graph convolution operation, σ() is the nonlinear activation function, A is the adjacency matrix of the graph, and W is the weight of the graph convolution;

[0019] In the graph, the adjacency matrix represents the connection relationship between the graph nodes, and the adjacency matrix is ​​normalized into a Laplace matrix In order to introduce the characteristics of the node itself during the node update process, the Laplacian matrix is ​​written as Where I is the identity matrix, the normalized Laplace matrix Denoted as A, the graph convolution formula is written as:

[0020] H (l+1) =σ(AH (l) W (l) )

[0021] In the formula, H (l) is the graph of layer l, H (l+1) Yes H (l) The graph obtained after graph convolution;

[0022] (2) After defining the graph structure in the semantic segmentation task, we need to construct a dual-domain position adjacency matrix:

[0023] Assume that the source domain feature map is Where D is the number of channels in the feature map, N = H × W is the number of channels in the feature map, Figure 1 The number of pixels on the channel; the target domain feature map is Where D is the number of channels in the feature map, N = H × W is the number of channels in the feature map, Figure 1 The number of pixels on the channel; using X s and Y T The similarity matrix of the dual domains, i.e., the position adjacency matrix, is constructed. The newly constructed adjacency matrix is ​​used for graph convolution to update the information between nodes and aggregate the feature information on the nodes to capture the long-distance contextual relationship. The similarity matrix is ​​constructed on the feature map output by the network. In cross-domain, in order not to lose the semantic information of the image, the diagonal matrix of the input feature map is introduced in the process of constructing the adjacency matrix of the dual domains, so that the adjacency matrix is ​​associated with the input:

[0024]

[0025]

[0026] in the formula and are the position adjacency matrices of the source domain and the target domain respectively; i is the pixel point in the intermediate feature map of the source domain, j is the pixel point in the intermediate feature map of the target domain, T is the transpose, Z S is the source domain intermediate layer feature map, Z T is the target domain intermediate layer feature map;

[0027] (3) After defining the graph structure in the semantic segmentation task, a dual-domain channel adjacency matrix must be constructed;

[0028] The graph structure is constructed at the feature graph level, and the node features are updated through graph convolution to establish long-distance context dependencies; feature aggregation between channels is also very helpful for semantic segmentation tasks. Therefore, in order to utilize the relationship between channels in the feature graph, a dual-domain channel adjacency matrix is ​​also proposed at the channel level:

[0029] First, the input source domain feature map Convert to The new feature map has N nodes, and the feature vector dimension of each node is D1; ​​for the target domain feature map Convert to The new feature graph has N nodes, and the dimension of the feature vector of each node is D2; similar to the principle of the position adjacency matrix of the dual domain, using v S and v T To find the channel adjacency matrix of a dual domain:

[0030]

[0031]

[0032] in the formula and are the channel adjacency matrices of the source domain and the target domain, respectively, Z′ S is the converted source domain intermediate layer feature map, Z′ T It is the converted target domain intermediate layer feature map.

[0033] As a further solution of the present invention, the graph convolution operation in S3 is:

[0034] (1) After obtaining the dual-domain position adjacency matrix, the newly constructed dual-domain position adjacency matrix is ​​used to perform graph convolution operations:

[0035]

[0036]

[0037] In the formula, M S and M T are the outputs of the source domain feature map and the target domain feature map after graph convolution, W S and W T is the weight of graph convolution;

[0038] (2) After obtaining the dual-domain channel adjacency matrix, the newly constructed dual-domain channel adjacency matrix is ​​used to perform graph convolution operations:

[0039]

[0040]

[0041] When performing graph convolution to update nodes, the graph convolution formula proposed in GCN is used to perform graph convolution. In the formula, N S and N T are the outputs of the source domain feature map and the target domain feature map after graph convolution, W S and W T is the weight of graph convolution.

[0042] As a further solution of the present invention, the operation of extracting features with domain-invariant properties by using group contrast learning in S5 is: using the group contrast loss function to shorten the distance between the same classes in the domains and to increase the distance between different classes in the domains, forcing the generator to extract features with domain-invariant properties:

[0043]

[0044] In the formula, C S is the source domain feature classifier, C T is the target domain feature classifier, is the predicted probability of the target domain output, is the predicted probability of the source domain output, is the predicted probability of the target domain output; the first term is to make the predicted output values ​​of the same class in the two domains closer, thereby shortening the distance between the same classes in the two domains, and the second term is to make the predicted output values ​​of different classes in the two domains more different, thereby increasing the distance between different classes in the two domains. In the formula, α is the conditional value. When the output of the source domain classifier and the output of the target domain classifier in the second term are of the same class, α=0, C k (k=1,2,3) represents the number of categories in the kth group.

[0045] As a further solution of the present invention, in S6, the self-training operation is: first, a cross entropy loss function is performed on the feature map output by the network. Since the target domain has no label, a pseudo label is established by using an entropy value to guide the pseudo label construction method, and self-supervised training is performed on the output prediction map of the target domain:

[0046]

[0047]

[0048] In the formula, is the label of the source domain image, G(Z s ) (i,j) is the output probability prediction result of the source domain, Y t (i,j)* is the predicted pseudo label of the target domain image, G(Z t ) (i,j) is the output probability prediction result of the target domain, formula L seg (G(Z s ),Y s ) is the cross entropy loss function of the source domain, formula L seg (G(Z t ),Y t * ) is the cross entropy loss function of the target domain, where Y t * Pseudo labels are created using the entropy-guided pseudo-label construction method.

[0049] The present invention first constructs a graph structure for the semantic segmentation task, regards each pixel in the image as a node in the graph, regards the similarity between pixels as the edge of the graph, and gives a graph convolution formula on the graph. Secondly, the present invention proposes a dual-domain adjacency matrix on the graph. Then, the present invention aggregates the pixels of the images in the two domains through graph convolution using the constructed adjacency matrix, and establishes a contextual relationship between the domains.

[0050] The beneficial effects of the present invention are:

[0051] (1) A graph structure is constructed for the semantic segmentation task. Each pixel in the image is regarded as a node in the graph, and the similarity between pixels is regarded as the edge of the graph. A graph convolution formula on the graph is given, so that graph convolution can be applied to the field of semantic segmentation.

[0052] (2) A dual-domain adjacency matrix on a graph is proposed. The constructed adjacency matrix is ​​used to perform graph convolution to aggregate the pixels of images in the two domains and establish a contextual relationship between the domains.

[0053] (3) A group contrast loss function is proposed, which can effectively improve the class imbalance problem in semantic segmentation datasets.

[0054] (4) Experimental results show that the method proposed in this paper can better extract domain-invariant information in the image and obtain better evaluation results in subjective and objective evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of an embodiment of the present invention;

[0056] Figure 2 It is a schematic diagram of the structure of the model of the present invention;

[0057] Figure 3 This is an experimental effect diagram of the present invention. DETAILED DESCRIPTION

[0058] Example 1: Figure 1-Figure 3 As shown in FIG, a cross-domain semantic segmentation method based on graph convolution and contrastive learning, the specific steps of the method are as follows:

[0059] S1: Select a network training dataset, wherein the network training dataset includes a source domain dataset and a target domain dataset, wherein the source domain dataset contains labels and the target domain dataset does not contain labels; the target domain dataset includes a training dataset and a test set;

[0060] S2: Construct a basic network model for semantic segmentation. First, use the source domain dataset to train the basic network model for semantic segmentation. Use the basic network model for semantic segmentation trained with the source domain dataset as the basic network. Send the source domain dataset and the target domain dataset together to the basic network for training.

[0061] S3: Use the feature graph output by the middle layer of the network to construct a dual-domain adjacency matrix, which is a position adjacency matrix and a channel adjacency matrix. Use the constructed source domain position adjacency matrix and source domain channel adjacency matrix to perform graph convolution operations on the graph constructed by the source domain feature graph. Use the constructed target domain position adjacency matrix and target domain channel adjacency matrix to perform graph convolution operations on the graph constructed by the target domain feature graph.

[0062] S4: Input the source domain feature map after the graph convolution operation into the source domain classifier to obtain the prediction probability map of the source domain image, and input the target domain feature map after the graph convolution operation into the target domain classifier to obtain the prediction probability map of the target domain image;

[0063] S5: Extract domain-invariant features using group contrastive learning: Use the group contrastive loss function to narrow the distance between the same classes in the domains and widen the distance between different classes in the domains, forcing the generator to extract features with domain-invariant properties.

[0064] S6: Use the idea of ​​entropy minimization to screen the prediction confidence of the target domain output prediction map, construct pseudo labels, and use the forged pseudo labels to self-train the network.

[0065] As a further solution of the present invention, the source domain dataset is a GTA5 dataset, the target domain dataset is a Cityscapes dataset, and the test set is a test set part of the target domain dataset.

[0066] As a further solution of the present invention, the source domain dataset is GTA5, which includes 24,966 pictures for training to synthesize street scene images and corresponding label images, and the target domain dataset is Cityscapes, which includes 5,000 pictures of 50 different city street scenes for training, real street scene pictures, and 500 real street scene pictures for testing.

[0067] As a further solution of the present invention, the implementation details of constructing the dual-domain adjacency matrix in S3 are as follows:

[0068] (1) First, we need to define a graph in the semantic segmentation task: Due to the characteristics of the graph structure itself, long-distance contextual relationships can be realized in a single-layer graph convolution operation. The graph consists of nodes and edges, and the graph convolution formula is defined as:

[0069]

[0070] In the formula, X is the graph before graph convolution, is the graph obtained after the graph convolution operation, σ() is the nonlinear activation function, A is the adjacency matrix of the graph, and W is the weight of the graph convolution;

[0071] In the graph, the adjacency matrix represents the connection relationship between the graph nodes, and the adjacency matrix is ​​normalized into a Laplace matrix In order to introduce the characteristics of the node itself during the node update process, the Laplacian matrix is ​​written as Where I is the identity matrix, the normalized Laplace matrix Denoted as A, the graph convolution formula is written as:

[0072] H (l+1) =σ(AH (l) W (l) )

[0073] In the formula, H (l) is the graph of layer l, H (l+1) Yes H (l) The graph obtained after graph convolution;

[0074] (2) After defining the graph structure in the semantic segmentation task, we need to construct a dual-domain position adjacency matrix:

[0075] Assume that the source domain feature map is Where D is the number of channels in the feature map, N = H × W is the number of channels in the feature map, Figure 1 The number of pixels on the channel; the target domain feature map is Where D is the number of channels in the feature map, N = H × W is the number of channels in the feature map, Figure 1 The number of pixels on the channel; using Xs and Y T The similarity matrix of the dual domains, i.e., the position adjacency matrix, is constructed. The newly constructed adjacency matrix is ​​used for graph convolution to update the information between nodes and aggregate the feature information on the nodes to capture the long-distance contextual relationship. The similarity matrix is ​​constructed on the feature map output by the network. In cross-domain, in order not to lose the semantic information of the image, the diagonal matrix of the input feature map is introduced in the process of constructing the adjacency matrix of the dual domains, so that the adjacency matrix is ​​associated with the input:

[0076]

[0077]

[0078] in the formula and are the position adjacency matrices of the source domain and the target domain respectively; i is the pixel point in the intermediate feature map of the source domain, j is the pixel point in the intermediate feature map of the target domain, T is the transpose, Z S is the source domain intermediate layer feature map, Z T is the target domain intermediate layer feature map;

[0079] (3) After defining the graph structure in the semantic segmentation task, a dual-domain channel adjacency matrix must be constructed;

[0080] The graph structure is constructed at the feature map level, and the node features are updated through graph convolution to establish long-distance context dependencies; feature aggregation between channels is also very helpful for semantic segmentation tasks. Therefore, in order to utilize the relationship between channels in the feature map, a dual-domain channel adjacency matrix is ​​also proposed at the channel level:

[0081] First, the input source domain feature map Convert to The new feature map has N nodes, and the feature vector dimension of each node is D1; ​​for the target domain feature map Convert to The new feature graph has N nodes, and the dimension of the feature vector of each node is D2; similar to the principle of the position adjacency matrix of the dual domain, using v S and v T To find the channel adjacency matrix of a dual domain:

[0082]

[0083]

[0084] in the formula and are the channel adjacency matrices of the source domain and the target domain, respectively, Z′ Sis the converted source domain intermediate layer feature map, Z′ T It is the converted target domain intermediate layer feature map.

[0085] As a further solution of the present invention, the graph convolution operation in S3 is:

[0086] (1) After obtaining the dual-domain position adjacency matrix, the newly constructed dual-domain position adjacency matrix is ​​used to perform graph convolution operations:

[0087]

[0088]

[0089] In the formula, M S and M T are the outputs of the source domain feature map and the target domain feature map after graph convolution, W S and W T is the weight of graph convolution;

[0090] (2) After obtaining the dual-domain channel adjacency matrix, the newly constructed dual-domain channel adjacency matrix is ​​used to perform graph convolution operations:

[0091]

[0092]

[0093] When performing graph convolution to update nodes, the graph convolution formula proposed in GCN is used to perform graph convolution. In the formula, N S and N T are the outputs of the source domain feature map and the target domain feature map after graph convolution, W S and W T is the weight of graph convolution.

[0094] As a further solution of the present invention, the operation of extracting features with domain-invariant properties by using group contrast learning in S5 is: using the group contrast loss function to shorten the distance between the same classes in the domains and to increase the distance between different classes in the domains, forcing the generator to extract features with domain-invariant properties:

[0095]

[0096] In the formula, C S is the source domain feature classifier, C T is the target domain feature classifier, is the predicted probability of the target domain output, is the predicted probability of the source domain output, is the predicted probability of the target domain output; the first term is to make the predicted output values ​​of the same class in the two domains closer, thereby shortening the distance between the same classes in the two domains, and the second term is to make the predicted output values ​​of different classes in the two domains more different, thereby increasing the distance between different classes in the two domains. In the formula, α is the conditional value. When the output of the source domain classifier and the output of the target domain classifier in the second term are of the same class, α=0, C k (k=1,2,3) represents the number of categories in the kth group.

[0097] As a further solution of the present invention, in S6, the self-training operation is: first, a cross entropy loss function is performed on the feature map output by the network. Since the target domain has no label, a pseudo label is established by using an entropy value to guide the pseudo label construction method, and self-supervised training is performed on the output prediction map of the target domain:

[0098]

[0099]

[0100] In the formula, Y s (i,j) is the label of the source domain image, G(Z s ) (i,j) is the output probability prediction result of the source domain, Y t (i,j)* is the predicted pseudo label of the target domain image, G(Z t ) (i,j) is the output probability prediction result of the target domain, formula L seg (G(Z s ),Y s ) is the cross entropy loss function of the source domain, formula L seg (G(Z t ), Y t * ) is the cross entropy loss function of the target domain, where Y t * Pseudo labels are created using the entropy-guided pseudo-label construction method.

[0101] The present invention evaluates the proposed method from two aspects: subjective and objective. The present invention uses the commonly used index miou in the field of semantic segmentation to measure the recognition rate of the category. At the same time, the present invention compares several currently more advanced methods in the field of cross-domain semantic segmentation. Subjective evaluation indicators such as Figure 3Table 1 shows the experimental results of cross-domain semantic segmentation comparison from GTA5 to Cityscapes, and Table 2 shows the experimental results of cross-domain semantic segmentation comparison from SYNTHIA to Cityscapes. The objective evaluation index miou is shown in Tables 1 and 2. The larger the value of miou of a certain category, the higher the recognition rate of the network for this category, and the better the experimental effect.

[0102] Table 1 shows the cross-domain semantic segmentation comparison experimental results from the dataset GTA5 to Cityscapes.

[0103]

[0104] Table 2 shows the cross-domain semantic segmentation comparison experimental results from the SYNTHIA dataset to Cityscapes.

[0105]

[0106] The specific embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A cross-domain semantic segmentation method based on graph convolution and contrastive learning, characterized in that: The method comprises: S1: Select a network training dataset, wherein the network training dataset includes a source domain dataset and a target domain dataset, wherein the source domain dataset contains labels and the target domain dataset does not contain labels; the target domain dataset includes a training dataset and a test set; S2: Construct a basic network model for semantic segmentation. First, use the source domain dataset to train the basic network model for semantic segmentation. Use the basic network model for semantic segmentation trained with the source domain dataset as the basic network. Send the source domain dataset and the target domain dataset together to the basic network for training. S3: Use the feature graph output by the middle layer of the network to construct a dual-domain adjacency matrix, which is a position adjacency matrix and a channel adjacency matrix. Use the constructed source domain position adjacency matrix and source domain channel adjacency matrix to perform graph convolution operations on the graph constructed by the source domain feature graph. Use the constructed target domain position adjacency matrix and target domain channel adjacency matrix to perform graph convolution operations on the graph constructed by the target domain feature graph. S4: Input the source domain feature map after the graph convolution operation into the source domain classifier to obtain the prediction probability map of the source domain image, and input the target domain feature map after the graph convolution operation into the target domain classifier to obtain the prediction probability map of the target domain image; S5: Extract domain-invariant features using group contrastive learning: Use the group contrastive loss function to narrow the distance between the same classes in the domains and widen the distance between different classes in the domains, forcing the generator to extract features with domain-invariant properties. S6: Use the idea of ​​entropy minimization to screen the prediction confidence of the target domain output prediction map, construct pseudo labels, and use the forged pseudo labels to self-train the network.

2. The cross-domain semantic segmentation method based on graph convolution and contrastive learning according to claim 1, characterized in that: The source domain dataset is the GTA5 dataset, the target domain dataset is the Cityscapes dataset, and the test set is the test set part of the target domain dataset.

3. The cross-domain semantic segmentation method based on graph convolution and contrastive learning according to claim 1, characterized in that: The source domain dataset is GTA5, which includes 24,966 pictures of synthesized street scene images and corresponding labeled images for training. The target domain dataset is Cityscapes, which includes 5,000 pictures of 50 different city street scenes for training (real street scene pictures) and 500 real street scene pictures for testing.

4. The cross-domain semantic segmentation method based on graph convolution and contrastive learning according to claim 1, characterized in that: The implementation details of constructing the dual-domain adjacency matrix in S3 are as follows: (1) First, we need to define a graph in the semantic segmentation task: Due to the characteristics of the graph structure itself, long-distance contextual relationships can be realized in a single-layer graph convolution operation. The graph consists of nodes and edges, and the graph convolution formula is defined as: In the formula, X is the graph before graph convolution, is the graph obtained after the graph convolution operation, σ() is the nonlinear activation function, A is the adjacency matrix of the graph, and W is the weight of the graph convolution; In the graph, the adjacency matrix represents the connection relationship between the graph nodes, and the adjacency matrix is ​​normalized into a Laplace matrix In order to introduce the characteristics of the node itself during the node update process, the Laplacian matrix is ​​written as Where I is the identity matrix, the normalized Laplace matrix Denoted as A, the graph convolution formula is written as: H (l+1) =σ(AH (l) W (l) ) In the formula, H (l) is the graph of layer l, H (l+1) Yes H (l) The graph obtained after graph convolution; (2) After defining the graph structure in the semantic segmentation task, we need to construct a dual-domain position adjacency matrix: Assume that the source domain feature map is Where D is the number of channels of the feature map, N = H × W is the number of pixels in one channel of the feature map; the target domain feature map is Where D is the number of channels in the feature map, N = H × W is the number of pixels in one channel of the feature map; s and Y T The similarity matrix of the dual domains, i.e., the position adjacency matrix, is constructed. The newly constructed adjacency matrix is ​​used for graph convolution to update the information between nodes and aggregate the feature information on the nodes to capture the long-distance contextual relationship. The similarity matrix is ​​constructed on the feature map output by the network. In cross-domain, in order not to lose the semantic information of the image, the diagonal matrix of the input feature map is introduced in the process of constructing the adjacency matrix of the dual domains, so that the adjacency matrix is ​​associated with the input: in the formula and are the position adjacency matrices of the source domain and the target domain respectively; i is the pixel point in the intermediate feature map of the source domain, j is the pixel point in the intermediate feature map of the target domain, T is the transpose, and Z S is the source domain intermediate layer feature map, Z T is the target domain intermediate layer feature map; (3) After defining the graph structure in the semantic segmentation task, a dual-domain channel adjacency matrix must be constructed; The graph structure is constructed at the feature map level, and the node features are updated through graph convolution to establish long-distance context dependencies; feature aggregation between channels is also very helpful for semantic segmentation tasks. Therefore, in order to utilize the relationship between channels in the feature map, a dual-domain channel adjacency matrix is ​​also proposed at the channel level: First, the input source domain feature map Convert to The new feature map has N nodes, and the feature vector dimension of each node is D1; ​​for the target domain feature map Convert to The new feature graph has N nodes, and the dimension of the feature vector of each node is D2; similar to the principle of the position adjacency matrix of the dual domain, using ν S and ν T To find the channel adjacency matrix of a dual domain: in the formula and are the channel adjacency matrices of the source domain and the target domain, respectively, Z′ S is the converted source domain intermediate layer feature map, Z′ T It is the converted target domain intermediate layer feature map.

5. The cross-domain semantic segmentation method based on graph convolution and contrastive learning according to claim 4, characterized in that: The graph convolution operation in S3 is: (1) After obtaining the dual-domain position adjacency matrix, the newly constructed dual-domain position adjacency matrix is ​​used to perform graph convolution operations: In the formula, M S and M T are the outputs of the source domain feature map and the target domain feature map after graph convolution, W S and W T is the weight of graph convolution; (2) After obtaining the dual-domain channel adjacency matrix, the newly constructed dual-domain channel adjacency matrix is ​​used to perform graph convolution operations: When performing graph convolution to update nodes, the graph convolution formula proposed in GCN is used to perform graph convolution. In the formula, N S and N T are the outputs of the source domain feature map and the target domain feature map after graph convolution, W S and W T is the weight of graph convolution.

6. The cross-domain semantic segmentation method based on graph convolution and contrastive learning according to claim 1, characterized in that: The operation of extracting domain-invariant features by using group contrast learning in S5 is: using the group contrast loss function to shorten the distance between the same classes in the domains and to increase the distance between different classes in the domains, so as to force the generator to extract features with domain-invariant properties: In the formula, C S is the source domain feature classifier, C T is the target domain feature classifier, is the predicted probability of the target domain output, is the predicted probability of the source domain output, is the predicted probability of the target domain output; the first term is to make the predicted output values ​​of the same class in the two domains closer, thereby shortening the distance between the same classes in the two domains, and the second term is to make the predicted output values ​​of different classes in the two domains more different, thereby increasing the distance between different classes in the two domains. In the formula, α is the conditional value. When the output of the source domain classifier and the output of the target domain classifier in the second term are of the same class, α=0, C k (k=1,2,3) represents the number of categories in the kth group.

7. The cross-domain semantic segmentation method based on graph convolution and contrastive learning according to claim 1, characterized in that: In S6, the self-training operation is: first, a cross entropy loss function is applied to the feature map output by the network. Since the target domain has no label, a pseudo label is established by using an entropy value to guide the pseudo label construction method, and self-supervised training is performed on the output prediction map of the target domain: In the formula, is the label of the source domain image, G(Z s ) (i,j) is the output probability prediction result of the source domain, Y t (i,j)* is the predicted pseudo label of the target domain image, G(Z t ) (i,j) is the output probability prediction result of the target domain, formula L seg (G(Z s ),Y s ) is the cross entropy loss function of the source domain, formula L seg (G(Z t ),Y t * ) is the cross entropy loss function of the target domain, where Y t * Pseudo labels are created using the entropy-guided pseudo-label construction method.