An unsupervised domain adaptive semantic segmentation method and system

Abstract

The present invention provides an unsupervised domain adaptive semantic segmentation method and system, comprising: obtaining a DSM image, a remote sensing image, and a semantically labeled image; inputting the DSM image and the remote sensing image into a multi-path encoder to obtain a final feature map of the DSM image and a final feature map of the remote sensing image; inputting the final feature map of the DSM image and the final feature map of the remote sensing image into a multi-task decoder to generate segmentation results of the source domain and segmentation results of the target domain; updating the semantic segmentation model using a segmentation loss to obtain a first updated semantic segmentation model; updating the first updated semantic segmentation model using an adversarial loss to obtain a second updated semantic segmentation model; and inputting a remote sensing image dataset of the target domain into the second updated semantic segmentation model to obtain a final semantic segmentation prediction map. The present invention corrects context confusion caused by data distribution mismatch and reduces common misclassifications of boundary areas, significantly improving segmentation accuracy and reliability.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and in particular to an unsupervised domain adaptive semantic segmentation method and system. Background Art

[0002] Advances in remote sensing technology and equipment are continuously improving the efficiency of acquiring high-resolution remote sensing imagery. Today, high-resolution remote sensing images are increasingly suitable for a wide range of applications, providing crucial information for the development of various remote sensing image interpretation tasks. Semantic segmentation of remote sensing images is a fundamental task in remote sensing. This task involves assigning a unique class label to each pixel in the image to generate a segmentation mask for the object of interest. It serves numerous applications, including land use and land cover, urban planning, change detection, and urban function extraction. In recent years, the booming development of deep learning methods has advanced semantic segmentation of remote sensing images. A large body of work, exemplified by deep convolutional neural networks and their variants, has been developed for semantic segmentation of remote sensing images, achieving significant performance improvements. However, these works primarily use supervised deep learning methods to train models, relying heavily on a large number of manually labeled pixel-level images. This is labor-intensive and time-consuming, increasing training costs. Furthermore, in practical remote sensing applications, models trained well on a specific dataset (a source domain dataset with labeled images) are generally not directly transferable to another dataset (a target domain dataset with unlabeled images). Because remote sensing images from different datasets vary greatly in lighting, imaging, and geographical environment, the data distribution mismatch (i.e., domain gap) between different datasets can lead to serious contextual confusion.

[0003] To reduce reliance on pixel-level labeled data and narrow the domain gap between the source and target domains, the development of weakly supervised and unsupervised learning methods to solve the semantic segmentation problem of remote sensing imagery is gaining momentum. Unsupervised domain adaptation (UDA) strategies have been introduced as a very promising unsupervised learning method for the semantic segmentation of remote sensing imagery, achieving significant performance improvements and significantly advancing the development of semantic segmentation. These methods utilize UDA to transfer segmentation knowledge of remote sensing imagery from a densely labeled source domain to an unlabeled target domain. By narrowing the domain gap between the source and target domains through various UDA strategies, they effectively improve the accuracy and reliability of segmentation results.

[0004] However, existing technologies that only use orthophotos or satellite images as training images to improve the segmentation performance of the model have some limitations: 1) Remote sensing images contain a large number of objects with similar spectra but different categories and heights. It is difficult to accurately identify these areas based solely on remote sensing images, resulting in classification confusion; 2) Pixel-level classification of adjacent boundaries and shadow areas of different objects is prone to noise, resulting in unreliable predictions.

[0005] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0006] To solve the above technical problems, the present invention provides an unsupervised domain adaptive semantic segmentation method, comprising:

[0007] S1: Obtain the remote sensing image datasets of the source domain and the target domain, crop the remote sensing image datasets of the source domain and the target domain to obtain DSM images, remote sensing images and semantic label images;

[0008] S2: Build a semantic segmentation model, which includes a multi-path encoder and a multi-task decoder. The DSM image and the remote sensing image are input into the multi-path encoder to obtain the final feature map of the DSM image and the final feature map of the remote sensing image.

[0009] S3: Input the final feature map of the DSM image and the final feature map of the remote sensing image into the multi-task decoder to generate the segmentation results of the source domain and the target domain;

[0010] S4: Calculate the segmentation loss using the source domain segmentation result and the semantic label image, and update the semantic segmentation model using the segmentation loss to obtain a first updated semantic segmentation model.

[0011] S5: Construct a discrimination model, input the segmentation results of the source domain and the target domain into the discrimination model, calculate the adversarial loss and the discrimination loss, update the first updated semantic segmentation model by the adversarial loss to obtain the second updated semantic segmentation model, and update the discrimination model by the discrimination loss to obtain the first updated discrimination model;

[0012] S6: Construct an overall model through the second updated semantic segmentation model and the first updated identification model, train the overall model to obtain a trained overall model, input the remote sensing image dataset of the target domain into the trained second updated semantic segmentation model to obtain the final segmentation result, perform fine post-fusion processing on the final segmentation result to obtain the final semantic segmentation prediction map.

[0013] Preferably, step S2 is specifically as follows:

[0014] S21: Input the DSM image into the DSM feature extractor of the multi-path encoder and output the feature map of the DSM image of the nth stage The remote sensing image is input into the remote sensing image feature extractor of the multi-path encoder, and the feature map of the remote sensing image of the nth stage is output. Where n is the stage number, 1≤n≤5;

[0015] S22: The feature map of the DSM image of the nth stage And the feature map of the remote sensing image at the nth stage Input to the attention-aware fusion block of the multi-path encoder to obtain fusion features

[0016] S23: Fusion features Feature map of the DSM image at the nth stage Weighted addition, the result of the addition is used as the DSM image input of the feature extractor at the n+1th stage; the feature map of the remote sensing image at the nth stage As the remote sensing image input of the n+1th stage of the feature extractor;

[0017] S24: The feature map of the weighted DSM image at the fifth stage is used as the final feature map of the DSM image, and the feature map of the remote sensing image at the fifth stage is used as the final feature map of the remote sensing image.

[0018] Preferably, step S3 is specifically as follows:

[0019] S31: Input the final feature map of the DSM image into the attention-aware filtering block of the multi-task decoder to obtain the DSM image segmentation results of the source domain and the DSM image segmentation results of the target domain; input the final feature map of the remote sensing image into the ASPP of the multi-task decoder to obtain the remote sensing image segmentation results of the source domain and the remote sensing image segmentation results of the target domain;

[0020] S32: Inputting the DSM image segmentation result of the source domain and the remote sensing image segmentation result of the source domain into the adaptive gated fusion block to obtain the fused segmentation result of the source domain; inputting the DSM image segmentation result of the target domain and the remote sensing image segmentation result of the target domain into the adaptive gated fusion block to obtain the fused segmentation result of the target domain;

[0021] S33: The segmentation result of the source domain is composed of the DSM image segmentation result of the source domain, the remote sensing image segmentation result of the source domain, and the fusion segmentation result of the source domain; the segmentation result of the target domain is composed of the DSM image segmentation result of the target domain, the remote sensing image segmentation result of the target domain, and the fusion segmentation result of the target domain.

[0022] Preferably, step S4 is specifically as follows:

[0023] The DSM image segmentation results of the source domain, the remote sensing image segmentation results of the source domain, the fusion segmentation results of the source domain and the semantic label image are calculated using the standard cross entropy loss to obtain the segmentation loss. The semantic segmentation model is updated by the segmentation loss to obtain the first updated semantic segmentation model.

[0024] Preferably, step S5 is specifically as follows:

[0025] S51: Inputting the remote sensing image segmentation result of the source domain, the remote sensing image segmentation result of the target domain, the fusion segmentation result of the source domain, and the fusion segmentation result of the target domain into the identification model to obtain the source domain identification output and the target domain identification output; setting the domain label Z, defining Z=1 as the source domain label and Z=0 as the target domain label;

[0026] S52: Calculate the adversarial loss using the target domain identification output and the domain label Z, and update the first updated semantic segmentation model using the adversarial loss to obtain a second updated semantic segmentation model.

[0027] S53: Calculate the discrimination loss through the source domain discrimination output and the target domain discrimination output, and update the discrimination model through the discrimination loss to obtain a first updated discrimination model.

[0028] Preferably, in step S6, a fine post-fusion processing operation is performed on the final segmentation result to obtain a final semantic segmentation prediction map, specifically:

[0029] S61: The final segmentation result includes: the final segmentation result of the DSM image of the target domain, the final segmentation result of the remote sensing image of the target domain, and the fused final segmentation result of the target domain; the final segmentation result of the DSM image of the target domain and the fused final segmentation result of the target domain are normalized through a softmax layer to obtain a DSM probability map and a fused probability map;

[0030] S62: Filter the DSM probability map and the fused probability map respectively to obtain the filtered DSM probability map mask Wa and the fused probability map mask Wb, and perform the & operation on Wa and Wb to obtain the final mask Wc;

[0031] S63: Multiply the DSM probability map by the final mask Wc to obtain a filtered DSM probability map, add the filtered DSM probability map to the fused probability map, and perform pixel-by-pixel classification to obtain the final semantic segmentation prediction map.

[0032] An unsupervised domain adaptive semantic segmentation system, comprising:

[0033] The cropping module is used to obtain the remote sensing image datasets of the source domain and the target domain, and crop the remote sensing image datasets of the source domain and the target domain to obtain DSM images, remote sensing images and semantic label images;

[0034] The feature map acquisition module is used to build a semantic segmentation model. The semantic segmentation model includes a multi-path encoder and a multi-task decoder. The DSM image and the remote sensing image are input into the multi-path encoder to obtain the final feature map of the DSM image and the final feature map of the remote sensing image.

[0035] The segmentation module is used to input the final feature map of the DSM image and the final feature map of the remote sensing image into the multi-task decoder to generate the segmentation results of the source domain and the target domain;

[0036] A first updating module is configured to calculate a segmentation loss based on the segmentation result of the source domain and the semantic label image, and update the semantic segmentation model based on the segmentation loss to obtain a first updated semantic segmentation model;

[0037] The second updating module is used to build a discrimination model, input the segmentation results of the source domain and the target domain into the discrimination model, calculate the adversarial loss and the discrimination loss, update the first updated semantic segmentation model by the adversarial loss to obtain the second updated semantic segmentation model, and update the discrimination model by the discrimination loss to obtain the first updated discrimination model;

[0038] The final prediction module is used to construct an overall model through the second updated semantic segmentation model and the first updated identification model, train the overall model to obtain a trained overall model, input the remote sensing image dataset of the target domain into the trained second updated semantic segmentation model to obtain the final segmentation result, and perform fine post-fusion processing operations on the final segmentation result to obtain the final semantic segmentation prediction map.

[0039] The present invention has the following beneficial effects:

[0040] 1. This paper provides a new semantic segmentation model, including: a multi-path encoder and a multi-task decoder. The multi-path encoder uses multiple attention-aware fusion blocks to optimize the feature representation of the DSM from low to high, enhancing the DSM's ability to identify ground targets. The multi-task decoder uses an attention-aware filtering block to focus on learning the local information of the DSM and an adaptive gated fusion block to fuse different data, enhancing the ability to separate easily confused targets and improving the prediction accuracy of boundaries and shadow areas.

[0041] 2. The present invention can further refine the classification information of DSM and optimize the segmentation results by performing fine post-fusion processing on the final segmentation results;

[0042] 3. The method of the present invention makes full use of the complementary information of DSM and remote sensing images, corrects the context confusion caused by data distribution mismatch, and reduces the common misclassification of boundary areas, greatly improving the segmentation accuracy and reliability, and solving the limitations of single-modal remote sensing image data in semantic segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0044] Figure 2 This is the structural diagram of the attention-aware fusion block;

[0045] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0047] Reference Figure 1 , the present invention provides an unsupervised domain adaptive semantic segmentation method, comprising:

[0048] S1: Obtain the remote sensing image datasets of the source domain and the target domain, crop the remote sensing image datasets of the source domain and the target domain to obtain DSM images, remote sensing images and semantic label images;

[0049] Specifically, the source and target domain remote sensing image datasets used are the ISPRS 2D semantic segmentation benchmark datasets. The source domain dataset is the Potsdam IRRG dataset, and the target domain dataset is the Vaihingen IRRG dataset. The semantic labels contain six types of ground objects, namely buildings, impervious surfaces, trees, low vegetation, clutter, and vehicles.

[0050] The DSM images and high-resolution remote sensing images of the source and target domains are cropped, and the semantic label images of the source domain are cropped. According to actual needs, the remote sensing images, DSM images, and semantic label images of the Potsdam IRRG dataset are cropped with a fixed step size of 512 pixels, and 4598 images with a pixel size of 512×512 are obtained respectively. The remote sensing images, DSM images, and semantic label images of the Vaihingen IRRG dataset are cropped with a fixed step size of 256 pixels. The semantic labels are only used to evaluate the model performance.

[0051] S2: Build a semantic segmentation model, which includes a multi-path encoder and a multi-task decoder. The DSM image and the remote sensing image are input into the multi-path encoder to obtain the final feature map of the DSM image and the final feature map of the remote sensing image.

[0052] S3: Input the final feature map of the DSM image and the final feature map of the remote sensing image into the multi-task decoder to generate the segmentation results of the source domain and the target domain;

[0053] S4: Calculate the segmentation loss using the source domain segmentation result and the semantic label image, and update the semantic segmentation model using the segmentation loss to obtain a first updated semantic segmentation model.

[0054] S5: Construct a discrimination model, input the segmentation results of the source domain and the target domain into the discrimination model, calculate the adversarial loss and the discrimination loss, update the first updated semantic segmentation model by the adversarial loss to obtain the second updated semantic segmentation model, and update the discrimination model by the discrimination loss to obtain the first updated discrimination model;

[0055] S6: Construct an overall model through the second updated semantic segmentation model and the first updated identification model, train the overall model to obtain a trained overall model, input the remote sensing image dataset of the target domain into the trained second updated semantic segmentation model to obtain the final segmentation result, perform fine post-fusion processing on the final segmentation result to obtain the final semantic segmentation prediction map.

[0056] In this embodiment, a multi-path encoder is constructed; the multi-path encoder includes a DSM feature extractor, a remote sensing image feature extractor, and an attention-aware fusion block; the DSM feature extractor is composed of ResNet34, in which the stride of the last two convolutional layers is changed from 2 to 1, and feature maps of five different stages are generated from low to high levels; the remote sensing image feature extractor is composed of an encoder of the DeeplabV2 framework using ResNet101 pre-trained on the ImageNet dataset as the backbone network, and feature maps of five different stages are generated from low to high levels; the five stages of the two feature extractors are sequentially connected to an attention-aware fusion block;

[0057] Step S2 is specifically as follows:

[0058] S21: Input the DSM image into the DSM feature extractor of the multi-path encoder and output the feature map of the DSM image of the nth stage The remote sensing image is input into the remote sensing image feature extractor of the multi-path encoder, and the feature map of the remote sensing image of the nth stage is output. Where n is the stage number, 1≤n≤5;

[0059] Specifically, the expression of the feature map of the DSM image is:

[0060]

[0061]

[0062]

[0063]

[0064]

[0065] The expression of the feature map of remote sensing image is:

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] S22: The feature map of the DSM image of the nth stage And the feature map of the remote sensing image at the nth stage Input to the attention-aware fusion block of the multi-path encoder to obtain fusion features

[0072] Specifically, refer to Figure 2 The structural diagram of the attention perception fusion block is as follows: First, the feature map of the remote sensing image is converted into Input a 1×1 convolution block to reduce the dimension and obtain a new feature map Compare it with the feature map of the DSM image The number of channels is kept consistent; the feature map of the new remote sensing image is spliced with the feature map of the DSM image in the channel dimension, and the spliced features are input into the spatial attention block and the channel attention block after obtaining the spliced features; in the spatial attention block, the spliced features are sequentially passed through the 1×1 convolution block, BN layer, ReLU activation function, 1×1 convolution block, BN layer and Sigmoid function, and the output features are Hadamard products with the spliced features to obtain the spatial feature map; in the channel attention block, the spliced features are sequentially passed through the global pooling layer, 1×1 convolution block, BN layer, ReLU activation function, 1×1 convolution block and Sigmoid function, and the output features are Hadamard products with the spliced features to obtain the channel feature map; the spatial feature map and the channel feature map are connected in the channel dimension, and sequentially passed through the 1×1 convolution block, BN layer, and ReLU activation function to obtain the fusion feature This fusion feature combines the feature representation learned from the texture and color of remote sensing images with the feature representation learned from the DSM height, which is helpful for assisting the DSM feature extractor to learn more detailed feature information. The specific parameters of the attention perception fusion block at each stage are shown in Table 1.

[0073] Table 1 Parameter settings of attention perception fusion blocks at different stages

[0074]

[0075] S23: Fusion features Feature map of the DSM image at the nth stage Weighted addition, the result of the addition is used as the DSM image input of the feature extractor at the n+1th stage; the feature map of the remote sensing image at the nth stage As the remote sensing image input of the n+1th stage of the feature extractor;

[0076] S24: The feature map of the weighted DSM image at the fifth stage is used as the final feature map of the DSM image, and the feature map of the remote sensing image at the fifth stage is used as the final feature map of the remote sensing image.

[0077] In this embodiment, the multi-task decoder includes an ASPP, an attention-aware filtering block, and an adaptive gated fusion block; the output ends of the ASPP and the attention-aware filtering block are connected to the adaptive gated fusion block;

[0078] Step S3 is specifically as follows:

[0079] S31: Input the final feature map of the DSM image into the attention-aware filtering block of the multi-task decoder to obtain the DSM image segmentation results of the source domain and the DSM image segmentation results of the target domain; input the final feature map of the remote sensing image into the ASPP of the multi-task decoder to obtain the remote sensing image segmentation results of the source domain and the remote sensing image segmentation results of the target domain;

[0080] Specifically, the attention-aware filter block consists of a classic dual-attention mechanism, including a position attention module (PAM) and a channel attention module; the attention-aware filter block is used to focus on the feature map of the DSM image. The local area helps DSM learn four types of objects with obvious height differences, namely buildings, impervious surfaces, trees and low vegetation, while filtering out the interference information of clutter and vehicles learned by the attention perception fusion block;

[0081] S32: Inputting the DSM image segmentation result of the source domain and the remote sensing image segmentation result of the source domain into the adaptive gated fusion block to obtain the fused segmentation result of the source domain; inputting the DSM image segmentation result of the target domain and the remote sensing image segmentation result of the target domain into the adaptive gated fusion block to obtain the fused segmentation result of the target domain;

[0082] Specifically, the specific working process of the adaptive gate fusion block is as follows: DSM image segmentation result P D And the remote sensing image segmentation result P I First, the connection is performed in the channel dimension. The result of the connection is passed through a 1×1 convolution block and a Sigmoid function to calculate the adaptive weight F, which determines the P in the fusion result. D and P IThe retained ratio can better integrate the information that is beneficial to classification; the adaptive weight F and P I The multiplication result and 1-F and P D The multiplication results are added to obtain the fusion segmentation result P F ;

[0083] S33: The segmentation result of the source domain is composed of the DSM image segmentation result of the source domain, the remote sensing image segmentation result of the source domain, and the fusion segmentation result of the source domain; the segmentation result of the target domain is composed of the DSM image segmentation result of the target domain, the remote sensing image segmentation result of the target domain, and the fusion segmentation result of the target domain.

[0084] In this embodiment, step S4 is specifically as follows:

[0085] The DSM image segmentation results of the source domain, the remote sensing image segmentation results of the source domain, the fusion segmentation results of the source domain and the semantic label image are calculated using the standard cross entropy loss to obtain the segmentation loss. The semantic segmentation model is updated by the segmentation loss to obtain the first updated semantic segmentation model.

[0086] Specifically, the segmentation loss is obtained by adding the segmentation loss of DSM, the segmentation loss of remote sensing images, the segmentation loss of fusion segmentation results, and the dice loss;

[0087] The expression of DSM segmentation loss is:

[0088]

[0089] The expression of segmentation loss of remote sensing image is:

[0090]

[0091] The expression of the segmentation loss of the fused segmentation result is:

[0092]

[0093] The expression of dice loss is:

[0094]

[0095] Among them, H, W and C represent the height, width and number of categories of the segmentation result, C* represents the number of categories of DSM feature learning; Y S One-hot encoding of the semantic label image representing the source domain; ε is set to 1; is the DSM image segmentation result of the source domain, is the remote sensing image segmentation result of the source domain, is the fusion segmentation result of the source domain.

[0096] In this embodiment, step S5 is specifically as follows:

[0097] S51: Inputting the remote sensing image segmentation result of the source domain, the remote sensing image segmentation result of the target domain, the fusion segmentation result of the source domain, and the fusion segmentation result of the target domain into the identification model to obtain the source domain identification output and the target domain identification output; setting the domain label Z, defining Z=1 as the source domain label and Z=0 as the target domain label;

[0098] S52: Calculate the adversarial loss using the target domain identification output and the domain label Z, and update the first updated semantic segmentation model using the adversarial loss to obtain a second updated semantic segmentation model.

[0099] Specifically, the expression of adversarial loss is:

[0100]

[0101]

[0102] Where, H′=H / 32, W′=W / 32, Z=1; is the remote sensing image segmentation result of the target domain, is the fusion segmentation result of the target domain;

[0103] S53: Calculate the discrimination loss using the source domain discrimination output and the target domain discrimination output, and update the discrimination model using the discrimination loss to obtain a first updated discrimination model;

[0104] Specifically, the expression of identification loss is:

[0105]

[0106]

[0107] Where, H′=H / 32, W′=W / 32, Z=1, is the remote sensing image segmentation result of the source domain, is the fusion segmentation result of the source domain.

[0108] In this embodiment, in step S6, a fine post-fusion processing operation is performed on the final segmentation result to obtain a final semantic segmentation prediction map, specifically:

[0109] S61: The final segmentation result includes: the final segmentation result of the DSM image of the target domain, the final segmentation result of the remote sensing image of the target domain, and the fused final segmentation result of the target domain; the final segmentation result of the DSM image of the target domain and the fused final segmentation result of the target domain are normalized through a softmax layer to obtain a DSM probability map and a fused probability map;

[0110] S62: Filter the DSM probability map and the fused probability map respectively to obtain the filtered DSM probability map mask Wa and the fused probability map mask Wb, and perform the & operation on Wa and Wb to obtain the final mask Wc;

[0111] Specifically, the areas where the probability values of the four types of objects in the DSM probability map are greater than the threshold μ are calculated to obtain the filtered DSM probability map mask Wa. The areas where the predicted category is not car are used as the fused probability map mask Wb. Wc represents the area where the DSM probability map is fused with the fused probability map.

[0112] S63: Multiply the DSM probability map with the final mask Wc to obtain a filtered DSM probability map. The filtered DSM probability map is beneficial to enhance the classification information and correct some misclassifications in the fused semantic segmentation prediction map. Add the filtered DSM probability map to the fused probability map and perform pixel-by-pixel classification to obtain the final semantic segmentation prediction map.

[0113] The present invention provides 8 comparative examples for comparison with the present invention.

[0114] Comparative Example 1:

[0115] Unlike the present invention, this example only includes the feature extractor and ASPP of high-resolution remote sensing images. This example does not have domain adaptation and directly uses the source domain dataset to train the model, and the target domain data is evaluated on the trained model.

[0116] Comparative Example 2:

[0117] The difference from the present invention is that the attention perception fusion block and the fine post-fusion processing module in the multi-path encoder are removed.

[0118] Comparative Example 3:

[0119] The difference from the present invention is that the attention-aware filtering block, the adaptive gated fusion block, and the fine post-fusion processing module in the multi-path encoder are removed, the attention-aware filtering block is replaced by ASPP, and the adaptive gated fusion block is replaced by element-wise summation.

[0120] Comparative Example 4:

[0121] The difference from the present invention is that the fine post-fusion processing module is removed.

[0122] Comparative Example 5:

[0123] The AdaptSegNet domain adaptive semantic segmentation model is used, which is trained only using high-resolution remote sensing images of the source and target domains.

[0124] Comparative Example 6:

[0125] The ADVENT domain-adaptive semantic segmentation model is used, which is trained only using high-resolution remote sensing images of the source and target domains.

[0126] Comparative Example 7:

[0127] The CLAN domain-adaptive semantic segmentation model is adopted, which is trained only using high-resolution remote sensing images of the source and target domains.

[0128] Comparative Example 8:

[0129] The DAST domain-adaptive semantic segmentation model is adopted, which is trained only using high-resolution remote sensing images of the source and target domains.

[0130] The experimental results of the present invention and comparative examples 1-8 are shown in Table 2;

[0131] Table 2 Segmentation results of comparative examples 1-8

[0132]

[0133]

[0134] The evaluation indicators are F1_Score and IoU, and the detailed formula is:

[0135]

[0136]

[0137]

[0138]

[0139] Where TP represents true positives, FP represents false positives, and FN represents false negatives. Precision represents the ratio of correctly predicted positive examples to all predicted positive examples, and recall represents the ratio of correctly predicted positive examples to all true positive examples. F1_Score represents the weighted average of precision and recall. P represents the prediction mask, G represents the ground truth mask, and IoU represents the intersection over union (IoU) of the prediction result and the ground truth.

[0140] As can be seen from the data in Table 1, compared with Comparative Example 1, Comparative Example 2 adds the attention-aware fusion block on the basis of domain adaptation, and the overall accuracy has exceeded the accuracy of the four domain adaptation methods. This proves that the attention-aware fusion block helps DSM better capture classification information and further provides more supporting information for predicting ground objects.

[0141] Compared with Comparative Example 1, Comparative Example 3 adds an attention-aware filtering block and an adaptive gated fusion block on the basis of domain adaptation, and obtains better segmentation performance, indicating that this module is conducive to reducing noise information and obtaining better fusion results.

[0142] Compared with Comparative Examples 2 and 3, Comparative Example 4 integrates the attention-aware fusion block, the attention-aware filtering block, and the adaptive gating fusion block, making the classification of clutter and buildings more accurate.

[0143] Compared with Comparative Example 4, the present invention adds a fine post-fusion processing module, which significantly improves the confusion between clutter and other ground objects, thereby improving the accuracy of other ground objects. This fully demonstrates that the fine post-fusion processing module can obtain more reliable DSM classification information to correct classification errors.

[0144] The present invention and comparative examples 5-8 are both domain adaptive methods, among which comparative examples 5-8 are unimodal methods. The present invention achieved the best mIoU and average F1_Score, which were 51.68% and 66.66% respectively, and achieved IoU of 34.34%, 65.71%, 37.90%, 38.80% and 79.05% on clutter, impervious surfaces, cars, low vegetation and buildings respectively. The accuracy of clutter and buildings is much higher than that of other methods, which shows that the present invention effectively improves the performance of domain adaptive semantic segmentation and proves that complementary multimodal data is beneficial to enhance the classification ability of ground objects and reduce misclassification.

[0145] The present invention provides an unsupervised domain adaptive semantic segmentation system, comprising:

[0146] The cropping module is used to obtain the remote sensing image datasets of the source domain and the target domain, and crop the remote sensing image datasets of the source domain and the target domain to obtain DSM images, remote sensing images and semantic label images;

[0147] The feature map acquisition module is used to build a semantic segmentation model. The semantic segmentation model includes a multi-path encoder and a multi-task decoder. The DSM image and the remote sensing image are input into the multi-path encoder to obtain the final feature map of the DSM image and the final feature map of the remote sensing image.

[0148] The segmentation module is used to input the final feature map of the DSM image and the final feature map of the remote sensing image into the multi-task decoder to generate the segmentation results of the source domain and the target domain;

[0149] A first updating module is configured to calculate a segmentation loss based on the segmentation result of the source domain and the semantic label image, and update the semantic segmentation model based on the segmentation loss to obtain a first updated semantic segmentation model;

[0150] The second updating module is used to build a discrimination model, input the segmentation results of the source domain and the target domain into the discrimination model, calculate the adversarial loss and the discrimination loss, update the first updated semantic segmentation model by the adversarial loss to obtain the second updated semantic segmentation model, and update the discrimination model by the discrimination loss to obtain the first updated discrimination model;

[0151] The final prediction module is used to construct an overall model through the second updated semantic segmentation model and the first updated identification model, train the overall model to obtain a trained overall model, input the remote sensing image dataset of the target domain into the trained second updated semantic segmentation model to obtain the final segmentation result, and perform fine post-fusion processing operations on the final segmentation result to obtain the final semantic segmentation prediction map.

[0152] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0153] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same item of hardware. The use of the terms first, second, and third, etc., does not denote any order and should be construed as identifiers.

[0154] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An unsupervised domain adaptive semantic segmentation method, characterized in that include: S1: Obtain the remote sensing image datasets of the source domain and the target domain, crop the remote sensing image datasets of the source domain and the target domain to obtain DSM images, remote sensing images and semantic label images; S2: Build a semantic segmentation model, which includes a multi-path encoder and a multi-task decoder. The DSM image and the remote sensing image are input into the multi-path encoder to obtain the final feature map of the DSM image and the final feature map of the remote sensing image. Step S2 is specifically as follows: S21: Input the DSM image into the DSM feature extractor of the multi-path encoder and output the feature map of the DSM image of the nth stage The remote sensing image is input into the remote sensing image feature extractor of the multi-path encoder, and the feature map of the remote sensing image of the nth stage is output. Where n is the stage number, 1≤n≤5; S22: The feature map of the DSM image of the nth stage And the feature map of the remote sensing image at the nth stage Input to the attention-aware fusion block of the multi-path encoder to obtain fusion features S23: Fusion features Feature map of the DSM image at the nth stage Weighted addition, the result of the addition is used as the DSM image input of the feature extractor at the n+1th stage; the feature map of the remote sensing image at the nth stage As the remote sensing image input of the n+1th stage of the feature extractor; S24: using the weighted feature map of the DSM image at the fifth stage as the final feature map of the DSM image, and using the feature map of the remote sensing image at the fifth stage as the final feature map of the remote sensing image; S3: Input the final feature map of the DSM image and the final feature map of the remote sensing image into the multi-task decoder to generate the segmentation results of the source domain and the target domain; Step S3 is specifically as follows: S31: Input the final feature map of the DSM image into the attention-aware filtering block of the multi-task decoder to obtain the DSM image segmentation results of the source domain and the DSM image segmentation results of the target domain; input the final feature map of the remote sensing image into the ASPP of the multi-task decoder to obtain the remote sensing image segmentation results of the source domain and the remote sensing image segmentation results of the target domain; S32: Inputting the DSM image segmentation result of the source domain and the remote sensing image segmentation result of the source domain into the adaptive gated fusion block to obtain the fused segmentation result of the source domain; inputting the DSM image segmentation result of the target domain and the remote sensing image segmentation result of the target domain into the adaptive gated fusion block to obtain the fused segmentation result of the target domain; S33: The segmentation result of the source domain is composed of the DSM image segmentation result of the source domain, the remote sensing image segmentation result of the source domain, and the fusion segmentation result of the source domain; the segmentation result of the target domain is composed of the DSM image segmentation result of the target domain, the remote sensing image segmentation result of the target domain, and the fusion segmentation result of the target domain; S4: Calculate the segmentation loss using the source domain segmentation result and the semantic label image, and update the semantic segmentation model using the segmentation loss to obtain a first updated semantic segmentation model. S5: Construct a discrimination model, input the segmentation results of the source domain and the target domain into the discrimination model, calculate the adversarial loss and the discrimination loss, update the first updated semantic segmentation model by the adversarial loss to obtain the second updated semantic segmentation model, and update the discrimination model by the discrimination loss to obtain the first updated discrimination model; S6: Construct an overall model through the second updated semantic segmentation model and the first updated identification model, train the overall model to obtain a trained overall model, input the remote sensing image dataset of the target domain into the trained second updated semantic segmentation model to obtain the final segmentation result, perform fine post-fusion processing on the final segmentation result to obtain the final semantic segmentation prediction map.

2. The unsupervised domain adaptive semantic segmentation method according to claim 1, characterized in that Step S4 is specifically as follows: The DSM image segmentation results of the source domain, the remote sensing image segmentation results of the source domain, the fusion segmentation results of the source domain and the semantic label image are calculated using the standard cross entropy loss to obtain the segmentation loss. The semantic segmentation model is updated by the segmentation loss to obtain the first updated semantic segmentation model.

3. The unsupervised domain adaptive semantic segmentation method according to claim 1, characterized in that Step S5 is specifically as follows: S51: Inputting the remote sensing image segmentation result of the source domain, the remote sensing image segmentation result of the target domain, the fusion segmentation result of the source domain, and the fusion segmentation result of the target domain into the identification model to obtain the source domain identification output and the target domain identification output; setting the domain label Z, defining Z=1 as the source domain label and Z=0 as the target domain label; S52: Calculate the adversarial loss using the target domain identification output and the domain label Z, and update the first updated semantic segmentation model using the adversarial loss to obtain a second updated semantic segmentation model. S53: Calculate the discrimination loss through the source domain discrimination output and the target domain discrimination output, and update the discrimination model through the discrimination loss to obtain a first updated discrimination model.

4. The unsupervised domain adaptive semantic segmentation method according to claim 1, characterized in that In step S6, the final segmentation result is subjected to a fine post-fusion processing operation to obtain the final semantic segmentation prediction map, specifically: S61: The final segmentation result includes: the final segmentation result of the DSM image of the target domain, the final segmentation result of the remote sensing image of the target domain, and the fused final segmentation result of the target domain; the final segmentation result of the DSM image of the target domain and the fused final segmentation result of the target domain are normalized through a softmax layer to obtain a DSM probability map and a fused probability map; S62: Filter the DSM probability map and the fused probability map respectively to obtain the filtered DSM probability map mask Wa and the fused probability map mask Wb, and perform the & operation on Wa and Wb to obtain the final mask Wc; S63: Multiply the DSM probability map by the final mask Wc to obtain a filtered DSM probability map, add the filtered DSM probability map to the fused probability map, and perform pixel-by-pixel classification to obtain the final semantic segmentation prediction map.

5. An unsupervised domain adaptive semantic segmentation system, used to implement the unsupervised domain adaptive semantic segmentation method according to any one of claims 1 to 4, characterized in that: include: The cropping module is used to obtain the remote sensing image datasets of the source domain and the target domain, and crop the remote sensing image datasets of the source domain and the target domain to obtain DSM images, remote sensing images and semantic label images; The feature map acquisition module is used to build a semantic segmentation model. The semantic segmentation model includes a multi-path encoder and a multi-task decoder. The DSM image and the remote sensing image are input into the multi-path encoder to obtain the final feature map of the DSM image and the final feature map of the remote sensing image. Step S2 is specifically as follows: S21: Input the DSM image into the DSM feature extractor of the multi-path encoder and output the feature map of the DSM image of the nth stage The remote sensing image is input into the remote sensing image feature extractor of the multi-path encoder, and the feature map of the remote sensing image of the nth stage is output. Where n is the stage number, 1≤n≤5; S22: The feature map of the DSM image of the nth stage And the feature map of the remote sensing image at the nth stage Input to the attention-aware fusion block of the multi-path encoder to obtain fusion features S23: Fusion features Feature map of the DSM image at the nth stage Weighted addition, the result of the addition is used as the DSM image input of the feature extractor at the n+1th stage; the feature map of the remote sensing image at the nth stage As the remote sensing image input of the n+1th stage of the feature extractor; S24: using the weighted feature map of the DSM image at the fifth stage as the final feature map of the DSM image, and using the feature map of the remote sensing image at the fifth stage as the final feature map of the remote sensing image; The segmentation module is used to input the final feature map of the DSM image and the final feature map of the remote sensing image into the multi-task decoder to generate the segmentation results of the source domain and the target domain; Step S3 is specifically as follows: S31: Input the final feature map of the DSM image into the attention-aware filtering block of the multi-task decoder to obtain the DSM image segmentation results of the source domain and the DSM image segmentation results of the target domain; input the final feature map of the remote sensing image into the ASPP of the multi-task decoder to obtain the remote sensing image segmentation results of the source domain and the remote sensing image segmentation results of the target domain; S32: Inputting the DSM image segmentation result of the source domain and the remote sensing image segmentation result of the source domain into the adaptive gated fusion block to obtain the fused segmentation result of the source domain; inputting the DSM image segmentation result of the target domain and the remote sensing image segmentation result of the target domain into the adaptive gated fusion block to obtain the fused segmentation result of the target domain; S33: The segmentation result of the source domain is composed of the DSM image segmentation result of the source domain, the remote sensing image segmentation result of the source domain, and the fusion segmentation result of the source domain; the segmentation result of the target domain is composed of the DSM image segmentation result of the target domain, the remote sensing image segmentation result of the target domain, and the fusion segmentation result of the target domain; A first updating module is configured to calculate a segmentation loss based on the segmentation result of the source domain and the semantic label image, and update the semantic segmentation model based on the segmentation loss to obtain a first updated semantic segmentation model; The second updating module is used to build a discrimination model, input the segmentation results of the source domain and the target domain into the discrimination model, calculate the adversarial loss and the discrimination loss, update the first updated semantic segmentation model by the adversarial loss to obtain the second updated semantic segmentation model, and update the discrimination model by the discrimination loss to obtain the first updated discrimination model; The final prediction module is used to construct an overall model through the second updated semantic segmentation model and the first updated identification model, train the overall model to obtain a trained overall model, input the remote sensing image dataset of the target domain into the trained second updated semantic segmentation model to obtain the final segmentation result, and perform fine post-fusion processing operations on the final segmentation result to obtain the final semantic segmentation prediction map.

Images