Unsupervised domain adaptation method for semantic segmentation of high-resolution remote sensing images

Abstract

The present invention discloses an unsupervised domain adaptation method for semantic segmentation of high-resolution remote sensing images, which belongs to the field of computer vision and remote sensing image processing. The method first obtains a labeled source domain dataset and an unlabeled target domain dataset, then uses a feature extractor to extract multi-level features, completes the dual-domain feature interaction through a bidirectional recursive attention pyramid module, discriminates features through a multi-discriminator module, uses segmentation loss, global feature difference loss, domain confusion loss and adversarial loss backpropagation calculation, and finally deploys the trained model to the target domain to achieve semantic segmentation. The invention solves the problems of domain offset and insufficient feature alignment granularity in the semantic segmentation of remote sensing images in existing unsupervised domain adaptation methods through a bidirectional recursive attention pyramid and a multi-discriminator module, and effectively improves the segmentation performance of the model in the target domain.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and remote sensing image processing, and in particular to an unsupervised domain adaptation method based on a bidirectional recursive attention pyramid and a multi-discriminator module for semantic segmentation of high-resolution remote sensing images. Background Art

[0002] With the rapid development of drones and remote sensing technology, high-resolution remote sensing imagery has been widely used in infrastructure planning, land management, urban change detection, and other fields. Semantic segmentation, a key technology in remote sensing image processing, aims to classify each pixel in an image into predefined categories. However, due to the complexity and diversity of remote sensing images, traditional supervised learning methods require large amounts of labeled data and struggle to cope with differences between different domains in practical applications.

[0003] Unsupervised domain adaptation (UDA) is a method that adapts to unlabeled data in the target domain by leveraging labeled data in the source domain, thereby reducing the reliance on labeled data in the target domain. However, existing UDA methods face the following challenges when processing remote sensing images:

[0004] Domain shift problem: The feature distribution difference between the source domain and the target domain is large, which leads to the deterioration of the model performance on the target domain.

[0005] Insufficient granularity of feature alignment: Existing methods often ignore fine-grained semantic information during feature alignment, resulting in insufficient adaptability of the model to complex scenarios.

[0006] To solve the above problems, the present invention proposes an unsupervised domain adaptation method based on a bidirectional recursive attention pyramid and a multi-discriminator module. It realizes fine-grained feature interaction between the source domain and the target domain through multi-scale pyramid decomposition, recursive bidirectional attention bridge and dynamic gated fuser, and uses a multi-discriminator module to further enhance the discriminative ability of features, thereby effectively solving the domain shift problem.

[0007] Compared to natural images, remote sensing images exhibit greater cross-domain variation in color, texture, spatial resolution, and scene content. These differences are attributed to two factors: First, remote sensing images exhibit greater spectral heterogeneity than natural images acquired in RGB bands. Furthermore, remote sensing images contain richer structural information and more significant object scale variations. Consequently, the performance of these UDA methods is less than satisfactory when directly applied to large variations in camera angle and image resolution. The substantial growth in the number and variety of remote sensing sensors over the past decade, coupled with the availability of higher-quality VHR data, has further widened the disparity between domains. Recently, several DL-based UDA methods have been applied to remote sensing tasks. For example, GANAI is the first GAN-based UDA algorithm designed for semantic segmentation of aerial imagery, while TriADA introduces a class-aware self-training technique to enhance its discriminability. WTIC further investigated invariant semantic features in VHR remote sensing imagery and proposed a dynamic training strategy by incorporating multiple weakly supervised constraints. Recently, subspace alignment, coupled with a shallow CNN-based feature extractor, has been designed to explore and align rich semantic vector subspaces, which can further represent and analyze complex image content. In particular, CCAGAN adopts global and category-level adversarial losses to enforce local semantic consistency by employing negative transfer during domain adaptation and a category attention module for processing deep features. Although these pioneering works have performed well in cross-domain semantic segmentation, they still face two major problems when processing remote sensing images. First, existing methods are insufficient in dual-domain information interaction and fail to fully explore the potential correlation between the source and target domains, resulting in limited feature alignment effects. This is mainly because the complexity of remote sensing images, including their spectral heterogeneity, rich structural information, and significant object scale variations, makes it difficult for simple feature alignment strategies to capture the complex semantic relationship between the source and target domains. Second, the granularity of feature alignment is relatively coarse and cannot accurately capture fine-grained semantic information, thereby affecting the model's adaptability to complex scenes and segmentation accuracy. Summary of the Invention

[0008] This paper aims to provide an unsupervised domain adaptation method for semantic segmentation of high-resolution remote sensing imagery, based on a bidirectional recursive attention pyramid (BRAP) and a multi-discriminator module (MDM). This method uses the BRAP module to achieve fine-grained feature interaction between the source and target domains, and utilizes the MDM module to further enhance the discriminative power of features, effectively addressing the domain shift problem.

[0009] To achieve the above objectives, an embodiment of the present invention provides an unsupervised domain adaptation method based on a bidirectional recursive attention pyramid and a multi-discriminator module, comprising the following steps:

[0010] S1: Obtain a labeled source domain dataset and an unlabeled target domain dataset;

[0011] S2: Extract multi-level features from source and target domain data through feature extractor;

[0012] S3: The extracted features are input into the bidirectional recursive attention pyramid module, and the dual-domain feature interaction is completed through multi-scale pyramid decomposition, recursive bidirectional attention bridge and dynamic gated fuser;

[0013] The bidirectional recursive attention pyramid module includes a multi-scale pyramid decomposition, a recursive bidirectional attention bridge, and a dynamic gated fuser. The multi-scale pyramid decomposition decomposes the feature map into multi-scale feature maps with sizes of 1×1, 3×3, 6×6, and the original size. The recursive bidirectional attention bridge realizes cross-domain closed-loop interaction through forward and reverse attention transfer between levels. The dynamic gated fuser aggregates features of each scale through dynamic gate weights.

[0014] S4: discriminate the interacted features through the multi-discriminator module;

[0015] The multi-discriminator module includes a feature discriminator, an adversarial discriminator, and a category discriminator. The feature discriminator is used to confuse dual-domain features, the adversarial discriminator is used to achieve adversarial learning of overall features, and the category discriminator completes feature adversarial learning at the category level to further improve the feature discrimination ability.

[0016] S5: Perform backpropagation calculations through the defined segmentation loss, global feature difference loss, domain confusion loss, and adversarial loss;

[0017] S6: Deploy the trained model to the target domain to achieve semantic segmentation of high-resolution remote sensing images in the target domain.

[0018] Furthermore, the bidirectional recursive attention pyramid module consists of three parts: a multi-scale pyramid decomposition unit, a recursive bidirectional attention interaction unit, and a dynamic gating fusion unit. The multi-scale pyramid decomposition unit is used to decompose the source domain and target domain features into hierarchical representations of different granularities. The recursive bidirectional attention interaction unit realizes deep coupling of dual-domain semantics and details through cross-level closed-loop transmission. The dynamic gating fusion unit aggregates the aligned features of each level through adaptive weights to output a unified cross-domain adaptation result.

[0019] Furthermore, the multi-scale pyramid decomposition unit decomposes the input feature map into four layers of feature representations with different resolutions through a spatial pyramid pooling strategy, including a 1×1 global semantic descriptor, a 3×3 medium local structure feature, a 6×6 fine-grained detail feature, and a high-resolution feature of the original size, respectively covering the scale differences from macro scenes to micro targets in remote sensing images, while retaining the original resolution features to avoid the loss of small target information.

[0020] Furthermore, the recursive bidirectional attention interaction unit realizes cross-level closed-loop interaction in three iterations, specifically including two paths of forward transfer and backward transfer: in the forward transfer, the query and key of the low-level features of the source domain act on the value matrix of the high-level features of the target domain, injecting global semantics into the detailed features of the target domain; in the backward transfer, the query and key of the high-level features of the target domain act on the value matrix of the low-level features of the source domain, and feed back the adapted local information to the global features of the source domain; in this process, the target domain value matrix and the source domain query matrix are mixed through the cross-domain residual connection to retain the cross-domain commonalities and differences and ensure the stability of feature alignment.

[0021] Furthermore, after upsampling the aligned features of each level, the dynamic gated fusion unit dynamically assigns weights based on feature similarity, prioritizes retaining the hierarchical features with high alignment quality, and finally generates adapted high-resolution features through weighted fusion; the unit generates anchor features through global average pooling as the weight calculation benchmark, and adaptively adjusts the contribution of each level in combination with the Softmax function, thereby balancing the importance differences of multi-scale features in cross-domain adaptation and improving the robustness of the model to complex scenarios.

[0022] Furthermore, the global feature difference loss Lglobal is obtained by calculating the difference between the high-level aligned feature maps of the source domain and the target domain; the domain confusion loss Ldc is obtained by taking the source domain true value label as the target domain true value, confusing the dual-domain feature distribution, and calculating the difference between the target domain prediction result and the source domain true value label; the segmentation loss Lseg is obtained by calculating the difference between the source domain feature and the true value boundary; the category confusion loss Lmc compares the mean difference of the dual-domain category features in the high-dimensional reproducing kernel Hilbert space.

[0023] Furthermore, the feature adversarial loss Ladv trains the discriminator ability by discriminating whether the feature comes from the source domain or the target domain, thereby realizing overall feature adversarial. The category adversarial loss Lamc judges the feature source at the category level to realize fine-grained feature alignment.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] This paper addresses the shortcomings of existing unsupervised domain adaptation methods in remote sensing image semantic segmentation through a bidirectional recursive attention pyramid interaction module and a multi-discriminator module. BRAP captures global-local dependencies between the source and target domains through a multi-scale pyramid bidirectional closed-loop interaction between the two domains, enabling fine-grained feature interaction. MDM uses multiple discriminators to discriminate features from different perspectives, further enhancing the discriminative power of features, effectively addressing the domain shift problem and improving the segmentation performance of the model in the target domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0027] Figure 1 : A framework flow chart of an unsupervised domain adaptation method for semantic segmentation of high-resolution remote sensing images provided by the first embodiment of the present invention;

[0028] Figure 2 : A schematic structural diagram of a bidirectional recursive attention pyramid module provided by the first embodiment of the present invention;

[0029] Figure 3 : A schematic structural diagram of a multi-discriminator module provided in the first embodiment of the present invention;

[0030] Figure 4 : A flowchart of the steps of an unsupervised domain adaptation method for semantic segmentation of high-resolution remote sensing images provided by the first embodiment of the present invention; DETAILED DESCRIPTION

[0031] The specific implementation of the present invention is as follows, which describes in detail the entire process from data acquisition to model deployment, ensuring that each step and technical details are fully explained.

[0032] like Figure 1 As shown, a framework flow chart of an unsupervised domain adaptation method for semantic segmentation of high-resolution remote sensing images provided by the first embodiment of the present invention includes:

[0033] S1: Get the dataset

[0034] Obtain a labeled source domain dataset Ds = {(xs, ys)} and an unlabeled target domain dataset Dt = {xt}, where xs and ys are samples in the source domain and their corresponding labels, respectively, and xt is a sample in the target domain;

[0035] S2: Feature Extraction

[0036] The DeepLab-v2 framework, based on ResNet-101 and pre-trained on the ImageNet dataset, is used as a feature extractor. The feature extractor extracts multi-level features from the source and target domain data, outputting high-level feature maps Fs and Ft with abstract semantic information. The output is of size 8 / H×8 / W×D, where D is the feature dimension (D=2048), and H and W are the height and width of the image, respectively.

[0037] S3: Feature Interaction

[0038] The extracted features Fs and Ft are input into the bidirectional recursive attention pyramid module, and fine-grained feature interaction is achieved through multi-scale pyramid decomposition, recursive bidirectional attention bridge and dynamic gated fuser. The details are as follows:

[0039] Multi-scale Pyramid Decomposition: The feature maps of the source and target domains are decomposed into four layers of feature maps of different resolutions using spatial pyramid pooling. The feature map sizes of each layer are 1×1, 3×3, 6×6, and the original size. The L1 layer extracts global semantic descriptors through global average pooling. The L2 layer is divided into 3×3 sub-regions and average pooled to capture medium-scale local structure. The L3 layer is divided into 6×6 sub-regions and pooled to preserve fine-grained local details. The L4 layer directly retains the high-resolution feature map to avoid information loss caused by downsampling.

[0040] Recursive Bidirectional Attention Bridge: The recursive design breaks through the limitations of traditional unidirectional attention and enables deep coupling of dual-domain features at the spatial and semantic levels. Each recursive iteration consists of two paths, forward and reverse, and a total of three iterations are performed to gradually deepen cross-domain interaction. In each of the three iterations, the i-th layer features of the source domain act on the i+1th layer features of the target domain through the attention mechanism, and the i-th layer features of the target domain are fed back to the i-1th layer features of the source domain through reverse attention, forming a cross-level closed-loop interaction.

[0041] Forward pass: query matrix Qs of the source domain i-th layer (i) and bond matrix Ks (i) The cross-domain residual connection matrix Vmixedt with the value of the target domain i+1 layer (i+1) Perform attention calculation, the formula is:

[0042]

[0043] Backward pass: query matrix Qt of the target domain i-th layer (i) and bond matrix Kt (i) The cross-domain residual connection matrix Vmixeds with the value of the target domain i-1 layer (i-1) Perform attention calculation, the formula is:

[0044]

[0045] The recursive bidirectional attention bridge introduces a cross-domain residual connection in each iteration, and mixes the value matrix Vt of the target domain with the query matrix Qs of the source domain through a skip connection during the forward propagation process:

[0046] V mixedt =V t +γMLP(Q S )

[0047] In the reverse propagation process, the value matrix Vs of the source domain is mixed with the query matrix Qt of the target domain through skip connections:

[0048] V mixeds =V s +γMLP(Q t )

[0049] Where γ is a learnable scaling factor and MLP is used for dimensionality matching.

[0050] The dynamic gated fusion upsamples the features of each layer to the original size at the top of the pyramid and uses the similarity weight β k Weighted summation, the formula is:

[0051]

[0052] Among them, Fanchor is the anchor feature after global average pooling, which is used to measure the importance of features at each level. The final output is:

[0053]

[0054] like Figure 2 As shown, the present invention provides a bidirectional recursive attention pyramid module for interacting with local and global information in both domains. Pyramid decomposition effectively addresses the large scale variations of objects in high-resolution remote sensing images, while the recursive mechanism enhances cross-level information flow. The recursive design overcomes the limitations of traditional unidirectional attention, enabling deep coupling of dual-domain features at both spatial and semantic levels. The gating mechanism avoids manual setting of layer weights, improving the model's adaptability to complex scenarios.

[0055] S4: The aligned features are discriminated through the multi-discriminator module;

[0056] The multi-discriminator module includes a feature discriminator, an adversarial discriminator, and a category discriminator. The feature discriminator is used to process high-level alignment features, the adversarial discriminator is used to achieve adversarial learning of overall features, and the category discriminator completes feature adversarial learning at the category level to further improve the feature discrimination ability.

[0057] S5: Perform backpropagation calculations through the defined segmentation loss, global feature difference loss, and adversarial loss;

[0058] like Figure 3 、 Figure 4 As shown, the present invention provides a multi-discriminator module, including a feature discriminator, an adversarial discriminator, and a category discriminator. The feature discriminator and the adversarial discriminator are composed of four self-attention modules, which enhance the discriminator's focus on semantic information. The category discriminator consists of a decoder and an encoder. Label information from the source domain is added to the encoded features during the encoding phase and passes through a self-attention module to achieve adversarial learning at the category level.

[0059] Global feature difference loss Lglobal: This loss function is used to align the high-level feature maps of the source domain and the target domain to reduce the distribution difference between the two domains. The maximum mean difference is used to calculate this loss:

[0060]

[0061] and are the high-level feature maps of the source and target domains after processing by the bidirectional recursive attention pyramid module. P represents the average pooling operation on the feature maps of each channel to extract spatial information. λg is a weight coefficient used to balance the contribution of this loss in the overall optimization.

[0062] Boundary loss Lseg: This loss function is used to ensure that the generator can accurately perform semantic segmentation on the source domain image. It is calculated based on the cross entropy loss, and the specific formula is:

[0063]

[0064] Ps and are the predicted probability maps of the classifier for the source domain image. Ys is the true label of the source domain image.

[0065] Domain confusion loss Ldc: This loss function is used to deceive the feature discriminator Df, causing it to misjudge the feature map of the target domain as the feature map of the source domain. The specific formula is:

[0066]

[0067] is the prediction result of the feature discriminator on the target domain feature map. λf is the weight coefficient.

[0068] Category confusion loss Lmc: compares the mean difference of dual-domain category features in the high-dimensional reproducing kernel Hilbert space. The loss function uses the maximum mean difference. The specific formula is:

[0069]

[0070] φ(·): Kernel function (such as Gaussian kernel, polynomial kernel) mapped to high-dimensional space. The Hilbert space corresponding to the kernel function.

[0071] These four loss functions are designed to train the generator, while the discriminator parameters are fixed and will not be updated during the generator training phase.

[0072] Feature adversarial loss Ladv: This loss function is used to train the feature discriminator Df so that it can distinguish the feature maps of the source domain and the target domain. The specific formula is as follows:

[0073]

[0074] and They are the prediction results of the feature discriminator on the source domain and target domain feature maps, respectively.

[0075] Category adversarial loss Lamc: The prediction results incorporate the true category information from the source domain, determine the source of features at the category level, and achieve fine-grained feature alignment. The specific formula is as follows:

[0076]

[0077] and are the prediction results of the feature discriminator for each category of the source domain and target domain feature maps, respectively. λk is the weight parameter of the proportion of the loss calculation result of each category in the entire loss function.

[0078] These two loss functions are designed to train the discriminator, improving its discriminative performance and enhancing the effectiveness of adversarial learning. During the discriminator training phase, the generated parameters are frozen and not updated. During the training phase, the generator and discriminator are trained alternately.

[0079] S6: Model Deployment

[0080] The trained model is deployed to the target domain to achieve semantic segmentation of high-resolution remote sensing images in the target domain.

[0081] The foregoing description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. An unsupervised domain adaptation method for semantic segmentation of high-resolution remote sensing images, characterized by: At least the following steps are included: S1: Obtain a labeled source domain dataset and an unlabeled target domain dataset; S2: Extract dual-domain features from source and target domain data through feature extractor; S3: The extracted features are input into the bidirectional recursive attention pyramid module, and the dual-domain feature interaction is completed through multi-scale pyramid decomposition, recursive bidirectional attention bridge and dynamic gated fuser; The bidirectional recursive attention pyramid module includes multi-scale pyramid decomposition, recursive bidirectional attention bridge, and dynamic gated fuser. The dual-domain feature decomposes the feature map into a multi-scale feature map through multi-scale pyramid decomposition. The recursive bidirectional attention bridge is used for the dual-domain multi-scale features to perform forward and reverse attention transfer between levels to achieve cross-domain closed-loop interaction. The dynamic gated fuser aggregates features of each scale through dynamic gate weights. Multi-scale pyramid decomposition decomposes the dual-domain features into four layers of feature maps with different resolutions through spatial pyramid pooling, with each layer having sizes of 1×1, 3×3, 6×6, and the original size; The recursive bidirectional attention bridge iterates three times between the dual-domain multi-scale features. The forward pass applies the source domain i-th layer feature to the target domain i+1-th layer feature through the attention mechanism. The formula is: ， The reverse transfer feeds back the target domain i-th layer feature to the source domain i-th layer feature through reverse attention. The formula is: ， The forward and reverse transfer between the three-dimensional dual-domain features forms a cross-level closed-loop interaction; The dynamic gated fusion layer upsamples the features of each layer to the original size at the top of the pyramid and uses the similarity weight Weighted summation, the formula is: ， in It is the anchor feature after global average pooling, which is used to measure the importance of features at each level; S4: discriminate the interacted features through the multi-discriminator module; The multi-discriminator module includes a feature discriminator, an adversarial discriminator, and a category discriminator. The feature discriminator is used to confuse dual-domain features, the adversarial discriminator is used to achieve adversarial learning of overall features, and the category discriminator completes feature adversarial learning at the category level to further improve the feature discrimination ability. S5: Perform backpropagation calculations through the defined segmentation loss, global feature difference loss, domain confusion loss, and adversarial loss; S6: Deploy the trained model to the target domain to achieve semantic segmentation of high-resolution remote sensing images in the target domain.

2. The unsupervised domain adaptation method for semantic segmentation of high-resolution remote sensing images according to claim 1, characterized in that: The recursive bidirectional attention bridge introduces a cross-domain residual connection in each iteration, and the value matrix of the target domain is converted into Query matrix with source domain Mixing via skip connections: ， In the back propagation process, the value matrix of the source domain is and the query matrix of the target domain Mixing via skip connections: ， in is a learnable scaling factor, Used for dimension matching.

3. The unsupervised domain adaptation method for semantic segmentation of high-resolution remote sensing images according to claim 1, wherein: The feature discriminator uses the source domain true value label as the target domain true value to confuse the dual-domain feature distribution. The adversarial discriminator trains the discriminator by discriminating the dual-domain features, thereby enhancing the discriminant ability and improving the model adversarial performance. The category discriminator first decodes the features to the original input image size, and then inputs them into the encoder together with the category information of the source domain. After the category information undergoes multi-level feature extraction, the obtained high-level semantic information is fused with the features to complete the category information access, thereby achieving feature adversarial performance from the category level. The feature discriminator and adversarial discriminator are composed of four self-attention modules, which enhance the discriminator's attention to semantic information. The category discriminator consists of a decoder, an encoder, and a self-attention module. The label information from the source domain is accessed to the encoded features during the encoding stage and passes through a self-attention module to achieve adversarial learning at the category level.

Images