Remote sensing image segmentation method based on multi-dimensional attention mechanism

Abstract

The application discloses a kind of multi-dimensional attention mechanism's remote sensing image segmentation method, selects UNet3+ as basic segmentation framework, by introducing dynamic attention mechanism and light structure design is improved in view of, aims at improving the precision and efficiency of building segmentation in high-resolution remote sensing image.The method has carried out key improvement in the following three aspects: firstly, in the feature coding and enhancement path, the multidimensional context attention module (MDCA) proposed in the application is embedded in the skip connection of original UNet3+, the module fuses spatial and channel double attention, dynamically models global context dependency and channel importance, so that the network can adaptively focus on the building area and fine boundary, enhance the discrimination ability to multi-scale target and complex background.

Description

Technical Field

[0001] This invention relates to the field of semantic segmentation of remote sensing images, and specifically to a remote sensing image segmentation method based on a multidimensional attention mechanism. Background Technology

[0002] Traditional attention modules generate static weights through global pooling or convolution operations, lacking adaptive adjustment to target scale and orientation, making it difficult to dynamically adapt to contextual changes in small targets, and their high computational cost limits real-time requirements. Current feature fusion mechanisms do not effectively integrate attention mechanisms to optimize feature selection, resulting in computational redundancy. This patent proposes a dynamic attention enhancement method that enhances the feature response of small targets by adaptively adjusting channel and spatial attention weights; it also enhances the semantic information of small targets by fusing feature information from multiple levels through a multi-dimensional feature fusion method; and it designs a lightweight network structure to reduce the complexity of high-resolution remote sensing image processing, providing a new technical approach to solving the above problems. Remote sensing technology can acquire large-scale, high-resolution surface images, in which accurate and automatic building segmentation is of great significance for urban planning, change monitoring, and the construction of digital twin cities. However, building segmentation in remote sensing images faces many unique challenges: images have high spatial resolution and high dimensionality (multispectral) characteristics, and building targets exhibit high variability in scale, shape, orientation, and spatial distribution; simultaneously, complex ground backgrounds, occlusion and adhesion between buildings, and changes in shadow and illumination often make the boundary between the target and the background unclear.

[0003] Traditional segmentation methods primarily rely on manually designed features (such as spectral density, texture, and shape) and image processing algorithms (such as thresholding, edge detection, and region growing). These methods are computationally complex, have weak generalization ability, and are difficult to adapt to the high-precision segmentation requirements of different scenes and imaging conditions. With the development of deep learning, semantic segmentation models, represented by fully convolutional networks and the U-Net series, have been widely used in remote sensing image interpretation. They can automatically learn hierarchical features and achieve end-to-end pixel-level classification. However, directly applying model architectures designed for natural images to remote sensing building segmentation still has significant shortcomings: First, the receptive field of standard convolutional neural networks is limited, making it difficult to fully model the global contextual semantic relationships of large-scale buildings, resulting in an incomplete understanding of the complete building structure; second, multi-scale buildings coexist in remote sensing images, and existing models are not adaptable enough to targets of different scales when fusing features, easily causing small targets to be missed or large targets to have rough edges; third, high-resolution image processing results in a huge number of model parameters, high computational and storage costs, which is not conducive to practical deployment; finally, the number of building pixels and background pixels in images is usually severely imbalanced, and the common cross-entropy loss function will cause the model training to be biased towards the dominant background category, thereby reducing the segmentation accuracy of building boundaries.

[0004] Attention mechanisms, by assigning differentiated importance weights to different locations or channels in feature maps, enable networks to focus on key information, providing an effective way to improve model performance. However, existing attention modules (such as SE and CBAM) mostly use operations like global pooling to generate static or generalized weights, lacking the ability to dynamically and adaptively model specific task contexts (such as the unique spatial structure and spectral characteristics of buildings), and their additional parameters and computational cost may further exacerbate the model burden. Furthermore, in advanced architectures such as U-Net++ and UNet3+ that utilize dense skip connections for multi-scale feature fusion, how to optimally coordinate features at different levels with attention mechanisms to dynamically filter and enhance building-related details and semantic information remains a problem that requires further exploration.

[0005] To address the aforementioned technical bottlenecks, this invention proposes a multi-dimensional contextual attention-enhanced UNet3+ segmentation method. This method aims to introduce a multi-dimensional attention mechanism that dynamically fuses spatial and channel context, enabling the network to adaptively focus on building regions and accurately locate their boundaries. A lightweight structure enhancement module is designed to improve feature representation capabilities while optimizing computational efficiency. Furthermore, a composite loss function that jointly optimizes class imbalance and structural similarity drives the model to achieve more refined segmentation results. This invention provides an innovative solution for high-precision, high-efficiency automated building extraction from remotely sensed images. Summary of the Invention

[0006] The purpose of this invention is to provide a remote sensing image segmentation method based on a multi-dimensional attention mechanism, in order to solve the problems of limited receptive field, blurred multi-scale target edges, high model computational complexity, and inaccurate boundary segmentation caused by class imbalance in the segmentation of buildings in high-resolution remote sensing images.

[0007] To achieve the above objectives, the present invention provides the following technical solution. The present invention proposes a remote sensing image segmentation method based on a multi-dimensional attention mechanism, comprising the following steps: acquiring and preprocessing a remote sensing image dataset containing buildings; normalizing and unifying the size of the images; and performing pixel-level semantic annotation to generate binary mask labels. The processed dataset is then divided into a training set, a validation set, and a test set.

[0008] The building segmentation method based on multidimensional contextual attention enhancement described in this paper selects UNet3+ as the basic network framework. By making targeted structural improvements and loss function designs to the basic network framework, the method aims to achieve a synergistic improvement in accuracy and efficiency in building segmentation tasks of remote sensing images.

[0009] This paper improves the UNet3+ network architecture by embedding a multi-dimensional contextual attention module into its feature fusion path. By adaptively fusing attention weights across spatial and channel dimensions, it dynamically enhances building-related features and suppresses interference from complex backgrounds. This design effectively enhances the model's ability to model global semantic context and local details, improving the accuracy of edge segmentation.

[0010] This paper presents a remote sensing image segmentation method based on a multi-dimensional attention mechanism. Using the UNet3+ network model as the baseline, a lightweight structure enhancement module is proposed to optimize the encoding path. This module, by fusing depthwise separable convolutions and edge-guided mechanisms, significantly reduces model parameters and computational complexity while enhancing the feature extraction capabilities for building outlines and structural information, achieving a balance between model lightweighting and feature representation capabilities.

[0011] This paper presents a remote sensing image segmentation method based on a multidimensional attention mechanism. Addressing the problem of blurred boundary segmentation caused by a severe imbalance in the number of pixels between buildings and the background, it proposes a composite loss function that integrates semantic similarity and focus weights. This loss function dynamically adjusts the optimization objective, driving the network to simultaneously focus on the similarity between difficult samples and boundary structures, thereby refining the segmentation results.

[0012] The multidimensional contextual attention module described in this paper is a dual attention mechanism that can be embedded in skip connections, consisting of parallel spatial context units and channel relation units. To enable the network to fully utilize the contextual relation information of cross-layer features and to focus on key features while ignoring redundant information, the MDCA module performs spatial long-range dependency modeling and channel importance recalibration on the feature maps transmitted by skip connections, and then fuses the outputs of the two, which greatly improves the ability to represent multi-scale features of buildings in complex scenes.

[0013] The lightweight structure enhancement module described in this paper significantly reduces computational load by improving standard convolutional layers, while maintaining strong representational capabilities. The core of the SEGB module consists of depthwise separable convolutional units and edge-guided units. Depthwise separable convolutions reduce the number of parameters by decomposing computations in spatial and channel dimensions; edge-guided units enhance contour perception by integrating high-frequency information. Embedding these into the encoding path effectively preserves the structural details of buildings while achieving lightweight design.

[0014] The composite loss function described in this paper, which integrates semantic similarity and focus weights, is primarily composed of a focus classification loss and a multi-scale structural similarity loss. The focus classification loss dynamically reduces the weight of easily classified samples through a modulation factor, focusing training on difficult-to-classify samples (such as boundary pixels). The multi-scale structural similarity loss measures the structural consistency between the predicted image and the ground truth label at multiple scales, particularly optimizing the visual fidelity of boundary regions. The weighted joint optimization of these two losses drives model performance improvement from both pixel classification and structural preservation perspectives.

[0015] In summary, this paper proposes a remote sensing image segmentation method based on a multi-dimensional attention mechanism, selecting UNet3+ as the baseline network. The multi-dimensional contextual attention module is embedded into the skip connection path to dynamically filter and enhance multi-scale features from the encoder, serving as the foundation of the algorithm and improving the model's ability to focus on building targets and their boundaries. The lightweight structure enhancement module described in this paper is integrated into the encoder, achieving a balance between efficiency and accuracy in the backbone feature extraction stage. The composite loss function described in this paper is applied during the training stage, effectively solving the problems of class imbalance and edge blurring by jointly optimizing classification difficulty and structural similarity. Through the synergistic effect of these improvements, the proposed method significantly improves the boundary integrity, inter-class discriminative power, and model efficiency of remote sensing image building segmentation. Attached Figure Description

[0016] Figure 1 This invention provides an improved MDCA-UNet3+ network structure diagram based on the U-Net3+ neural network.

[0017] Figure 2 This is a structural diagram of the jump connection portion of U-Net3+ used in this invention.

[0018] Figure 3 This is a schematic diagram of an MDCA attention mechanism module provided by the present invention.

[0019] Figure 4 A schematic diagram of an SEGB module provided by the present invention. Figure 5 This invention provides a schematic diagram of a combined SEGB+MDCA application. Figure 6 This invention provides a technical roadmap for a remote sensing image building segmentation algorithm. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application.

[0022] The experimental data in this embodiment is constructed based on the publicly available WHU building dataset and Inria aerial image dataset. The WHU dataset provides high-resolution optical remote sensing images and their corresponding pixel-level building annotations, suitable for the model to learn fine segmentation of regular buildings. The Inria dataset contains remote sensing images of different cities and different seasons, with diverse scenes, which helps to improve the model's generalization ability. Before model training, the raw data underwent systematic preprocessing: Data cleaning and screening: Samples with low image quality (such as severe cloud and fog obscuration) and incomplete annotations were removed to ensure data quality. Format unification: All images and their corresponding label masks were uniformly scaled to a fixed size (such as 512×512 pixels), and pixel values ​​were normalized to the [0,1] range. Data augmentation: To prevent overfitting and improve model robustness, the training set was augmented online in real time, including: random horizontal / vertical flipping, random rotation (-15 to +15), random scaling (0.8 to 1.2 times), and brightness / contrast fine-tuning. Dataset partitioning: The preprocessed data was randomly divided into training, validation, and test sets in a ratio of approximately 7:2:1. The validation set is used for hyperparameter tuning and training process monitoring, while the test set is used for final performance evaluation.

[0023] This embodiment uses UNet3+ as the baseline network architecture and makes three core improvements to it, constructing the MDCA-UNet3+ model. The model is implemented in the PyTorch framework.

[0024] This paper proposes a multidimensional context attention module, which is the core of this invention to improve the network context modeling capability. Its structure is designed as follows: Input and parallel path: given input feature map The MDCA module obtains the result by averaging the feature maps in the vertical, horizontal, and channel directions. The specific equations are as follows: Where i, j, and k represent the indices of the vertical, horizontal, and channel directions, respectively. The H and W matrices are multiplied to obtain a similarity matrix, and then the spatial attention map is calculated using the SoftMax function. .

[0025] To comprehensively capture channel dependencies and learn the nonlinear and non-exclusive relationships between channels, C is sequentially passed through a fully connected layer, a ReLU activation function, another fully connected layer, and a sigmoid activation function to obtain the channel attention weights. in, Represents the ReLU activation function. This represents the sigmoid activation function. and These are the weight coefficients of the fully connected layer, and r is a scaling parameter, set to 16 in this paper. This parameter reduces the number of channels, thereby lowering computational complexity. Subsequently, the spatial attention map and channel attention weights are fused with the input features through matrix multiplication to generate features. Finally, by adding these two features, the module output is obtained. .

[0026] This paper proposes a lightweight structure enhancement module, which aims to enhance feature extraction capabilities without significantly increasing computational burden. Specifically, the SEGB module retains the lightweight bottleneck path from 1×1 dimensionality reduction to Depthwise + Pointwise convolution to 1×1 dimensionality upscaling, and adds an auxiliary edge branch. This branch first performs channel compression on the input feature map, then extracts edge structure features through a learnable EdgeConv (such as 3×3 convolution + BN + ReLU), and generates an edge weight map after 1×1 convolution and Sigmoid activation. This weight map not only participates in feature modulation (such as...) It also provides explicit supervision with the GroundTruth edge map to optimize the learning ability of the target structural region.

[0027] The SEGB module has approximately 42,000 parameters, which is only slightly more than BP-Bottleneck (about 12%). However, it significantly improves the model's ability to identify fine-grained boundaries and express structural integrity, making it particularly suitable for boundary-sensitive tasks such as remote sensing building segmentation.

[0028] In multi-scale semantic segmentation networks like UNet3+, an improved bottleneck structure is introduced in the encoder, which not only ensures rich feature representation but also significantly reduces computational complexity and memory consumption, promoting rapid convergence and efficient inference when processing complex data such as high-resolution remote sensing images. Simultaneously, this lightweight design makes it possible to deploy the model on resource-constrained edge devices. Introducing the SEGB module into the UNet3+ framework as a feature enhancement unit for the encoder and skip connection paths not only improves the model's edge localization capabilities and building contour recognition accuracy but also effectively controls model complexity, adapting to the dual requirements of high-resolution remote sensing image segmentation and edge device deployment.

[0029] This paper proposes a structure-aware composite loss function. Traditional cross-entropy loss functions assign equal weights to all samples when dealing with imbalanced data, ignoring differences in sample difficulty and potentially neglecting the learning of a small number of samples. To address the problem of imbalanced building category weights in remote sensing imagery, this paper proposes to replace the original Unet3+ model's loss function with a FSAL design combining focalLoss and MS-SSIM loss functions.

[0030] FocalLoss is a loss function used to address class imbalance problems and is widely used in tasks such as object detection and semantic segmentation. Traditional cross-entropy loss is easily affected by the majority class when dealing with class imbalance, leading to poor model performance in classifying the minority class. FocalLoss introduces an adjustable parameter γ to more strongly penalize misclassifications of the minority class, thereby improving the model's ability to classify the minority class.

[0031] The traditional cross-entropy loss function has the following form: In the above formula, y takes values ​​of 1 and -1, representing the foreground and background respectively. p ranges from 0 to 1 and represents the probability that the model predicts the element to be in the foreground. Next, we define a function of p: Thus, the CE function can be expressed as: The BCE loss function, a common method for addressing class imbalance, introduces a weighting factor α ∈ [0, 1]. When the sample is positive, the weighting factor is α; when the sample is negative, the weighting factor is 1 - α. Therefore, the loss function can also be rewritten as: While BCE addresses the imbalance between positive and negative samples, it doesn't differentiate between easily distinguishable and difficult-to-distinguish samples. When there are an overwhelming number of easily distinguishable negative samples, the entire training process will revolve around them, overwhelming the positive samples and causing significant loss. Therefore, a modulation factor is introduced to focus on difficult-to-distinguish samples, as shown in the following formula: γ is a parameter ranging from [0, 5]. When γ is 0, it becomes the initial CE loss function. This can reduce the loss contribution of easily distinguishable samples, thereby increasing the loss proportion of difficult-to-distinguish samples. When pt approaches 1, it indicates that the sample is easily distinguishable, and the modulation factor approaches 0, indicating a smaller contribution to the loss, i.e., reducing the loss proportion of easily distinguishable samples. When pt is very small, that is, if a sample is classified as a positive sample, but the probability of that sample being the foreground is extremely small, i.e., it is misclassified as a positive sample, the modulation factor approaches 1, and it has little impact on the loss.

[0032] The technical features of the above embodiments can be combined in any way. In order to keep the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in these combinations of technical features, they should be considered to be within the scope of this specification.

[0033] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A remote sensing image segmentation method with a multidimensional attention mechanism, characterized in that, The steps are as follows: S1: Dataset Acquisition and Preprocessing: The publicly available WHU building dataset and Inria remote sensing dataset were selected as benchmarks. Necessary preprocessing operations, such as normalization and size unification, were performed on the images to provide high-quality input for model training. S2: Perform semantic segmentation and labeling on the images in the dataset to ensure that each pixel is accurately labeled as "building" or "background", and generate corresponding binary label masks for supervised model learning; S3: Data augmentation of training set images is performed through random rotation, flipping, scaling, and color jittering to increase sample diversity and improve the model's generalization ability to buildings under different scales, orientations, and lighting conditions. S4: Divide the dataset into training, validation, and test sets proportionally, and use them for model parameter training, hyperparameter tuning, and final performance evaluation, respectively, to ensure the objectivity and reliability of the evaluation results. S5: The MDCA-UNet3+ remote sensing image building segmentation method described above uses UNet3+ as the baseline network and systematically optimizes feature extraction, context aggregation and boundary refinement capabilities by integrating a multi-dimensional context attention mechanism, a lightweight structure enhancement module and a composite loss function. S6: A multidimensional contextual attention module (MDCA) is embedded in the skip connection path to dynamically adjust feature weights by fusing spatial and channel attention, thereby enhancing the model's modeling of global semantics and local boundaries of buildings; a lightweight structure enhancement module (SEGB) is introduced in the encoding path, which uses depthwise separable convolution and edge guidance mechanisms to enhance structure awareness while reducing the number of parameters; a composite loss function (FSAL) that fuses semantic similarity and focus weights is designed and applied to alleviate class imbalance and optimize boundary segmentation quality; finally, the segmentation accuracy, boundary integrity, and inference efficiency of the improved model are evaluated on the test set.

2. The remote sensing image segmentation method based on the multidimensional attention mechanism according to claim 1, characterized in that, The model's performance is evaluated using the crossover-union ratio (CUNR), Dice coefficient, precision, recall, and mean CUNR as core metrics.

3. The remote sensing image segmentation method based on the multidimensional attention mechanism according to claim 2, characterized in that, The intersection-over-union ratio (IoU) is used to quantify the overlap between the predicted region and the ground truth region; the Dice coefficient is used to measure the similarity between the predicted segmentation result and the ground truth region, and it has good robustness to class imbalance problems; the average IoU is used to comprehensively evaluate the overall segmentation performance of the model by averaging the IoU of all categories.

4. The remote sensing image segmentation method based on the multidimensional attention mechanism according to claim 1, characterized in that, The multidimensional contextual attention module is embedded in the skip connection path of the UNet3+ network to dynamically integrate attention weights of spatial and channel dimensions. This adaptively enhances the building feature representation transmitted by the encoder and suppresses irrelevant background information, thereby improving the segmentation accuracy of the decoder when reconstructing details.

5. The remote sensing image segmentation method based on the multidimensional attention mechanism according to claim 1, characterized in that, The lightweight structure enhancement module is deployed in the encoder path. By integrating depthwise separable convolution and edge-guided supervision mechanisms, it significantly enhances the network's ability to extract and perceive building outlines and structural information while effectively compressing the number of model parameters and computational complexity.

6. The remote sensing image segmentation method based on the multidimensional attention mechanism according to claim 1, characterized in that, The composite loss function that integrates semantic similarity and focus weights achieves dynamic gradient adjustment during model training by weighting FocalLoss and multi-scale structural similarity loss, thereby synergistically optimizing the mitigation of class imbalance and the refinement of segmentation boundary details.

Images