A remote sensing image segmentation method based on integrated cross-attention mechanism

Abstract

The application discloses a remote sensing image segmentation method based on an integrated cross attention mechanism. The application adopts a deep residual network ResNet101 to perform feature extraction on an RGB remote sensing image, and obtains an initial feature map; a cross attention mechanism is introduced, the original image and the initial feature map are taken as inputs, the region correlation of the RGB remote sensing image is mined, the key region features of target recognition are strengthened, the initial feature map is subjected to feature enhancement, and a focused feature map is output; a deep learning model DeepLabV3 is adopted to perform multi-scale feature extraction on the focused feature map, context features under different receptive fields are captured through an atrous spatial pyramid pooling module ASPP, and the context features are fused with low-layer detail features from the ResNet101, so that the final features with rich context and clear boundaries are obtained, and a semantic segmentation result is output. The application improves the segmentation capability in a complex scene through the cross attention mechanism, and can effectively improve the segmentation precision of complex urban and rural objects by using image space and context information.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a remote sensing image segmentation method based on an integrated cross-attention mechanism. Background Technology

[0002] In the fields of ecological environmental protection and territorial spatial planning, high-precision urban and rural land cover classification is a crucial foundation for resource monitoring and decision-making management. Traditional manual survey methods are inefficient, costly, and time-consuming, making it difficult to meet the needs of large-scale dynamic monitoring. High-resolution multispectral remote sensing imagery, with its rich details and convenient acquisition, has become the mainstream data source for land cover information extraction. However, it still faces severe challenges in practical applications: urban and rural land cover types are complex, with significant intra-class differences and blurred inter-class boundaries, and are easily affected by factors such as illumination and atmosphere, resulting in prominent issues of "same object, different spectrum" and "different object, same spectrum"; in addition, there are significant domain differences between urban and rural areas in terms of landscape patterns and land cover scales, making it difficult for a single model to adapt to both scenarios simultaneously. Traditional methods rely on shallow features or conventional convolutional structures, which are insufficient for modeling complex contexts, multi-scale spatial structures, and cross-domain differences, easily leading to misclassification, omissions, and blurred boundaries. Therefore, it is urgent to innovate at the level of deep learning model structure to improve the model's feature representation, domain adaptation, and fine segmentation capabilities in complex urban and rural remote sensing scenarios.

[0003] To address the challenges of multi-scale feature capture and resolution control in image segmentation, Chen et al. proposed a dilated convolution solution. By adjusting the dilation rate to control the receptive field of the convolution kernel, they designed two modules: cascaded and parallel (dilated spatial pyramid pooling (ASPP)). These modules respectively deepen the capture of long-distance spatial features and cover multi-scale target features. At the same time, they introduced image-level features to enhance global context modeling, alleviating the problem of effective weight degradation of the convolution kernel under a large dilation rate. This enabled the model to achieve excellent semantic segmentation performance without post-processing.

[0004] Based on the above research, Chen et al. optimized and constructed an encoder-decoder architecture: the encoder uses DeepLabv3's dilated convolution and ASPP module to extract high-level semantic features, and the decoder fuses low-level details and high-level semantic features to improve the segmentation accuracy of the target boundary; they introduced depthwise separable convolution to reduce computation and maintain performance, improved the Xception network as the backbone model, and enhanced feature extraction efficiency by replacing the pooling layer with depthwise separable convolution with stride, thus achieving a balance between segmentation accuracy and efficiency on general datasets.

[0005] While existing technologies excel in general semantic segmentation, they still suffer from three limitations when applied to urban and rural remote sensing scenarios: First, they rely excessively on hyperspectral features, failing to adapt to the patterns in RGB images, leading to misclassification and confusion of similar features; second, the multi-scale fusion module lacks the ability to collaboratively model global and local features, resulting in blurred boundaries and missed classification of small targets; and third, the models are mostly designed for single urban scenes, lacking domain adaptability and failing to account for the urban-rural differences in the LoveDA dataset. Improvements are urgently needed to enhance the segmentation accuracy and robustness in complex urban and rural remote sensing scenarios. Summary of the Invention

[0006] This invention aims to address the problems existing in the fine segmentation of urban and rural remote sensing images: traditional methods rely too heavily on hyperspectral features, making it difficult to adapt to RGB multispectral images and easily causing misclassification and confusion of visually similar ground objects; existing models have weak collaborative modeling capabilities for global and local features, easily leading to blurred ground object boundaries and missed classification of small-scale targets; at the same time, the models have insufficient domain adaptability and cannot simultaneously adapt to urban and rural differentiated scenarios, and the segmentation accuracy and robustness are difficult to meet the requirements of high-precision land cover classification.

[0007] To address this, this invention proposes a remote sensing image segmentation method based on an integrated cross-attention mechanism. Utilizing deep learning-based remote sensing image segmentation and processing techniques, this method integrates a residual neural network (ResNet101), a cross-attention mechanism, and a DeepLabV3 deep learning model. Using RGB images as the core learning object, it improves the accuracy, anti-interference capability, and refined recognition level of remote sensing image segmentation, thus meeting the needs for automated and precise identification of ground features. It is particularly suitable for automated and precise identification scenarios of various ground features.

[0008] The method proposed in this invention is specifically as follows:

[0009] RGB remote sensing images of the target region are acquired as the original images. A deep residual network, ResNet101, is used to extract features from the original images, resulting in an initial feature map. A cross-attention mechanism is introduced, using the initial feature map as input, to mine the region associations in the RGB remote sensing images, enhance key region features for target recognition, and perform feature enhancement on the initial feature map, outputting a focused feature map. A deep learning model, DeepLabV3, is used to extract features from the focused feature map at multiple scales. The Spatial Pyramid Pooling (ASPP) module captures contextual features under different receptive fields and fuses them with low-level detail features from ResNet101, resulting in a final feature map with rich context and clear boundaries. The final feature map is then subjected to pixel-level classification using a classification layer (e.g., 1×1 convolution) and upsampled to the original image size, outputting the semantic segmentation result.

[0010] The RGB remote sensing image comprises RGB images in three bands: red, green, and blue, and is acquired by a visible light imaging device.

[0011] Preferably, preprocessing and data augmentation operations such as normalization, scaling, and random cropping are performed on the acquired RGB remote sensing images.

[0012] Furthermore, the deep residual network ResNet101 performs the following operations on the input raw image:

[0013] The original image is sequentially processed through multiple residual blocks. Each residual block consists of two or three convolutional layers, skip connections, and element-wise addition operations. Specifically, the original image is first spatially downsampled through a first convolutional layer and a pooling layer to obtain intermediate features; these intermediate features are then sequentially processed through multiple residual blocks. After all residual blocks have been processed, the initial feature map is output.

[0014] Furthermore, within each residual block, the main path performs a series of operations: convolution, batch normalization, and ReLU activation. The skip connections directly pass the input features of the residual block to the end of the block. Finally, the main path output and the skip connection input are fused by element-wise addition, and then the output is sent to the next residual block after ReLU activation.

[0015] Furthermore, the cross-attention module performs the following operations on the initial feature map:

[0016] First, the original image is reduced in dimensionality using a single convolutional layer to generate the query matrix Q. The initial feature map is then reduced in dimensionality using two separate convolutional layers to generate the key matrix K and the value matrix V. Subsequently, Q, K, and V are all flattened into N×C dimensional feature matrices. Let Q be the number of spatial locations and C be the channel dimension. Calculate the dot product of Q and the transpose of K to obtain the similarity matrix, then divide by the scaling factor. , The attention weight matrix is ​​obtained by normalizing K using the Softmax function, which represents the channel dimension of K. Finally, this weight matrix is ​​multiplied by V to complete the weighted aggregation and output the focused feature map. In this process, Q originates from the original image, while K and V originate from the initial feature map, achieving cross-layer interaction from the original image to the initial feature map and enhancing the features of the target's key regions.

[0017] Furthermore, the DeepLabV3 model performs the following operations on the focused feature map:

[0018] First, the feature map input is processed by the Spatial Pyramid Pooling (ASPP) module. ASPP consists of five parallel branches: one convolutional branch, three dilated convolutional branches with different dilation rates, and a global average pooling followed by a convolutional branch. The outputs of each branch are concatenated along the channel dimension, and then subjected to a convolutional layer for feature integration, outputting a multi-scale fused feature. This multi-scale fused feature is first upsampled using bilinear interpolation to achieve the same spatial size as the low-level features of ResNet101, and then pixel-wise added and fused with the corresponding low-level features in ResNet101. Further upsampling and upsampling to the original image size is then performed, and finally, a convolutional classification layer outputs pixel-level semantic segmentation results.

[0019] The beneficial effects of this invention are as follows:

[0020] 1. After acquiring RGB remote sensing images, this invention performs preprocessing and data augmentation operations such as normalization, scaling, and random cropping. Normalization eliminates differences between illumination and sensor data, accelerating network convergence; online random cropping and flipping effectively expand sample diversity, enhance the model's generalization ability to changes in shooting angle, resolution, and ground scale, and provide high-quality input for the residual network;

[0021] 2. Residual neural networks avoid gradient problems through residual connections, ensuring the stable operation of deep networks. Processing with residual blocks can reduce information loss, and the output optimized feature maps provide a high-quality foundation for subsequent processing, improving the starting point of the overall process.

[0022] 3. The cross-attention mechanism can deeply explore the spatial distribution characteristics and contextual relationships of urban and rural features in RGB remote sensing images, strengthen the features of target features such as buildings and farmland, weaken the interference of complex backgrounds, significantly improve the ability to distinguish visually similar features, and enhance the segmentation accuracy in complex urban and rural scenes.

[0023] 4. DeepLabV3 relies on the ASPP module to integrate multi-scale features, and the output context-rich feature map can comprehensively reflect the relationship between urban and rural land features and the surrounding environment, thereby improving the application value of segmentation results in land spatial planning and ecological monitoring.

[0024] In practical applications, the remote sensing image segmentation method based on the integrated cross-attention mechanism proposed in this invention can effectively improve the segmentation accuracy of complex urban and rural features. The resulting classification results show that the regional distribution of features is uniform and smooth, with clear and accurate boundary contours, providing high-precision data support for urban and rural land cover monitoring, surface resource analysis, and ecological protection. Attached Figure Description

[0025] To more clearly illustrate the technical solutions used in this invention, the accompanying drawings used in the technical description are briefly introduced below.

[0026] Figure 1 This is a schematic diagram of the network flow of an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of the network structure of the present invention;

[0028] Figure 3 This is a schematic diagram of the cross-attention module used in this invention;

[0029] Figure 4 This is an image and image segmentation result diagram provided in the example of the present invention. Detailed Implementation

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0031] like Figure 1 As shown, a remote sensing image segmentation method based on an integrated cross-attention mechanism is implemented using the following steps:

[0032] Step 1: Acquire and preprocess the original images. After acquiring the RGB remote sensing images, perform routine preprocessing and enhancement operations such as normalization, scaling, and online random cropping and flipping. Normalization accelerates network convergence, and random cropping and flipping expands sample diversity, enhances the model's generalization ability to changes in shooting angle, resolution, and ground scale, and provides high-quality input for the residual network.

[0033] Step 2: Construct a deep learning image segmentation model, such as... Figure 2 As shown, the deep learning image segmentation model includes a deep residual network ResNet101, a cross-attention module, and a deep learning model DeepLabV3, as detailed below:

[0034] The deep residual network ResNet101 consists of 101 convolutional layers and residual connections. The residual connections directly transfer shallow features to deep layers through skip paths, allowing gradients to be effectively backpropagated during backpropagation. This fundamentally avoids the gradient vanishing and exploding problems that are common in deep networks, enabling the network to stably expand to deeper layers and improve feature learning capabilities.

[0035] After receiving the original image, the deep residual network ResNet101 performs initial 7×7 convolutions (stride 2) and 3×3 max pooling (stride 2) sequentially, reducing the output resolution to 1 / 4 and the number of channels to 64. Subsequently, it passes through multiple bottleneck residual blocks in four stages. The main path of each bottleneck residual block performs 1×1 dimensionality reduction → 3×3 spatial convolution → 1×1 dimensionality increase (with batch normalization and ReLU at each step). Skip connections are directly passed when the dimension is matched; otherwise, the dimension is adjusted by 1×1 convolution (stride 2). The two are added element-wise and then activated by ReLU. Finally, the initial feature map is output with a resolution of 1 / 32 of the original image and the number of channels to 2048.

[0036] like Figure 3 As shown, the cross-attention module takes the original image and initial feature map as input and mines the spatial correlation between the RGB image and depth features. Specifically:

[0037] The cross-attention module effectively mines the correlation between cross-source features using an attention mechanism, strengthening target features and weakening background interference. This module employs a cross-attention approach: the query matrix Q is generated from the original image through 1×1 convolution dimensionality reduction; the key matrix K and value matrix V are generated from the initial feature map through two independent 1×1 convolution dimensionality reductions. By calculating the similarity weights between Q and K, the V feature is weighted and aggregated, enabling the model to focus on the key regions indicated in the original image, suppressing complex background interference, and thus generating a more discriminative focused feature map.

[0038] Attention is calculated using a standard formula: ;

[0039] in, for The scaling operation can prevent the softmax gradient from vanishing due to excessively large scores.

[0040] Next is the generation of attention weights. The scores are normalized using Softmax to obtain attention weights representing the degree of attention.

[0041] ;

[0042] The attention score is calculated and scaled using dot product. Input the Softmax function to obtain the normalized attention weights. .

[0043] The Softmax function transforms the scores into a probability distribution with values ​​between [0,1] and a sum of 1. The larger the value, the more important the feature position or band needs to be, thus enabling the focus on key information.

[0044] Finally, the weighted output is calculated, using weights to... Perform a weighted summation to obtain the final output:

[0045] ;

[0046] in It is the attention weight matrix after Softmax normalization (dimension 1). ,in for Quantity, for (quantity) yes Feature matrix (dimension 1) , for (Feature dimensions). The two are weighted and aggregated through matrix multiplication: each element in the attention weight matrix quantifies the "specific" feature dimensions. With specific "The strength of the association between them", after multiplication, the high-weighted ones will be... Feature enhancement, corresponding to low weights Feature weakening, the final output dimension is Attention feature matrix This matrix is ​​the feature representation after fusing key information.

[0047] In the above process, Q, K, and V are respectively flattened into dimensional feature matrix ( (where C is the number of spatial locations and C is the channel dimension). Through this process, the model can autonomously learn the spatial mapping relationship from the original image to deep features, generating a focused feature map that focuses on key targets.

[0048] The core of the DeepLabV3 deep learning model lies in its Spatial Pyramid Pooling (ASPP) module. The ASPP module achieves multi-scale feature processing through dilated convolutions. Its specific structure consists of parallel branches: 1×1 convolutions are used for basic feature preservation and dimensionality compression; 3×3 dilated convolutions with different sampling rates (such as 6, 12, and 18) can expand the receptive field without increasing parameters or computational cost; global average pooling layers work in conjunction with 1×1 convolutions to extract global contextual features. The outputs of each branch are concatenated and then integrated into a multi-scale fused feature through a 1×1 convolution. This fused feature contains rich multi-scale contextual information.

[0049] Subsequently, the multi-scale fusion feature is upsampled and amplified to the same spatial size as the low-level features of ResNet101 through bilinear interpolation, and then fused pixel by pixel with the low-level features extracted by the ResNet101 residual neural network (such as the output of the second residual block) to restore the spatial details of the target boundary; it is then upsampled and amplified to the original image size, and finally output as a pixel-level semantic segmentation result through a convolutional classification layer.

[0050] The DeepLabV3 deep learning model takes a focused feature map as input and extracts multi-scale contextual features through the ASPP module mentioned above. These features are then fully integrated with low-level texture features, transforming the input focused feature map into a final feature map with rich context and clear boundaries. A classification layer then outputs pixel-level classification results, and a loss function (such as cross-entropy loss) is used to calculate the difference between the predicted results and the true labels. Backpropagation optimizes the network parameters until the model stably learns pixel-level distinguishing features between the target and the background.

[0051] Step 3: Divide the samples into training set and validation set to ensure the scientific nature of model training and evaluation. Input the samples into the training set in batches for iterative training, monitor the loss and mean intersection-over-union (mIoU) curve in real time, and automatically save the optimal weights until the model converges.

[0052] In terms of training parameter configuration, several adjustable parameters are provided. The number of classes is set to 2 by default, which users can adjust according to their actual needs; the batch size is set to 4 by default, using a mini-batch stochastic gradient descent strategy, which can reasonably control memory usage while ensuring training efficiency; the learning rate is set to 0.001, which can adaptively adjust the learning rate when used with the Adam optimizer, helping the model to converge stably; the number of training epochs is 10 by default, and combined with the early stopping mechanism, when the mIoU metric on the validation set does not improve for 3 consecutive epochs, the system will automatically save the best weights and terminate training, which can effectively prevent overfitting.

[0053] During the training monitoring and optimization phase, the system monitors the training loss and mean intersection-over-union (MUI) curves in real time, providing a clear view of the model's convergence process. Simultaneously, based on the mIoU metric of the validation set, the system automatically saves the weights of the best-performing model, ensuring that the model with the strongest generalization ability is ultimately used. Furthermore, a cosine annealing learning rate scheduling strategy is employed, gradually reducing the learning rate in the later stages of training, which helps the model capture complex features more accurately.

[0054] Step 4: Use the trained model to predict new images, output the classification results, and visualize them.

[0055] After completing model training and saving the optimal weights, the system performs the following operations based on these weights: loads the optimal DeepLabV3 model, preprocesses the input remote sensing imagery (maintaining the same processing flow as the training set), performs ResNet101 feature extraction, associates spatial features using a cross-attention mechanism, and performs DeepLabV3 multi-scale context fusion, outputting pixel-level classification results. The classification results are used to generate a visual segmentation mask through color mapping and are overlaid on the original image for display.

[0056] The system simultaneously evaluates model performance: it calculates the mIoU metric using an independent validation set and visualizes the prediction results against the ground truth labels. The final output includes a segmentation mask overlay, a quantitative evaluation report, and exportable pixel-level classification results, providing data support for subsequent GIS analysis.

[0057] In this embodiment, a trained model is used to achieve accurate segmentation of new images, and the performance is comprehensively evaluated using a validation set. To illustrate the effectiveness of the remote sensing image segmentation method, the publicly available dataset LoveDA is used for classification experiments. LoveDA is a high-resolution land cover semantic segmentation dataset adapted to urban and rural landscapes, covering three cities and including both urban and rural scenes. It has 7 classes of effectively labeled objects, a spatial resolution of 0.3 m, and a single image size of 1024×1024 pixels, effectively testing the model's segmentation accuracy and generalization ability in complex urban and rural landscapes.

[0058] Table 1. Details of the experimental dataset

[0059]

[0060] The dataset was classified using this invention, and the classification results are shown in Figure 4. The accuracy is shown in Table 2. The mIoU index of the method of this invention reached 0.6068, which is significantly better than the best performing existing comparative methods, DeeplabV3+ResNet101 (0.5811), as well as U-Net (0.5616), SFA-Net (0.5490), ViT-G12X4 (0.5440), MAE+MTP (0.5417) and IMP+MTP (0.5417). This verifies the effectiveness and superiority of this invention in remote sensing image segmentation tasks.

[0061] Table 2. Comparison of semantic segmentation accuracy of different models on the LoveDA dataset

[0062]

[0063] In summary, this invention combines a cross-attention mechanism with a deep learning image segmentation model based on DeepLabV3. Experimental results demonstrate that this model significantly outperforms various mainstream models in terms of mIoU, effectively improving the semantic segmentation performance of remote sensing images. The output classification map shows uniform and smooth land cover regions with clear boundaries, providing reliable technical support for urban and rural land cover monitoring, surface resource analysis, and ecological protection.

Claims

1. A remote sensing image segmentation method based on an integrated cross-attention mechanism, characterized in that, Specifically: RGB remote sensing images of the target area are acquired as the original images; a deep residual network ResNet101 is used to extract features from the original images to obtain an initial feature map after feature extraction; [The text then abruptly ends here, likely due to an incomplete translation or missing information.] The cross-attention mechanism takes the original image and initial feature map as input, mines the region association of RGB remote sensing image, strengthens the key region features for target recognition, enhances the features of the initial feature map, and outputs a focused feature map. The DeepLabV3 deep learning model is used to extract features at multiple scales from the focused feature map. The spatial pyramid pooling module ASPP is used to capture contextual features under different receptive fields and fuse them with low-level detail features from ResNet101 to obtain a final feature map with rich context and clear boundaries. The final feature map is then classified at the pixel level through a classification layer and upsampled to the original image size to output the semantic segmentation result. The cross-attention mechanism described above performs the following operations on the initial feature map: First, the original image is reduced in dimensionality by a separate convolutional layer to generate the query matrix Q. The initial feature map is then reduced in dimensionality by two separate convolutional layers to generate the key matrix K and the value matrix V. Subsequently, Q, K, and V are all flattened into N×C dimension feature matrices. Let Q be the number of spatial locations and C be the channel dimension; calculate the dot product of Q and the transpose of K to obtain the similarity matrix, then divide by the scaling factor. , The attention weight matrix is ​​obtained by normalizing K using the Softmax function, which is the channel dimension of K. Finally, the weight matrix is ​​multiplied by V to complete the weighted aggregation and output the focused feature map. In this process, Q comes from the original image, and K and V both come from the initial feature map, realizing cross-layer interaction and feature enhancement from the original image to the initial feature map.

2. The remote sensing image segmentation method based on an integrated cross-attention mechanism as described in claim 1, characterized in that, The ResNet101 deep residual network performs the following operations on the input raw image: The original image is sequentially passed through multiple residual blocks. Each residual block consists of two or three convolutional layers, skip connections, and element-wise addition operations. The specific process is as follows: the original image is first spatially downsampled through the first convolutional layer and pooling layer to obtain intermediate features. These intermediate features are then sequentially passed through multiple residual blocks. After all residual blocks have been processed, the initial feature map is output.

3. The remote sensing image segmentation method based on an integrated cross-attention mechanism as described in claim 2, characterized in that, Within each residual block, the main path performs a series of operations: convolution, batch normalization, and ReLU activation. The skip connections directly pass the input features of the residual block to the end of the block. Finally, the main path output and the skip connection input are fused by element-wise addition, and then the output is activated by ReLU before being sent to the next residual block.

4. The remote sensing image segmentation method based on integrated cross-attention mechanism as described in claim 1, characterized in that, The DeepLabV3 deep learning model performs the following operations on the focused feature map: First, the feature map is input to the Spatial Pyramid Pooling (ASPP) module. The ASPP consists of five parallel branches: one convolutional branch, three dilated convolutional branches with different dilation rates, and a global average pooling followed by a convolutional branch. The outputs of each branch are concatenated along the channel dimension and then integrated through a convolution to output multi-scale fused features. Subsequently, this multi-scale fused feature is upsampled and enlarged to the same spatial size as the low-level features of ResNet101 through bilinear interpolation, and then fused pixel-by-pixel with the corresponding low-level features in ResNet101. It is then upsampled and enlarged to the original image size, and finally output as pixel-level semantic segmentation results through a convolutional classification layer.

5. The remote sensing image segmentation method based on an integrated cross-attention mechanism as described in claim 1, characterized in that, The RGB remote sensing images were acquired by visible light imaging equipment.

6. The remote sensing image segmentation method based on an integrated cross-attention mechanism as described in claim 1, characterized in that, Preprocessing and data augmentation operations are performed on the acquired RGB remote sensing images.

7. The remote sensing image segmentation method based on an integrated cross-attention mechanism as described in claim 1, characterized in that, The difference between the predicted result and the true label is calculated by using a loss function, and the network parameters are optimized by backpropagation until the model stably learns pixel-level distinguishing features between the target and the background.

Images