A hybrid structure remote sensing image segmentation method based on state space model

Abstract

The application relates to a hybrid structure remote sensing image segmentation method based on a state space model, which comprises the following steps: obtaining a remote sensing image to be processed, inputting the remote sensing image to be processed into a hybrid structure remote sensing image segmentation model, and obtaining a segmentation result; the hybrid structure remote sensing image segmentation model is obtained by training a training set, and the training set comprises remote sensing images; the hybrid structure remote sensing image segmentation model is composed of a residual network submodel based on a convolutional neural network and an improved Mamba submodel to form a double-encoder structure, is used for extracting multi-scale features, is composed of a Transformer submodel based on a hybrid attention convolution module to form a decoder, is used for capturing global-range feature dependency through the multi-scale features, extracting local features and capturing spatial information, and generating the segmentation result; wherein the improved Mamba submodel is an original Mamba submodel into which a two-dimensional selective scanning module and a CSAM attention fusion module are introduced.

Description

Technical Field

[0001] This invention relates to the field of remote sensing technology, and in particular to a hybrid structure remote sensing image segmentation method based on a state-space model. Background Technology

[0002] In recent years, with the development of remote sensing technology and sensors, the acquisition of a large number of high-resolution remote sensing images has brought enormous challenges to the intelligent interpretation of these images. These images not only contain rich spatial details but also possess significant semantic potential. Remote sensing image semantic segmentation technology, by classifying land cover types in remote sensing data pixel-by-pixel, has become an important tool in fields such as land use classification, urban planning, environmental monitoring, and disaster assessment. However, the characteristics of remote sensing images—multi-scale targets, complex scenes, and high resolution—lead to numerous challenges in semantic segmentation tasks.

[0003] Traditional convolutional neural networks (CNNs) have made significant progress in semantic segmentation of remote sensing images. For example, models such as U-Net and DeepLab have achieved good segmentation performance through hierarchical feature extraction and skip connection mechanisms. However, due to the limited receptive field of CNNs, they struggle to capture long-range dependencies and global contextual information, resulting in bottlenecks when processing complex backgrounds or high-resolution images.

[0004] In recent years, Transformers based on self-attention mechanisms have demonstrated outstanding semantic modeling capabilities. Models such as SwinTransformer and UNetFormer have achieved excellent performance in semantic segmentation tasks. However, the high computational complexity and memory requirements of Transformers limit their application in high-resolution remote sensing imagery, especially in resource-constrained equipment. Furthermore, Transformer training relies on large-scale data, resulting in limited performance on small datasets.

[0005] State-space models (SSMs) offer a novel approach to semantic segmentation tasks due to their linear computational complexity and ability to model long-range dependencies. Models such as Mamba and DenseMamba have shown potential in visual tasks. However, the adaptability of SSMs to complex remote sensing scenes and their ability to represent multi-scale features still require further research and optimization. Summary of the Invention

[0006] To address the problems existing in the prior art, the present invention aims to provide a hybrid structure remote sensing image segmentation method based on a state-space model. This method employs a dual encoder structure based on CNN and a state-space model, and introduces a window-based attention mechanism decoder, which significantly improves the semantic segmentation performance of remote sensing images.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] A hybrid structure remote sensing image segmentation method based on a state-space model includes:

[0009] A remote sensing image to be processed is acquired, and the image is input into a hybrid structure remote sensing image segmentation model to obtain a segmentation result. The hybrid structure remote sensing image segmentation model is trained using a training set, which includes remote sensing images.

[0010] The hybrid structure remote sensing image segmentation model utilizes a residual network sub-model based on a convolutional neural network and an improved Mamba sub-model to construct a dual encoder structure for extracting multi-scale features. A Transformer sub-model based on a hybrid attention convolution module is used to construct a decoder, which captures global feature dependencies through the multi-scale features while extracting local features and spatial information to generate the segmentation result. The improved Mamba sub-model is the original Mamba sub-model that incorporates a two-dimensional selective scanning module and a CSAM attention fusion module.

[0011] Optionally, the residual network sub-model includes:

[0012] Several residual blocks are used to perform convolution operations to generate multi-scale features;

[0013] Several RCM modules are used to fuse the multi-scale features with the different-scale features generated by the improved Mamba sub-model, and input the fusion result into the decoder.

[0014] Optionally, generating features at different scales using the improved Mamba sub-model includes:

[0015] After performing layer normalization on the input features, the subsequent processing is divided into two paths:

[0016] The first path: sequentially passes through linear layers, depthwise convolution, a two-dimensional selective scanning module, and layer normalization to extract features;

[0017] The second path: By using the CSAM attention fusion module and fusing the output of the first path based on element-wise multiplication, features of different scales are generated.

[0018] Optionally, the expression for the improved Mamba sub-model is:

[0019] Q x =f LN (x)

[0020] W x =f CSAM (Q x )

[0021] S x =f LN (f SS2D (D conv (f Linear (Q x )))))⊙W x

[0022]

[0023] Where x represents the input feature, f LN (·) represents the layer normalization operation, f CSAM (·) represents the fusion attention operation, f Linear (·) represents a linear transformation operation, D conv (·) represents depthwise convolution, ⊙ represents element-wise multiplication, and f SS2D (·) represents a 2D selective scan operation, F x VSS Q represents the result obtained by processing the input feature x through the VSS block. x For the normalized result, W x For the results of attention operations, S x Select the scan operation and residual connection results for two dimensions.

[0024] Optionally, the two-dimensional selective scanning module includes:

[0025] The two-dimensional selective scanning submodule is used to perform forward and backward scanning on the input feature map through cross-scanning. During the scanning process, the image is expanded along multiple directions to generate different sequences, and the different sequences are processed using the S6 operation.

[0026] Optionally, processing the different sequences using the S6 operation includes:

[0027] The input feature map is transformed into three linear transformations to obtain the first, second, and third transformation results. The first transformation result is subjected to element-wise exponential operation to introduce nonlinear dynamic characteristics. The second transformation result is calculated using matrix transformation, which is a composite operation of matrix exponentiation and inversion. The intermediate states are further updated by combining the initial state and the input feature tensor to provide contextual information for the sequence. Each time step is calculated using a linear combination method. The third transformation result is processed by the state space vector to obtain the output result. All time steps are integrated to generate a complete output sequence.

[0028] Optionally, the expression for processing the different sequences using the S6 operation is:

[0029] X O =Sse (x,O)

[0030]

[0031] Where O represents different directions, X represents the input feature, and S se (·) and S sm (·) represents the scan unfolding and scan merging operations respectively, S6(·) represents the S6 operation, X O express, These represent the results of four different scanning directions for S6.

[0032] Optionally, the CSAM attention fusion module includes:

[0033] The channel attention module is used to extract channel features through global average pooling and max pooling operations, and further extract local features using the first depthwise separable convolution, and enhance important channels through the first sigmoid activation function;

[0034] The spatial attention module is used to combine the channel features, extract features through the first convolutional block, and obtain key spatial locations in the feature map through the second sigmoid activation function.

[0035] A multi-scale convolution module is used to extract features at different scales by employing second depthwise separable convolution kernels of different sizes, combining batch normalization and ReLU activation functions, and integrating the features at different scales with the second convolution block using channel rearrangement.

[0036] The channel attention module, the spatial attention module, and the multi-scale convolution module are connected in sequence.

[0037] Optionally, generating the segmentation result using the decoder includes:

[0038] After the multi-scale features are batch-normalized and standardized, they are divided into two parallel processing paths for subsequent processing:

[0039] The first processing path: capture global feature dependencies through a windowed multi-head self-attention module, while using a window mechanism to reduce computational complexity;

[0040] The second processing path: Local feature extraction and spatial information capture are performed through the convolutional group module;

[0041] The outputs of the two processing paths are merged and then batch normalized again. The merged features are then input into a multilayer perceptron layer for nonlinear transformation. The multiscale features are then superimposed with the nonlinearly transformed features through residual connections to generate the segmentation result.

[0042] The beneficial effects of this invention are as follows:

[0043] This invention introduces a CVSS module based on attention mechanism and visual semantic fusion (VSS) as an auxiliary encoder, which is combined with a CNN network to form a dual encoder structure. This provides additional global perception information while minimizing linear computational complexity. Furthermore, this paper designs a novel Region Convolutional Module (RCM) to fuse different features from the main encoder and auxiliary encoder, thereby improving feature representation capabilities.

[0044] To avoid increasing model complexity, this paper adopts a window-based self-attention mechanism in the decoder. Through a series of benchmark tests and ablation experiments on the ISPRS Vaihingen and Potsdam datasets, the UAVid dataset, and the LoveDA dataset, the effectiveness and efficiency of CMT-UNet in remote sensing image segmentation tasks are verified. Experimental results show that CMT-UNet significantly outperforms other existing semantic segmentation models based on CNN, Transformer, and Mamba with reasonable computational overhead. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is an overall architecture diagram of a hybrid structure remote sensing image segmentation method based on a state-space model according to this embodiment;

[0047] Figure 2 This is a schematic diagram of the CVSS module in the auxiliary encoder of this embodiment;

[0048] Figure 3 This is a schematic diagram illustrating the working principle of the two-dimensional selective scanning module (SS2D) in this embodiment;

[0049] Figure 4 This is an internal structure diagram of the Channel and Spatial Attention Fusion Module (CSAM) in this embodiment;

[0050] Figure 5 This is a schematic diagram of the feature fusion module (RCM) in this embodiment;

[0051] Figure 6 This is a schematic diagram of the Hybrid Attention Convolutional Module (HAC) decoder in this embodiment;

[0052] Figure 7 This is a visualization comparison of the different methods in this embodiment on the Vaihingen dataset;

[0053] Figure 8 This is a visualization comparison of the different methods in this embodiment on the Potsdam dataset. Detailed Implementation

[0054] The technical solutions in this embodiment will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] This embodiment discloses a hybrid structure remote sensing image segmentation method based on a state-space model, comprising: acquiring a remote sensing image to be processed; inputting the remote sensing image to be processed into a hybrid structure remote sensing image segmentation model to obtain a segmentation result; the hybrid structure remote sensing image segmentation model is trained using a training set, which includes remote sensing images; the hybrid structure remote sensing image segmentation model utilizes a residual network sub-model based on a convolutional neural network and an improved Mamba sub-model to construct a dual encoder structure for extracting multi-scale features, and a Transformer sub-model based on a hybrid attention convolution module to construct a decoder for capturing global feature dependencies through multi-scale features while extracting local features and spatial information, thereby generating a segmentation result; wherein, the improved Mamba sub-model is the original Mamba sub-model that incorporates a two-dimensional selective scanning module and a CSAM attention fusion module.

[0057] Specifically:

[0058] This embodiment discloses a hybrid structure remote sensing image segmentation method based on a state-space model, including the following steps:

[0059] S1: Acquire remote sensing images and divide them into training set, validation set and test set;

[0060] S2: Construct a hybrid structure remote sensing image segmentation model based on a state-space model;

[0061] S3: Train the hybrid structure remote sensing image segmentation model using the training set and validation set;

[0062] S4: Input the remote sensing image to be processed into the trained hybrid structure remote sensing image segmentation model to obtain the segmentation result.

[0063] In S2, a hybrid architecture is designed with a dual encoder consisting of a CNN-based residual network and an improved Mamba model. The decoder is constructed from a Transformer model based on a hybrid attention convolutional module. This network comprises three branches, each containing four corresponding blocks for fusing features extracted by the Mamba and CNN networks. After feature extraction and fusion at four scales, multi-scale features are obtained. These features are then fed into the decoder through region convolutional modules to generate the final prediction map.

[0064] Dual encoder design: The hybrid structure employs a CNN-based main encoder and a state-space model-based auxiliary encoder, which are responsible for local and global feature extraction, respectively.

[0065] The main encoder uses ResNet50 as its base network and consists of four residual blocks and four region convolutional modules (RCM). The residual blocks are responsible for extracting multi-scale local features, while the RCMs compensate for the main encoder's lack of global information capture by integrating global features from the auxiliary branches.

[0066] Auxiliary Encoder: A CVSS module is introduced, and a VSS block is used in the backbone network as a deep feature processing module. The VSS block enhances semantic information extraction capabilities by maintaining linear computational complexity while capturing long-range contextual information. Input features are processed by LayerNorm and then divided into two paths: the first path sequentially passes through linear layers, depthwise convolutions, 2D selective scan (SS2D) operations, and layer normalization to extract fine-grained features. The second path uses a channel and spatial attention module (CSAM) and fuses the results from the first path with the results through element-wise multiplication to output enhanced features.

[0067] RCM merges features of different scales derived from auxiliary branches into the main branch to compensate for the main branch's limitations in extracting global information. The outputs of the main encoder and auxiliary encoders are first processed by LayerNorm. A window-based multi-head self-attention mechanism is used to model long-range dependencies. Features from the auxiliary branches are extracted using the CVSS module with long dependencies, and convolutional networks are used to learn local details.

[0068] Furthermore, the residual network sub-model includes: several residual blocks for performing convolution operations to generate multi-scale features; and several RCM modules for fusing the multi-scale features with different scale features generated by the improved Mamba sub-model, and inputting the fusion result into the decoder.

[0069] Furthermore, generating features at different scales using the improved Mamba sub-model includes: after layer normalization of the input features, the subsequent processing is divided into two paths: the first path: sequentially passing through linear layers, depthwise convolution, a two-dimensional selective scanning module, and layer normalization to extract features;

[0070] The second path: By using the CSAM attention fusion module and fusing the output of the first path based on element-wise multiplication, features of different scales are generated.

[0071] Specifically:

[0072] The data is first fed into the system as input and undergoes initial batch normalization to standardize the feature distribution, helping the model converge faster. The data is then divided into two processing paths:

[0073] The first path uses a window-based multihead self-attention module to capture feature relationships and contextual information globally.

[0074] The second path uses a convolutional group module, which focuses on extracting local features and capturing spatial information.

[0075] Feature fusion: The results of the two paths are merged to complete the feature integration.

[0076] Batch normalization: The merged features are batch normalized again to further improve training stability.

[0077] Multilayer Perceptron (MLP) processing: The features are then transformed nonlinearly through a multilayer perceptron module to increase the model's expressive power.

[0078] Residual Connections: Finally, residual connections combine the initial input with the processed features to form the final output. This mechanism helps alleviate the vanishing gradient problem and improves feature propagation efficiency.

[0079] Furthermore, the expression for the improved Mamba sub-model is:

[0080] Q x =f LN (x)

[0081] W x =f CSAM (Q x )

[0082] S x =f LN (f SS2D (Dconv (f Linear (Q x )))))⊙W x

[0083]

[0084] Where x represents the input feature, f LN (·) represents the layer normalization operation, f CSAM (·) represents the fusion attention operation, f Linear (·) represents a linear transformation operation, D conv (·) represents depthwise convolution, ⊙ represents element-wise multiplication, and f SS2D (·) represents a 2D selective scan operation, F x VSS Q represents the result obtained by processing the input feature x through the VSS block. x For the normalized result, W x For the results of attention operations, S x Select the scan operation and residual connection results for two dimensions.

[0085] Furthermore, the two-dimensional selective scanning module includes:

[0086] The two-dimensional selective scanning submodule is used to perform forward and backward scanning of the input feature map through cross-scanning. During the scanning process, the image is expanded in multiple directions to generate different sequences, and the S6 operation is used to process the different sequences.

[0087] Furthermore, processing different sequences using the S6 operation includes:

[0088] The input feature map is transformed into three linear transformations to obtain the first, second, and third transformation results. The first transformation result is subjected to element-wise exponential operations to introduce nonlinear dynamic characteristics. The second transformation result is calculated using matrix transformations, which are composite operations of matrix exponentiation and inversion. The intermediate states are further updated by combining the initial state and the input feature tensor to provide contextual information for the sequence. Each time step is calculated using a linear combination method. The third transformation result is then processed using a state-space vector to obtain the output result. All time steps are integrated to generate a complete output sequence, activating all pixels and significantly emphasizing cross-type activation. The cross-scan module scanning mechanism ensures that the center pixel is most affected by pixels in the cross-direction; for each pixel, its long-range content takes precedence over local information.

[0089] Specifically:

[0090] Two-dimensional selective scanning (SS2D): Cross-scanning enables forward and backward scanning of an image. This process involves expanding the image in four directions, generating four distinct sequences, which are then processed by S6. The S6 operation allows each element in the one-dimensional array to interact with all previously scanned samples through compressed hidden states, thereby capturing different features.

[0091] The implementation process of S6 operations in SS2D mainly includes the following steps: First, the input feature tensor x, with shape [B,L,D], is transformed into Δ, B, and C through three linear transformations. Then, element-wise exponentiation is performed on Δ to introduce nonlinear dynamic characteristics. Next, B is calculated using matrix transformations, involving a composite operation of matrix exponentiation and inversion, which efficiently captures long-range dependencies in the sequence. Then, the intermediate state h is updated by combining the initial state h0 and the input x, providing contextual information for sequence generation. Finally, y is calculated at each time step using a linear combination method. t And integrate all time steps to generate a complete output sequence y.

[0092] Furthermore, the expression for processing different sequences using the S6 operation is as follows:

[0093] X O =S se (x,O)

[0094]

[0095] Where O represents different directions, X represents the input feature, and S se (·) and S sm (·) represents the scan unfolding and scan merging operations respectively, S6(·) represents the S6 operation, X O express, These represent the results of four different scanning directions for S6.

[0096] Furthermore, the CSAM attention fusion module includes:

[0097] The channel attention module is used to extract channel features through global average pooling and max pooling operations, and further extract local features using the first depthwise separable convolution, and enhance important channels through the first sigmoid activation function;

[0098] The spatial attention module is used to combine channel features, extract features through the first convolutional block, and obtain key spatial locations in the feature map through the second sigmoid activation function.

[0099] The multi-scale convolution module is used to extract features at different scales by employing second depthwise separable convolution kernels of different sizes, combining batch normalization and ReLU activation functions, and integrating features at different scales with the second convolution block using channel rearrangement.

[0100] The channel attention module, spatial attention module, and multi-scale convolution module are connected sequentially.

[0101] Specifically:

[0102] Multi-scale attention mechanism (CSAM): Combining channel attention and spatial attention modules, it improves the segmentation ability for small objects and complex backgrounds. The CSAM module combines channel attention, spatial attention, and multi-scale convolution modules to enhance the segmentation ability for complex backgrounds and small objects.

[0103] Channel attention module: Features are extracted through global average pooling, max pooling and depthwise separable convolution, and combined with sigmoid activation function to enhance the expression of important channels.

[0104] Spatial Attention Module: Combining the average pooling and max pooling results of the channels, it uses 7×7 convolution to highlight key positions in the feature map.

[0105] Multi-scale convolution module: By combining multi-size convolution kernels with batch normalization and ReLU activation function, the ability to express multi-scale features is further enriched.

[0106] Furthermore, the segmentation results generated using the decoder include:

[0107] After batch normalizing the feature distribution of the multi-scale features, they are divided into two parallel processing paths for subsequent processing:

[0108] The first processing path: capture global feature dependencies through a windowed multi-head self-attention module, while using a window mechanism to reduce computational complexity;

[0109] The second processing path: Local feature extraction and spatial information capture are performed through the convolutional group module;

[0110] The outputs of the two processing paths are merged and then batch normalized again. The merged features are then input into a multilayer perceptron layer for nonlinear transformation. Finally, the multi-scale features are superimposed with the nonlinearly transformed features through residual connections to generate the segmentation result.

[0111] Specifically:

[0112] Construct an HAC decoder. This decoder combines a window-based multi-head self-attention module with a group convolution module to build a lightweight Transformer decoder.

[0113] First, the feature distribution is standardized using batch normalization, then processed through two parallel paths. One path uses a window-based multihead self-attention module to capture global feature dependencies while reducing computational complexity using a windowing mechanism. The other path uses a convolutional group module to focus on local feature extraction and spatial information capture. The outputs of the two paths are then merged and batch normalized again to enhance the feature integration. Next, the fused features are fed into a multilayer perceptron (MLP) for nonlinear transformation, further improving the model's expressive power. Finally, residual connections are used to superimpose the initial input with the processed features, mitigating the vanishing gradient problem and improving feature propagation efficiency, ultimately generating the output data. This architecture combines the advantages of self-attention and convolutional operations, effectively capturing multi-scale features while maintaining training stability and model performance.

[0114] like Figure 1 As shown, this embodiment provides a hybrid structure remote sensing image segmentation method based on a state-space model, including the following steps:

[0115] S1: Acquire remote sensing images and divide them into training set, validation set and test set;

[0116] S2: Construct a hybrid structure remote sensing image segmentation model based on a state-space model;

[0117] S3: Train the hybrid structure remote sensing image segmentation model using the training set and validation set;

[0118] S4: Input the remote sensing image to be processed into the trained segmentation model to obtain the segmentation result.

[0119] In this method, the segmentation model is designed as a dual encoder consisting of a CNN-based residual network and an improved Mamba model, while the decoder is constructed from a Transformer model based on a Hybrid Attention Convolutional Module (HAC). The model contains three branches, each comprising four blocks, for fusing features extracted by the CNN and Mamba models. After feature extraction and fusion at four scales, multi-scale features are generated. These features are then fed into the decoder via Region Convolutional Modules (RCMs) to produce the final prediction map.

[0120] This invention uses ResNet50 as the main encoder to learn local features. For example... Figure 1 As shown, it contains four residual blocks and four region convolutional modules (RCMs). These four residual blocks perform convolution operations to generate multi-scale features, represented as follows: Compared to the auxiliary encoder, the main encoder effectively extracts features from remote sensing images using existing pre-trained models. Therefore, this invention uses RCM to merge features of different scales derived from the auxiliary branch into the main branch, thereby compensating for the limitations of the main branch in extracting global information. Figure 5 As shown, the outputs of the main encoder and the auxiliary encoder are... and First, each part undergoes layer normalization. This helps accelerate network convergence and improves the stability of the training process. The main encoder extracts features through a CNN network, thus employing a window-based multi-head self-attention mechanism to model long-range dependencies while maintaining linear complexity. Features of the auxiliary branches are extracted using a CVSS module with long dependencies, therefore convolutional networks are used to learn local details.

[0121] The auxiliary encoder constructed in this invention is as follows: Figure 2 As shown, the auxiliary encoder incorporates a CVSS module, and the backbone network of the module uses a VSS block as a deep feature processing module.

[0122] The VSS block captures remote contextual information while maintaining linear complexity and enhances the refined extraction of semantic information. During the training phase, the VSS block interacts with the semantic transformer to achieve joint training.

[0123] Feature processing flow: After the input features are normalized by Layer Normalization, they are divided into two paths:

[0124] The first path sequentially extracts fine-grained features through linear layers, depthwise convolution, two-dimensional selective scanning (SS2D) operations, and layer normalization.

[0125] The second path passes through the channel and spatial attention module (CSAM) and is fused with the first path through element-wise multiplication to obtain an enhanced feature output.

[0126] The formulaic expression for the auxiliary encoder is as follows:

[0127] Q x =f LN (x) (1)

[0128] W x =f CSAM (Q x (2)

[0129] S x =f LN (f SS2D (D conv (f Linear (Qx )))))⊙W x (3)

[0130]

[0131] Where x represents the input feature, f LN (·) represents the layer normalization operation, f CSAM (·) represents the fusion attention operation, f Linear (·) represents a linear transformation operation, D conv (·) represents depthwise convolution, ⊙ represents element-wise multiplication, and f SS2D (·) represents a 2D selective scan operation, F x VSS This represents the result obtained by processing the input feature x through the VSS block.

[0132] The two-dimensional selective scanning (SS2D) structure utilizes cross-scanning to achieve forward and backward scanning of the image, effectively expanding the receptive field. Specific implementation details are as follows... Figure 3 As shown, the SS2D module generates four sequences by expanding the image along four directions and uses the S6 operation to capture feature information in different directions. Finally, the output image is restored to its original size by scanning and merging. The formula is as follows:

[0133] X O =S se (x,O) (5)

[0134]

[0135] Where O represents different directions, O∈[1,2,3,4]. X represents the input feature. S se (·) and S sm (·) represents the scan unfold and scan merge operations, respectively. S6(·) represents the S6 operation.

[0136] The pseudocode for the S6 operation is shown in Table 1.

[0137] Table 1

[0138]

[0139] like Figure 4As shown, the CSAM attention fusion module includes a channel attention module, a spatial attention module, and a multi-scale convolution module. The channel attention module enhances the representation of important channels in the feature map through global average pooling and max pooling, depthwise separable convolution, and the sigmoid activation function. The spatial attention module combines the results of channel average pooling and max pooling, followed by 7x7 convolution and sigmoid activation, to highlight important spatial locations in the feature map. The multi-scale convolution module uses depthwise separable convolution kernels of different sizes, combined with batch normalization and the ReLU activation function, and further enhances the model's multi-scale feature representation capability through channel rearrangement and 1x1 convolution.

[0140] The HAC decoder constructed in this invention is as follows: Figure 6 As shown, this decoder combines a window-based multi-head self-attention module with a group convolution module to build a lightweight Transformer decoder.

[0141] First, the feature distribution is standardized using batch normalization, then processed through two parallel paths. One path uses a window-based multihead self-attention module to capture global feature dependencies while reducing computational complexity using a windowing mechanism. The other path uses a convolutional group module to focus on local feature extraction and spatial information capture. The outputs of the two paths are then merged and batch normalized again to enhance the feature integration. Next, the fused features are fed into a multilayer perceptron (MLP) for nonlinear transformation, further improving the model's expressive power. Finally, residual connections are used to superimpose the initial input with the processed features, mitigating the vanishing gradient problem and improving feature propagation efficiency, ultimately generating the output data. This architecture combines the advantages of self-attention and convolutional operations, effectively capturing multi-scale features while maintaining training stability and model performance.

[0142] The loss function in the training phase of this invention Using dice loss and cross-entropy loss The combination of these can be expressed using the following formula:

[0143]

[0144] Where N and K represent the number of samples and the number of categories, respectively. (n) and represents the one-hot encoding of the real semantic label and the corresponding softmax output of the network, respectively, n∈[1,…,N]. This represents the confidence level that sample n belongs to class k.

[0145] In some embodiments, this application provides an experimental comparison between the "remote sensing image segmentation method" of this application and existing remote sensing image segmentation methods.

[0146] Experiment 1: Performance Comparison on the UAVid Test Set

[0147] On the UAVid test set, this invention was compared with various high-efficiency segmentation networks, and its performance was evaluated based on the following metrics:

[0148] mIoU (mean intersection-over-union ratio): measures the segmentation accuracy of the prediction results;

[0149] GPU memory usage (MB): reflects the resource requirements during network operation;

[0150] Computational complexity (FLOPs): Evaluates the computational complexity of a network;

[0151] Number of parameters (M): Measures the storage requirements of the model;

[0152] Speed ​​(FPS): Measured inference efficiency in frames per second.

[0153] Experiment 2: Study on Network Stability of Remote Sensing Images of Different Sizes

[0154] To evaluate the stability of the present invention under different input sizes, remote sensing images with square (512×512, 1024×1024, 2048×2048) and rectangular (512×1024, 1024×2048) sizes were selected for training and testing, and the mIoU fluctuation of the segmentation results was analyzed.

[0155] Experiment 3: Quantitative and Qualitative Comparison on the UAVid Test Set

[0156] On the UAVid test set, the method of this invention is quantitatively compared with other advanced remote sensing image segmentation algorithms using metrics such as mean Intersection over Union (mIoU), mean F1 score, and overall accuracy (OA). Furthermore, visualizations of the segmentation results are provided to intuitively illustrate the differences in segmentation performance.

[0157] Experimental conditions:

[0158] All experimental models of this invention were run on a server equipped with four NVIDIA Tesla A100 40GB GPUs and two Intel Xeon 6330 CPUs. The experimental environment was Ubuntu 22.04, the programming language was Python 3.10, and the deep learning framework was PyTorch 2.0.1. To accelerate convergence, the models were trained using the AdamW optimizer. The base learning rate was set to 6 × 10⁻⁶. -4 And a cosine annealing strategy is used to adjust the learning rate.

[0159] During training on the UAVid dataset, the input image size was 1024×1024, and data augmentation techniques were used, including random vertical flipping, horizontal flipping, and random brightness adjustment. The training epochs were set to 60, and the batch size was 16. Vertical and horizontal flipping test augmentation strategies were used during the testing phase.

[0160] Evaluation indicators:

[0161] This application uses mean Intersection over Union (mIoU), mean F1 score, and overall accuracy (OA) as evaluation metrics. The mean Intersection over Union (mIoU) measures the accuracy of predictions by calculating the ratio of the intersection to the union of the predicted values ​​and the true values. The F1 score considers both recall and precision to evaluate the balance and completeness of the prediction results. Overall accuracy (OA) represents the proportion of correctly classified pixels out of the total number of pixels.

[0162]

[0163] Where N is the number of semantic categories. TP i Denotes the true instance, and represents FP. i False positives, FN i This indicates a false negative. Precision refers to the proportion of positive samples correctly predicted by the model out of all samples predicted as positive, while recall refers to the proportion of all actual positive samples captured by the model out of all actual positive samples.

[0164] Experimental Results and Analysis:

[0165] As shown in Table 2, this paper presents the quantitative comparison results with state-of-the-art lightweight networks on the UAVid test set. Compared with various advanced segmentation networks, this invention performs excellently in several key metrics:

[0166] Compared to the fastest model Fast-SCNN, the present invention improves mIoU by 24.6%;

[0167] Compared with the UNetformer model with similar parameter count, the inference speed of this invention reaches 106.9 FPS, and the mIoU is improved by 7.7%;

[0168] Compared to the Mamba architecture, CMT-UNet achieves a 1.8% improvement in mIoU;

[0169] The mIoU of this invention is 11.3% higher than that of the pure Transformer network Segmenter.

[0170] This further verifies the effectiveness of the hybrid structure, CSAM, and RCM modules in this invention.

[0171] Table 2

[0172]

[0173] To evaluate the stability of the network, the CMT-UNet model was trained with different input sizes, including square (512×512, 1024×1024, 2048×2048) and rectangular (512×1024, 1024×2048) sizes. Experimental results are summarized in Table 3, showing that the proposed model is stable under different input sizes, with an mIoU bias of less than 0.7%. The 1024×1024 input size achieved the highest mIoU on the UAVid validation set. Furthermore, square inputs achieved higher scores than rectangular inputs, but excessively large input sizes (such as 2048×2048) reduced the mIoU for small targets.

[0174] Table 3

[0175]

[0176] This invention trained multiple segmentation networks and conducted detailed result comparisons on the official Vaihingen test set. As shown in Table 4, this invention outperforms other methods in IoU across all categories, achieving a best performance of 69.1% in mIoU. Specifically, the invented remote sensing image segmentation method not only surpasses the efficient CNN-based network ABCNet by 6.8% in mIoU, but also outperforms the latest hybrid Mamba networks RS-Mamba and CM-UNet by 3.3% and 1.8%, respectively. Particularly in the challenging segmentation of small targets such as 'people', this invention leads other methods by at least 8.9% in IoU. Figure 7 The presentation compares the segmentation results of this invention with existing methods on the Vaihingen dataset. Visualization results demonstrate the superior segmentation performance of this invention for complex scenes. Figure 8 A visualization comparing the present invention with existing methods on the Potsdam dataset is shown.

[0177] Table 4

[0178]

[0179] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A hybrid structure remote sensing image segmentation method based on a state-space model, characterized in that, include: Acquire the remote sensing image to be processed, input the remote sensing image to be processed into the hybrid structure remote sensing image segmentation model, and obtain the segmentation result; The hybrid structure remote sensing image segmentation model is obtained by training a training set, which includes: remote sensing images; The hybrid remote sensing image segmentation model utilizes a residual network sub-model based on a convolutional neural network and an improved Mamba sub-model to construct a dual encoder structure for extracting multi-scale features. A Transformer sub-model based on a hybrid attention convolution module is used to construct a decoder, which captures global feature dependencies while extracting local features and spatial information through the multi-scale features, thereby generating the segmentation result. The improved Mamba sub-model is the original Mamba sub-model that incorporates a two-dimensional selective scanning module and a CSAM attention fusion module. The residual network sub-model includes: Several residual blocks are used to perform convolution operations to generate multi-scale features; Several RCM modules are used to fuse the multi-scale features with the different-scale features generated by the improved Mamba sub-model, and input the fusion result into the decoder; The CSAM attention fusion module includes: The channel attention module is used to extract channel features through global average pooling and max pooling operations, and further extract local features using the first depthwise separable convolution, and enhance important channels through the first sigmoid activation function; The spatial attention module is used to combine the channel features, extract features through the first convolutional block, and obtain key spatial locations in the feature map through the second sigmoid activation function. A multi-scale convolution module is used to extract features at different scales by employing second depthwise separable convolution kernels of different sizes, combining batch normalization and ReLU activation functions, and integrating the features at different scales with the second convolution block using channel rearrangement. The channel attention module, the spatial attention module, and the multi-scale convolution module are connected in sequence.

2. The hybrid structure remote sensing image segmentation method based on a state-space model according to claim 1, characterized in that, Generating features at different scales using the improved Mamba sub-model includes: After performing layer normalization on the input features, the subsequent processing is divided into two paths: The first path: sequentially passes through linear layers, depthwise convolution, a two-dimensional selective scanning module, and layer normalization to extract features; The second path: By using the CSAM attention fusion module and fusing the output of the first path based on element-wise multiplication, features of different scales are generated.

3. The hybrid structure remote sensing image segmentation method based on a state-space model according to claim 1, characterized in that, The expression for the improved Mamba sub-model is: ； ； ； ； in, Indicates input features, For layer normalization operation, To integrate attention-based operations, For linear transformation operations, For depthwise convolution operations, For element-wise multiplication, This is a 2D selective scan operation. This indicates that the input features are processed through the VSS block. The result obtained through processing For the normalized result, For the results of attention operations, Select the scan operation and residual connection results for two dimensions.

4. The hybrid structure remote sensing image segmentation method based on a state-space model according to claim 1, characterized in that, The two-dimensional selective scanning module includes: The two-dimensional selective scanning submodule is used to perform forward and backward scanning of the input feature map through cross scanning. During the scanning process, the image is expanded along multiple directions to generate different sequences, and the different sequences are processed using the S6 operation. Processing the different sequences using the S6 operation includes: The input feature map is transformed into three linear transformations to obtain the first, second, and third transformation results. The first transformation result is subjected to element-wise exponential operation to introduce nonlinear dynamic characteristics. The second transformation result is calculated using matrix transformation, which is a composite operation of matrix exponentiation and inversion. The intermediate states are further updated by combining the initial state and the input feature tensor to provide contextual information for the sequence. Each time step is calculated using a linear combination method. The third transformation result is processed by the state space vector to obtain the output result. All time steps are integrated to generate a complete output sequence.

5. The hybrid structure remote sensing image segmentation method based on a state-space model according to claim 4, characterized in that, The expression for processing the different sequences using the S6 operation is as follows: ； ； ； in, Representing different directions. Indicates input features, and These represent the scan unfolding and scan merging operations, respectively. Indicates S6 operation, They represent The results were processed in four different scanning directions.

6. The hybrid structure remote sensing image segmentation method based on a state-space model according to claim 1, characterized in that, Generating the segmentation result using the decoder includes: After the multi-scale features are batch-normalized and standardized, they are divided into two parallel processing paths for subsequent processing: The first processing path: capture global feature dependencies through a windowed multi-head self-attention module, while using a window mechanism to reduce computational complexity; The second processing path: Local feature extraction and spatial information capture are performed through the convolutional group module; The outputs of the two processing paths are merged and then batch normalized again. The merged features are then input into a multilayer perceptron layer for nonlinear transformation. The multiscale features are then superimposed with the nonlinearly transformed features through residual connections to generate the segmentation result.

Images