Remote sensing image segmentation method and device

Abstract

The invention discloses a remote sensing image segmentation method and device, and is applied to the technical field of remote sensing image segmentation, and the method comprises the steps: carrying out the segmentation processing of a remote sensing image through an SCFEMMamba model, and obtaining a contour map of a ground object region, the model is obtained by replacing two visual state space blocks stacked in the first three decoders of a UNetMamba model with a feature enhancement block and at least one sequentially connected improved decoding block; in the improved decoding block, the input of the improved decoding block is divided into a first input and a second input, the first input is subjected to convolution processing to obtain a convolution branch feature map, and the second input is subjected to visual state space processing to obtain a state branch feature map; obtaining the output of the improved decoding block according to the convolution branch feature map, the state branch feature map and the input of the improved decoding block; and the feature enhancement block performs feature enhancement processing on the output of the last improved decoding block. The method effectively improves the segmentation precision of the remote sensing image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image segmentation, and particularly relates to a remote sensing image segmentation method and device. BACKGROUND

[0002] With the rapid development of remote sensing technology, high-resolution remote sensing images have become the main source of obtaining rich ground object information on the earth's surface. Semantic segmentation of ground objects in these high-resolution remote sensing images has practical significance and plays a crucial role in multiple downstream tasks. However, the gradual enrichment of detailed information in high-resolution images leads to more complex appearance characteristics of ground objects, which brings major challenges to segmentation accuracy and segmentation efficiency. Generally, the unique challenges of remote sensing images come from the diversity of targets and the variability of scales under high-viewing-angle shooting conditions, including low recognition of objects with similar color and shape, widespread existence of small-size targets, complex and variable background, and complex edge details.

[0003] With the development of computer vision technology, a variety of computer vision basic models have emerged, such as models based on convolutional neural network (CNN) architecture and models based on Transformer architecture, which have enabled high-resolution remote sensing images to flourish in the field of semantic segmentation.

[0004] Among them, in the model based on CNN architecture, the U-Net model is a milestone in semantic segmentation, which establishes a basic architecture for subsequent work, i.e., an encoder-decoder architecture with skip connections. In addition, the DeepLab model introduces atrous convolution and spatial pyramid pooling, which improves segmentation accuracy and speed; the PSPNet model uses a pyramid pooling module to obtain semantic information at different scales, which helps to accurately divide object boundaries. Although CNNs show significant advantages in local feature extraction, their limited receptive field limits the ability to model global context, which can lead to spatial information loss when processing high-resolution remote sensing images, thereby affecting segmentation accuracy.

[0005] The model based on the Transformer architecture introduces a self-attention mechanism, which brings breakthroughs in the fields of natural language processing and computer vision, and exhibits significant advantages in modeling long-distance dependencies and capturing global semantic associations, effectively overcoming the limitations of the local receptive field of traditional CNNs. Some related technologies innovatively embed Transformer modules into U-shaped network frameworks, which greatly improves the accuracy of high-resolution remote sensing images in semantic segmentation. Inspired by this, subsequent research gradually shifted to the Transformer-based technical route. However, the intrinsic quadratic computational complexity and massive parameter quantity of Transformer pose a challenge to its efficiency in processing high-resolution remote sensing images, which is an important challenge to its practical application.

[0006] Recently, the Mamba architecture has become a new hotspot, with its linear time complexity and strong long-distance dependency modeling capability, attracting widespread attention in the field of remote sensing image semantic segmentation. The Mamba architecture successfully solves the quadratic computational complexity problem of traditional Transformer models, while ensuring accuracy through its competitive long-distance dependency modeling capability. With the continuous development of Mamba technology in the field of computer vision, models based on the Mamba architecture have emerged in the field of remote sensing image semantic segmentation, such as the UNetMamba model obtained by embedding the Mamba architecture into the U-shaped network framework. Although the UNetMamba model has made significant progress in performing remote sensing image segmentation tasks, it has deficiencies in feature extraction and fusion of diverse features in complex scenes, making it difficult to capture the details and key feature differences between different feature targets, resulting in blurred boundaries and class confusion for some features during segmentation, thus leading to low segmentation accuracy of the UNetMamba model for remote sensing images. SUMMARY

[0007] The embodiments of the present application provide a remote sensing image segmentation method and device for improving the segmentation accuracy of remote sensing images.

[0008] In one aspect, the embodiments of the present application provide a remote sensing image segmentation method, comprising the following steps:

[0009] obtaining a remote sensing image;

[0010] segmenting the remote sensing image using the SCFEMamba model to obtain a feature region contour map;

[0011] The SCFEMamba model is obtained by replacing two stacked visual state space blocks in the first three decoders of the UNetMamba model with feature enhancement blocks and at least one sequentially connected improved decoding block. The feature enhancement block is used to perform feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder. The improved decoding block includes:

[0012] A channel segmentation layer is used to segment the input of the improved decoding block to obtain a first input and a second input;

[0013] The convolutional branch is used to perform convolution processing on the first input to obtain the convolutional branch feature map;

[0014] The state branch is used to perform visual state space processing on the second input to obtain the state branch feature map;

[0015] The pre-output module is used to obtain the output of the improved decoding block based on the convolutional branch feature map, the state branch feature map, and the input of the improved decoding block.

[0016] On the other hand, embodiments of this application provide a remote sensing image segmentation apparatus, including:

[0017] The acquisition module is used to acquire remote sensing images;

[0018] The segmentation module is used to segment the remote sensing image using the SCFEMamba model to obtain the outline map of the ground feature area;

[0019] The SCFEMamba model is obtained by replacing two stacked visual state space blocks in the first three decoders of the UNetMamba model with feature enhancement blocks and at least one sequentially connected improved decoding block. The feature enhancement block is used to perform feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder. The improved decoding block includes:

[0020] A channel segmentation layer is used to segment the input of the improved decoding block to obtain a first input and a second input;

[0021] The convolutional branch is used to perform convolution processing on the first input to obtain the convolutional branch feature map;

[0022] The state branch is used to perform visual state space processing on the second input to obtain the state branch feature map;

[0023] The pre-output module is used to obtain the output of the improved decoding block based on the convolutional branch feature map, the state branch feature map, and the input of the improved decoding block.

[0024] According to an embodiment of this application, a remote sensing image segmentation method and apparatus are provided. First, a remote sensing image is acquired. Then, the remote sensing image is segmented using the SCFEMamba model to obtain a contour map of the ground feature region. The SCFEMamba model is obtained by replacing two stacked visual state space blocks in the first three decoders of the UNetMamba model with a feature enhancement block and at least one sequentially connected improved decoding block. In the improved decoding block, the input is divided into a first input and a second input. The first input is convolved to obtain a convolution branch feature map, and the second input is processed using visual state space to obtain a state branch feature map. The output of the improved decoding block is obtained based on the convolution branch feature map, the state branch feature map, and the input of the improved decoding block. The feature enhancement block performs feature enhancement processing on the output of the last improved decoding block. This application effectively improves the segmentation accuracy of remote sensing images.

[0025] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0026] Figure 1 This is a structural diagram of the existing UnetMamba model;

[0027] Figure 2 This is a structural diagram of the SCFEMamba model provided in this application;

[0028] Figure 3 This is a structural diagram of the SS-Conv-SSM block provided in this application;

[0029] Figure 4 This is a schematic diagram of the SS2D-CS mechanism provided in this application;

[0030] Figure 5 This is a structural diagram of the E-FFN block provided in this application;

[0031] Figure 6 This is a structural diagram of the FRH block provided in this application;

[0032] Figure 7 This is a visualization comparing the results of the SCFEMamba model provided in this application with other models. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] The present application will be further described below with reference to the accompanying drawings and specific embodiments. The described embodiments should not be considered as limitations on the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0035] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0037] To address the limitations of segmentation models based on architectures such as CNN, Transformer, and UnetMamba in remote sensing image segmentation tasks, this application proposes a remote sensing image segmentation method and apparatus. It improves the existing UNetMamba model to obtain the SCFEMamba model. This model improves the image segmentation accuracy and efficiency of remote sensing images by introducing the Mamba architecture and combining the local feature extraction capability of CNN and the global context modeling capability of Transformer.

[0038] First, the structure of existing Mamba models is introduced. The UNetMamba model is a UNet-like semantic segmentation model based on the Mamba architecture. Its overall architecture is similar to U-Net, and it is designed to efficiently process high-resolution remote sensing images. (Refer to...) Figure 1 , Figure 1 In this model, "Input" refers to the remote sensing image to be segmented, "Output" refers to the contour map of the land feature region segmented from the remote sensing image, "ResT Stage1-4" refers to the first to fourth encoders in the encoder structure, "H" refers to the height of the feature map, "W" refers to the width of the feature map, "Sum" refers to the summation operation (e.g., direct summation or weighted summation), and "Principal Loss" and "Auxiliary Loss" refer to different loss functions. The model consists of three components:

[0039] The ResT Backbone-based segmented encoder structure has four encoders (ResT Stage1-4) for performing staged encoding processing on the input image.

[0040] The Mamba Segmentation Decoder (MSD) is a semantic segmentation decoder based on Mamba that can effectively decode complex semantic information in high-resolution images. Specifically, thanks to the VMamba model, the Mamba architecture has demonstrated excellent capabilities in computer vision. In semantic segmentation, the VMamba model is widely used due to its long sequence modeling capabilities. MSD uses the VSS blocks from the VMamba model to construct a four-layer decoder structure for the UNe tMamba model. Specifically, each layer of the MSD structure consists of an upsampling module (Patch Expanding) for upsampling and two stacked VSS blocks (VSS Block × 2) for semantic information extraction. The upsampling module upsamples the spatial resolution by a factor of 2 while compressing the channel dimension by a factor of 2. The VSS blocks incorporate a 2D selective scan (SS2D) mechanism. By decoding multi-scale features through MSD, the semantic segmentation result is finally output through a 1×1 convolution (1×1 Conv).

[0041] The Local Supervision Module (LSM) consists of three LSM blocks (LSMBlocks), each corresponding to one of the first three decoder layers. These blocks significantly enhance the ability to perceive local information. Specifically, while the VSS block in the MSD (Model-Side Decoder) can extract a large receptive field from the feature map and is adept at capturing long-range dependencies, it neglects local information, which contains crucial factors for the accuracy of semantic segmentation. Therefore, introducing LSM effectively enhances the perception of local details. It's important to note that LSM only plays a role in model training and is not used in real-time applications.

[0042] Secondly, this application briefly introduces the improvements made to the existing UNetMamba model by its embodiments. (Refer to...) Figure 2 The embodiments of this application make the following improvements to the UNetMamba model:

[0043] (i) A new improved decoding block (SS-Conv-SSM block) is introduced in MSD to replace the original two stacked VSS blocks. This improved decoding block integrates a convolutional (Conv) architecture for extracting local features and a state space model (SSM) architecture for capturing global dependencies.

[0044] (ii) The output of the decoder integrates a feature enhancement block (E-FFN block) to enhance the feature representation capability.

[0045] (III) Improved decoding block retains SS2D mechanism and introduces attention branch on the basis of existing SS2D mechanism to form scanning attention mechanism (SS2D-CS mechanism) to enhance the segmentation model’s ability to capture key ground features and improve the robustness of segmentation model.

[0046] (iv) A fusion output module (FRH block) is introduced at the end of the UNetMamba model to replace the original 1×1 convolutional layer, which is used to combine with the spatial information in the shallow features to improve the segmentation accuracy of remote sensing images.

[0047] Based on the above, the following will describe in detail, with reference to the accompanying drawings, a remote sensing image segmentation method provided by the embodiments of this application.

[0048] This application provides a remote sensing image segmentation method that can be applied to a terminal, a server, or software running on either a terminal or a server. The terminal can be a tablet, laptop, desktop computer, etc., but is not limited to these. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Furthermore, the server can be a node server in a blockchain network, but is not limited to these. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0049] Reference Figure 2 and Figure 3 The above-mentioned remote sensing image segmentation method may include the following steps S101-S102:

[0050] S101, acquire remote sensing images;

[0051] S102, the remote sensing image is segmented using the SCFEMamba model to obtain the outline map of the ground feature area;

[0052] The SCFEMamba model is obtained by replacing the two stacked visual state space blocks in the first three decoders of the UNetMamba model with feature enhancement blocks and at least one sequentially connected improved decoding block. The feature enhancement block is used to perform feature enhancement processing on the output of the last improved decoding block to obtain the decoder output. The improved decoding block includes:

[0053] The channel splitting layer is used to split the input of the improved decoding block to obtain the first input and the second input;

[0054] The convolutional branch is used to perform convolution processing on the first input to obtain the convolutional branch feature map;

[0055] The state branch is used to perform visual state space processing on the second input to obtain the state branch feature map;

[0056] The pre-output module is used to obtain the output of the improved decoding block based on the convolution branch feature map, the state branch feature map, and the input of the improved decoding block.

[0057] In this embodiment, firstly, a remote sensing image to be segmented is acquired, which contains land feature regions, i.e., regions of interest in the remote sensing image segmentation task. Then, the remote sensing image is input into the SCFEMamba model for image segmentation processing. The SCFEMamba model is an improved neural network model based on the existing UNetMamba model, trained using multiple preset remote sensing image samples and the corresponding label information for each sample. The label information for each remote sensing image sample refers to the land feature regions contained within it. The SCFEMamba model can then segment the contour map of the land feature regions from the target remote sensing image.

[0058] Specifically, such as Figure 1 As shown, in the existing UNetMamba model, MSD serves as the decoder structure, which has four decoders, each containing two stacked VSS blocks, i.e. Figure 1 The "VSS Block×2" is shown. Based on this, as follows... Figure 2 As shown, with the input image at the top, the decoders are arranged sequentially from the fourth to the first. The SCFEMamba model in this embodiment retains the upsampling modules from the first three decoders of the MSD, i.e., it retains... Figure 1The “Patch Expanding” shown replaces the two stacked VSS blocks in the first three decoders of the MSD with a feature enhancement block and at least one sequentially connected improved decoding block, i.e., as shown in the figure. Figure 1 The “VSS Block” shown is replaced with the following: Figure 2 The “SS-Conv-SSM” and “E-FFN” blocks shown are designed to perform convolutional and visual state space processing on their inputs, while the feature enhancement blocks are used to further enhance the ground feature representation output by the last improved decoding block.

[0059] It is worth noting that the number of improved decoding blocks in each decoder is not specifically limited in this embodiment, and can be flexibly set according to actual conditions. For example, experimental verification shows that the SCFEMamba model performs best when the number of improved decoding blocks in the first and third decoders is 1, and the number of improved decoding blocks in the second decoder is 2. It should be understood that if the decoder contains two stacked improved decoding blocks, the output of the first improved decoding block serves as the input of the second improved decoding block, and the output of the second improved decoding block serves as the input of the feature enhancement block.

[0060] In the SCFEMamba model, the data flow is as follows: four multi-scale features are extracted through a ResT-based segmented encoder structure, denoted as F. ResT1 F ResT2 F ResT3 and F ResT4 Then, these four features are sequentially input into the MSD. In the MSD, the multi-scale feature F... ResT4 The input is fed into the first decoder. In the first decoder, upsampling is first performed by an upsampling module, followed by convolution and visual state space processing through at least one sequentially connected improved decoding block. Finally, feature refinement is performed on the output of the last improved decoding block by a feature enhancement block, resulting in the output F of the first decoder. SCD4 Then, the output F of the first decoder is... SCD4 With multi-scale features F ResT3 After concatenation on the channel, the data is input into the second decoder, which performs the same data stream as the first decoder. This process is repeated to obtain the four outputs F of MSD. SCD4 F SCD3 F SCD2 and F SCD1 .

[0061] More specifically, grouped convolution uses a set of convolutions (multiple kernels per layer) to facilitate the learning of various high-level and low-level features by a neural network model. This concept was first proposed in AlexNet. In recent years, some studies have found that this module can be used to improve the performance of neural network models while reducing the number of parameters and complexity. To enable the SCFEMamba model to more effectively model land cover regions in remote sensing images, inspired by grouped convolution, this implementation introduces an improved decoding block as a major component of the SCFEMamba model's MSD. Figure 3 As shown, the improved decoding block is a lightweight dual-branch structure, which includes a channel splitting layer, a convolutional branch (Conv-Branch), a state branch (SSM-Branch), and a pre-output module. The data flow is as follows:

[0062] First, a channel splitting operation is performed in the channel splitting layer, that is, the input X∈R of the improved decoding block is split. H×W×C The feature maps are segmented into two sets along the channel dimension, namely the first input and the second input. Second input Where C is the number of channels, the first input X1 is used as the input to the convolution branch, and the second input X2 is used as the input to the visual state space path.

[0063] Then, in the convolution branch, the first input X1 is convolved to obtain the convolution branch feature map. Here, the convolutional branch effectively captures local features of the ground cover region, enhancing the SCFEMamba model's ability to extract local ground cover features. Simultaneously, in the state branch, the second input X2 undergoes visual state space processing to obtain the state branch feature map. Here, state branches effectively capture long-range global dependencies of ground features, enhancing the global modeling capability of the SCFEMamba model. This allows for the extraction of both global and local information associated with the ground feature region from each set of features.

[0064] Finally, the convolutional branch feature maps are processed by the pre-output module. State branch feature map The input X of the improved decoding block is pre-output integrated, aiming to combine these three into the output X′ of the improved decoding block. Here, the input X of the improved decoding block represents the shallow, original ground features, and the convolutional branch feature map... and state branch feature map This represents deep global and local ground feature features. Through pre-output operations, it can effectively integrate shallow and deep ground feature feature information, comprehensively capturing more effective information related to the ground feature area. This can avoid the loss of information between channels caused by grouped convolution, accelerate the convergence of the SCFEMamba model by using residual connections, improve the comprehensiveness of ground feature features, and reduce the computational complexity of the SCFEMamba model.

[0065] After obtaining the output of the last improved decoding block, feature enhancement processing is performed on the output of the last improved decoding block through a feature enhancement block to obtain the decoder output. This enables the generation of richer and more representative representations of ground features, thereby improving the performance of the SCFEMamba model in remote sensing image segmentation tasks.

[0066] Therefore, this application improves the existing UNetMamba model to obtain the SCFEMamba model. Specifically, the improvement involves replacing the two stacked VSS blocks in the first three decoders of the MSD with an improved decoding block and a feature enhancement block. In the improved decoding block, the input is divided into two branches—a convolutional branch and a state branch—through grouped convolution operations of the channel segmentation layer, and parallel feature processing is performed. The convolutional branch extracts rich ground feature details from remote sensing images, such as local textures, and enhances the expression of ground feature characteristics. The state branch extracts global ground feature characteristics from remote sensing images and captures long-distance dependencies. This fully leverages the efficiency of convolutional neural networks in local feature extraction and the powerful capabilities of visual state space models in global modeling. Subsequently, the outputs of these two branches are pre-outputted with the initial input of the improved decoding block through a pre-output module. This avoids the loss of inter-channel information caused by grouped convolution while accelerating the convergence of the SCFEMamba model using residual connections, improving the comprehensiveness of ground feature characteristics, and reducing the computational complexity of the SCFEMamba model. Building upon the above, in order to generate richer and more representative representations of ground features, a feature enhancement block is introduced to enhance the output of the last improved decoding block, thereby generating richer and more representative representations of ground features and enhancing the expression of ground features.

[0067] In summary, on the one hand, the embodiments of this application introduce an improved decoding block, which enhances the SCFEMamba model's ability to perceive local ground features while also improving its ability to capture global ground features and long-distance dependencies. This helps the SCFEMamba model to capture the semantic features of complex ground features such as small target objects in remote sensing images, thus enhancing the comprehensiveness of semantic information. On the other hand, the embodiments of this application introduce a feature enhancement block to enhance the SCFEMamba model's extraction and channel fusion of ground feature representations in space, reducing the number of parameters and computational cost of the SCFEMamba model while enhancing the representation of ground features. Therefore, the embodiments of this application can effectively improve the performance of the SCFEMamba model in feature extraction and fusion of diverse ground features in complex scenes, enhancing its ability to perceive detailed and key features of ground feature regions, especially small target objects, thereby effectively improving the segmentation accuracy of remote sensing images.

[0068] The following will combine Figure 3 The improved decoding block described above will be further elaborated.

[0069] In some implementations, the above-described convolutional processing of the first input to obtain a convolutional branch feature map may include:

[0070] The first input is batch normalized to obtain the batch normalized first input; a normal convolution operation is performed on the batch normalized first input to obtain the first convolution feature map; and a point convolution operation is performed on the first convolution feature map to obtain the convolution branch feature map.

[0071] In this embodiment, in the convolution branch, firstly, batch normalization (BN) is performed on the first input X1 to obtain the batch-normalized first input X1′, aiming to ensure the stability of the input distribution. Then, a normal convolution operation is performed on the batch-normalized first input X1′ to extract local ground feature features. Specifically, a first normal convolution operation is performed on the batch-normalized first input X1′ to obtain the first input x′1 after the first normal convolution. Then, a second normal convolution operation is performed on the first input x′1 after the first convolution to obtain the first input x′2 after the second normal convolution, which is the first convolution feature map. The normal convolution operation sequentially includes convolution (Conv), batch normalization (BN), and activation. The convolution kernel size and activation function can be flexibly set according to the actual situation. For example, the convolution kernel can be 3×3, and the activation function can be ReLU activation function, but it is not limited to these. Finally, a pointwise convolution operation is performed on the first convolution feature map x′2 to obtain the convolution branch feature map. The aim is to integrate channel information. The pointwise convolution operation consists of pointwise convolution (PWConv) and activation. The activation function can be flexibly set according to the actual situation. For example, the activation function can be the ReLU activation function, but it is not limited to this.

[0072] For ease of understanding, assume that the activation functions of the convolution branches are all ReLU activation functions, then the data flow of the convolution branches can be expressed as the following formula (1):

[0073] x′1=ReLU(BN(Conv(BN(X1)))), x′2=ReLU(BN(Conv(x′1))),

[0074]

[0075] Therefore, this implementation introduces a convolution branch into the improved decoding block. By performing two consecutive ordinary convolution operations, it can effectively extract rich details of ground features in remote sensing images, such as local textures. Furthermore, by performing pointwise convolution operations, it can further integrate channel information and enhance the expression of ground feature characteristics. This enhances the SCFEMamba model's ability to perceive local ground feature characteristics and helps the SCFEMamba model to understand the semantic features of complex ground features such as small target ground features in remote sensing images, thereby enhancing the comprehensiveness of semantic information.

[0076] In some implementations, the above-described visual state space processing of the second input to obtain a state branch feature map may include:

[0077] The second input is segmented to obtain the third and fourth inputs; the third input is subjected to linear embedding and activation operations to obtain the side branch feature map; the fourth input is subjected to depthwise separable convolution and scanning attention operations to obtain the main branch feature map; and the state branch feature map is obtained based on the side branch feature map and the main branch feature map.

[0078] In this embodiment, in the state branch, the second input is first split, specifically by performing layer normalization (LN) on the second input X2 to improve the robustness of the SCFEMamba model to input changes and to adapt to lightweight design. Then, a channel splitting operation is performed, that is, the layer-normalized second input is split into a third input x1 and a fourth input x2. The third input x1 is used as the input of the side branch, while the fourth input x2 is used as the input of the main branch.

[0079] Then, in the side branch, the third input x1 is first subjected to a linear embedding operation, followed by an activation operation, to obtain the side branch feature map x″1. The aim is to complete the feature mapping through linear transformation, and then break the linear constraint through activation, so that the SCF EMamba model can fit complex patterns. The activation function can be flexibly set according to the actual situation. For example, the activation function can be the SiLU activation function, but it is not limited to this.

[0080] Simultaneously, in the main branch, a linear embedding operation is first performed on the fourth input x2 to achieve feature mapping. Then, a depthwise separable convolution operation is performed on the linearly embedded fourth input. Finally, a scanning attention operation is performed on the depthwise separable convolution fourth input to obtain the main branch feature map x″2. This effectively captures global ground feature characteristics and long-range dependencies. The depthwise separable convolution operation sequentially includes a depthwise separable convolution (DwConv) and activation. The kernel size and activation function can be flexibly set according to the actual situation; for example, the depthwise separable convolution uses a 3×3 kernel, and the activation function can be the SiLU activation function, but it is not limited to this. Furthermore, the scanning attention operation sequentially includes an attention-based SS2D operation (SS2D) and layer normalization (LN).

[0081] Finally, the side branch feature map x″1 output by the side branch and the main branch feature map x″2 output by the main branch are multiplied element-wise, and the result of the multiplication is linearly mapped to obtain the state branch feature map. The aim is to enhance the SCFEMamba model's ability to select, fuse, and express key features of geographic regions by performing feature interaction and linear adjustment on the processing results of the two branches.

[0082] For ease of understanding, assume that the activation function in the state branch is the SiLU activation function and the kernel of the depthwise separable convolution is 3×3. Then the data flow of the state branch can be expressed as the following formula (2):

[0083] x1, x2=Split(LN(X2)), x″1=SiLU(Linear(x1)),

[0084] x″2=LN(SS2D(SiLU(DWConv(Linear(x2))))),

[0085] Therefore, this implementation introduces state branches into the improved decoding block. In the state branches, a series of processes such as linear embedding, depthwise separable convolution, scan attention operation, and activation are performed to capture global ground features and long-distance dependencies. This enhances the SCFEMamba model's ability to capture global ground features and long-distance dependencies, and helps the SCFEMamba model to capture the semantic features of complex ground features such as small target ground features in remote sensing images, thereby enhancing the comprehensiveness of semantic information.

[0086] In some implementations, refer to Figure 4 The aforementioned scanning attention operation may include:

[0087] A two-dimensional selective scanning operation is performed on the input of the scanning attention operation to obtain a scanning feature map; attention operations are performed on the input of the scanning attention operation to obtain an attention weight matrix; and the output of the scanning attention operation is obtained based on the scanning feature map and the attention weight matrix.

[0088] In this embodiment, the original SS2D mechanism originates from VMamba and aims to resolve the contradiction between the non-sequential nature of visual data and the sequential nature of natural language processing tasks. The SS2D mechanism traverses the image through four scan paths (from the four corners of the image to their relative positions), ensuring that each image patch obtains contextual information only through compressed hidden states calculated along the scan paths, thereby reducing computational complexity from quadratic to linear. Considering that existing UNetMamba models do not adequately address important ground feature information in the channel and spatial dimensions, this embodiment introduces a scan attention operation in the state branch of the improved decoding block. This is an attention-based SS2D operation, referred to as the SS2D-CS mechanism.

[0089] Specifically, the output of the depthwise separable convolution operation serves as the input to the scan attention operation. First, a standard SS2D operation is performed on the input to the scan attention operation to obtain a scan feature map. Simultaneously, an attention operation is performed on the input to the scan attention operation to obtain an attention weight matrix. Then, the scan feature map and the attention weight matrix are multiplied element-wise to fuse the information from the two branches. This introduces an attention mechanism into the original SS2D mechanism, enhancing the expressive power of ground features. Compared to simple concatenation, element-wise multiplication allows for a more refined fusion of ground feature information and avoids information mixing, enabling the SCFEMamba model to focus more on key ground features and improve the relevance and accuracy of ground feature representation.

[0090] More specifically, the SS2D operation is an existing technology. Simply put, first, a four-directional cross-scan operation is performed, that is, the input patches of the SS2D module are scanned and traversed along the top, bottom, left, and right directions. Each image patch in the input is flattened into a one-dimensional vector, resulting in initial feature sequences for the four directions. This design allows each image patch to acquire global contextual information from different directions, while avoiding the computational bottleneck caused by the quadratic complexity in traditional self-attention mechanisms. Then, an independent S6 block is set for each direction. Feature extraction is performed on the corresponding initial feature sequences of each S6 block to obtain the ground feature sequence for each S6 block, i.e., the ground feature sequence for the four directions. The S6 blocks retain feature information relevant to the ground feature area through a selection mechanism, while filtering out feature information irrelevant to the ground feature area. Finally, the ground feature sequences for the four directions are recombine (cross-merge) to obtain a complete, two-dimensional scanned feature map (Output Patches), thereby restoring the spatial structure of the feature map.

[0091] In the Channel-Spatial Branch, Channel Attention (CA) and Spatial Attention (SA) mechanisms are introduced. First, Channel Attention is performed on the input of the attention module to obtain the Channel Attention Feature Map. Then, Spatial Attention is performed on the Channel Attention Feature Map to obtain the Attention Weight Matrix.

[0092] Here, the channel attention mechanism dynamically calibrates weights by quantifying the contribution of each channel, enabling the SCFEMamba model to adaptively assign high weights to key channels and low weights to secondary channels, thereby accurately learning the differentiated contributions of different channels. This design captures the dependencies between channels and automatically adjusts the responses of feature channels, significantly improving the feature representation capabilities of the SCFEMamba model. The channel attention mechanism works as follows: First, the input is convolved, then the feature map is compressed using global average pooling, and the global average value for each channel is calculated. This average value reflects the global feature response of each channel. Next, these average values ​​are processed using an activation function (such as ReLU) to generate the weights for each channel. Finally, these weights are multiplied by the original features to obtain the channel attention feature map, thus completing the dynamic weighting of feature channels. This process not only helps the SCFEMamba model understand the data more effectively but also highlights important features, thereby improving the overall performance and generalization ability of the SCFEMamba model.

[0093] The core idea of ​​the spatial attention mechanism is to enable the SCFEMamba model to learn which spatial regions in the feature map are important for the current task, thereby improving the performance and generalization ability of the SCFEMamba model. The process of the spatial attention mechanism is as follows: First, a global pooling operation is performed on the input along the channel dimension. The pooling method is usually max pooling or average pooling, resulting in two feature maps with one channel. Second, the two pooled feature maps with one channel are concatenated along the channel dimension to combine the advantages of the two pooling methods. Then, the concatenated feature map is converted into a 1-channel feature map through a convolutional layer. Finally, the feature map is fed into an activation function to be normalized to the range [0, 1], resulting in the final weight matrix of the spatial attention mechanism, i.e., the attention weight matrix.

[0094] It is worth noting that the execution of the channel attention mechanism before the spatial attention mechanism stems from the logical order of ground feature processing. The channel attention mechanism can prioritize the selection of key ground feature information from the channel dimension, providing higher-quality input for the spatial attention mechanism. This allows the spatial attention mechanism to more accurately locate important spatial areas. The application of this hierarchical attention mechanism can significantly enhance the SCFEMamba model's ability to understand the vision of remote sensing images and improve its performance in remote sensing image segmentation tasks.

[0095] Therefore, the scanning attention operation provided in this embodiment introduces channel attention and spatial attention mechanisms on the basis of the SS2D mechanism. The result of the attention operation is fused with the result of the SS2D operation element-wise after multiplication to obtain the final output. This approach introduces an attention mechanism into the original SS2D mechanism, significantly enhancing its ability to express ground features. The former achieves efficient contextual information extraction through four-directional scanning and a state-space model, while the latter enhances the extraction and fusion capabilities of ground features through the attention mechanism. This combination not only improves the computational efficiency and robustness of the SCFEMamba model but also further enhances its ability to capture key ground features, addressing the problem of insufficient attention to important information in the channel and spatial dimensions of the original UNetMamba model, thereby improving its performance in remote sensing image segmentation tasks.

[0096] In some implementations, obtaining the output of the improved decoding block based on the convolutional branch feature map, the state branch feature map, and the input of the improved decoding block may include:

[0097] The convolutional branch feature map and the state branch feature map are fused to obtain a concatenated feature map; the concatenated feature map is then subjected to channel shuffling to obtain a shuffled concatenated feature map; the shuffled concatenated feature map is then fused with the input of the improved decoding block to obtain the output of the improved decoding block.

[0098] In this embodiment, in the pre-output module, firstly, the convolutional branch feature map is... and state branch feature map The concatenation of features along the channel dimension yields a concatenated feature map, aiming to initially interact and fuse the outputs of the convolutional branches and the state branches. Then, a channel shuffle operation is performed on the concatenated feature map to obtain a shuffled concatenated feature map. Channel shuffling promotes information flow between different groups of channels by rearranging the channel order, thereby improving the feature representation capability of the SCFEMamba model without significantly increasing computational cost and addressing the problem of information isolation between channels caused by grouped convolutions. Finally, the shuffled concatenated feature map and the input X of the improved decoding block are added via residual connection to obtain the output X′ of the improved decoding block. The input X of the improved decoding block represents the original ground feature information, while the shuffled concatenated feature map represents the ground feature information after deep feature extraction. By fusing the two, shallow and deep ground feature information can be effectively integrated, comprehensively capturing more effective information associated with the ground feature region.

[0099] Therefore, this implementation introduces a pre-output module into the improved decoding block, which can not only avoid the loss of information between channels caused by group convolution, but also accelerate the convergence of the SCFEMamba model by utilizing residual connections, improve the comprehensiveness of ground feature characteristics, and reduce the computational complexity of the SCFEMamba model, thereby improving the performance of the SCFEMamba model in remote sensing image segmentation tasks.

[0100] The following will combine Figure 2 and Figure 5 The feature enhancement block will be further explained.

[0101] In some implementations, refer to Figure 5 The above-described feature enhancement processing of the output of the last improved decoding block, to obtain the decoder output, may include:

[0102] The output of the last improved decoding block is processed by channel adjustment to obtain an adjusted feature map; parallel depthwise separable convolution operations are performed on the adjusted feature map to obtain multiple second convolution feature maps; the decoder output is obtained based on the multiple second convolution feature maps.

[0103] In this embodiment, the purpose of the feature enhancement block is to fuse ground feature features of different sizes to generate richer and more representative ground feature representations, thereby improving the performance of the SCFEMamba model in remote sensing image segmentation tasks. Specifically, the output of the last improved decoding block is... Figure 5 The middle is represented as a feature map F uFirst, the feature map F is processed through a 1×1 convolutional layer. u Channel-dimensional compression is performed to obtain the adjusted feature map F. u This helps reduce the computational cost and number of parameters in the SCFEMamba model while preserving important ground feature information. Then, the feature map F is adjusted. u Two parallel depthwise separable convolutions (DW-Conv) are performed to obtain two second convolutional feature maps. The kernel of the depthwise separable convolution can be flexibly set according to the actual situation, for example, it can be a 3×3 depthwise separable convolution or a 5×5 depthwise separable convolution, but it is not limited to this. Here, depthwise separable convolution is used to extract ground feature features at different scales. Replacing ordinary convolution with depthwise separable convolution allows the SCFEMamba model to effectively reduce computational complexity while maintaining good feature extraction capabilities. Finally, the two second convolutional feature maps are added together to obtain the summed feature F. u ", and then pass the summed features F through a 1×1 convolutional layer u "Perform channel adjustment to match the original input channel count, thus obtaining the decoder's output."

[0104] For ease of understanding, the data flow of the above feature enhancement block can be represented by the following formula (3):

[0105] F u ′=Conv 1×1 (F u ), F u "=DWConv(F u ′)+DWConv(F u ′),

[0106]

[0107] Therefore, this implementation introduces a feature enhancement block into the SCFEMamba model. Compared to traditional feature fusion methods, the feature enhancement block further optimizes the architecture of the SCFEMamba model through depthwise separable convolution, 1×1 convolution, and feature fusion operations. This enables the SCFEMamba model to effectively capture ground feature features at different scales, generating more comprehensive and robust ground feature representations, thereby enhancing the representational power of ground features. In this way, the ability of the SCFEMamba model to capture detailed and global information of ground features, as well as its ability to extract spatial features and fuse channels, is effectively enhanced. It also achieves a dual compression of computational complexity and parameter quantity, thus significantly improving the performance of the SCFEMamba model in remote sensing image segmentation tasks, especially inference efficiency and inference accuracy.

[0108] The following will combine Figure 2 andFigure 6 This section introduces other improvements to the SCFEMamba model.

[0109] In some implementations, refer to Figure 6 The aforementioned SCFEMamba model is further obtained by replacing the last layer of the UNetMamba model with a feature refinement block, which includes:

[0110] The input refinement module is used to obtain the fifth input based on the output of the first encoder and the output of the last decoder;

[0111] The channel path is used to refine the features of the fifth input along the channel dimension, resulting in a channel-refined feature map.

[0112] Spatial path, used to refine the features of the fifth input in the spatial dimension, to obtain a spatially refined feature map;

[0113] The refinement output module is used to obtain the outline map of the ground feature area based on the channel refinement feature map, the spatial refinement feature map, and the fifth input.

[0114] In this embodiment, to enhance the refinement of feature mapping, in addition to introducing the aforementioned improved decoding block and feature enhancement block into the original UNetMamba model, this embodiment also introduces a feature refinement block into the UNetMamba model. Specifically, as follows... Figure 1 As shown, the last layer of the original UNetMamba model is a 1×1 convolutional layer, which is as follows: Figure 1 The “1×1 Conv” shown is used for dimensionality reduction and outputting the final segmentation result. Accordingly, this implementation replaces the 1×1 convolutional layer with a feature thinning block, that is, as shown in the figure. Figure 1 Replace “1×1Conv” as shown below with: Figure 2 The “FRH” feature refinement block shown aims to optimize features by bridging spatial details with semantic information.

[0115] More specifically, the feature refinement block includes a refinement input module, a channel path, a spatial path, and a refinement output module, with the data flow as follows:

[0116] In the input refinement module, the output F of the first encoder is... ResT1 (i.e., shallow spatial features of ground features) and the output F of the last decoder SCD1The weighted summation operation is performed on the deep semantic features of the land cover. This method achieves the complementary advantages of the two types of land cover features by constructing a learnable weight allocation mechanism, and finally forms a fused feature map with both semantic accuracy and spatial detail capability. Then, a 3×3 convolutional layer is used to process the fused feature map to enhance its feature representation, thereby forming the fifth input, which is input into the channel path and spatial path respectively.

[0117] In the channel path, the fifth input undergoes feature refinement processing along the channel dimension, resulting in a channel-refined feature map. This aims to enhance the semantic expressive power of ground features by adjusting the importance of each channel in the fifth input, filtering key channel information, reducing noise and redundant ground feature information, and thus improving the SCFEMamba model's ability to perceive key ground features. In the spatial path, the fifth input undergoes feature refinement processing along the spatial dimension, resulting in a spatially refined feature map. This aims to improve the detection accuracy of the SCFEMamba model by locating important ground feature regions, reducing the interference of noise and redundant ground feature information on the segmentation results.

[0118] The channel refinement feature map, spatial refinement feature map, and the fifth input are fed into the refinement output module. In the refinement output module, the channel refinement feature map and the spatial refinement feature map are first fused together using a summation operation (Sum) to form the target refinement feature map. This aims to perform a preliminary interaction between the refined land feature features in the channel dimension and the refined land feature features in the spatial dimension, integrating important information about the land feature region in both the spatial and channel dimensions.

[0119] Then, the target refined feature map is sequentially subjected to depth-wise convolution (DIV), batch normalization (BAT ch Norm), and channel adjustment (DIV 1×1) to obtain the adjusted target refined feature map, which aims to further refine and reduce the dimensionality of ground feature features. The channel adjustment uses a 1×1 convolutional layer. The kernel size of the depth-wise convolution can be flexibly set according to the actual situation; for example, the kernel of the depth-wise convolution can be 3×3, but it is not limited to this.

[0120] Next, the fifth input residual connection is connected to the adjusted target refined feature map, and the two are further fused by summation to obtain a fused feature map. This not only alleviates the gradient vanishing problem and prevents the SCFEMamba model from degrading, but also improves the stability of the SCFEMamba model, enabling it to learn and express complex semantic features of land features more stably.

[0121] Finally, the fused feature map is sequentially activated, channel-adjusted (Conv 1×1), and upsampled to obtain the outline map of the ground feature region (Output map). This final processing reduces the output dimensionality and ensures the segmentation accuracy of the remote sensing image and the stability of the SCFEMamba model. The activation function and the number of upsampling operations can be flexibly set according to the actual situation; for example, the activation function can be the ReLU activation function, and the number of upsampling operations can be four, but it is not limited to these.

[0122] Therefore, compared to the semantic and spatial information imbalance problem in the existing UNetMamba model when fusing ground feature, the feature refinement block shows significant advantages. By using parallel spatial and channel paths, it can enhance the expression of ground feature features from the spatial and channel dimensions respectively. The former accurately locates important ground feature regions, while the latter filters key channel information of ground feature regions. The collaboration of the two can compensate for the shortcomings of single-dimensional processing and comprehensively improve the quality of ground feature features. In subsequent processing, through the first summation operation and subsequent depthwise separable convolution, batch normalization, and channel adjustment, the features of the two paths can be effectively fused and refined, thereby integrating the important information of ground feature regions in the spatial and channel dimensions. Through the second summation operation and subsequent activation, channel adjustment, and upsampling, on the one hand, it can not only alleviate the gradient vanishing problem and prevent the SCFEMamba model from degrading, but also improve the stability of the SCFEMamba model, enabling it to learn and express complex ground feature semantic features more stably. On the other hand, it can reduce the output dimension of the contour map and ensure the segmentation accuracy of remote sensing images and the stability of the SCFEMamba model.

[0123] Thus, the shallow features generated by the first encoder contain rich spatial details but lack semantic representation capabilities; while the global-local fusion features output by the last decoder possess deep semantic expression capabilities but lose spatial resolution. Although directly fusing the two types of features has computational advantages, it may lead to an imbalance between spatial and semantic information. To address this, the feature refinement block, through an adaptive feature calibration mechanism, can effectively compensate for the semantic gap between shallow and deep ground feature features, enabling the SCFEMamba model to significantly improve segmentation accuracy while maintaining computational efficiency.

[0124] In some implementations, the above-described feature refinement processing of the fifth input along the channel dimension to obtain a channel-refined feature map may include:

[0125] The fifth input is pooled to obtain a channel attention map; the channel attention map is adjusted to obtain an adjusted channel attention map; the adjusted channel attention map is activated to obtain channel attention weights; and the channel refinement feature map is obtained based on the fifth input and the activated channel attention map.

[0126] In this embodiment, in the channel path, firstly, a global average pooling operation is performed on the fifth input to generate a channel attention map C′∈R. 1×1×C Then, channel adjustment processing is performed on the channel attention map C′ to obtain the adjusted channel attention map. Specifically, a dimensionality reduction operation is first performed to reduce the number of channels by a factor of four, in order to extract key ground feature features by compressing the number of channels. Then, an expansion operation is performed to restore the number of channels to C, in order to enhance the expression of ground feature features by restoring the number of channels. After that, an activation operation is performed on the adjusted channel attention map to obtain the activated channel attention map, which introduces non-linear changes, enabling the SCFEMamba model to learn complex ground feature patterns. The activation function can be flexibly set according to the actual situation, for example, the activation function can be the sigmoid function, but it is not limited to this. Finally, matrix multiplication is performed on the fifth input and the activated channel attention map to obtain the channel refined feature map. This allows for dynamic adjustment of the ground feature features in the fifth input according to the channel attention weights, enhancing the sensitivity of the SCFEMamba model to key ground feature information.

[0127] As can be seen, this implementation introduces channel paths into the feature refinement block, which dynamically adjusts the importance of each channel by assigning different weights to each channel, and filters important channel information, thereby enhancing the semantic expression of ground features, reducing noise and redundant information, and thus improving the SCFEMamba model's ability to perceive key ground features, which helps to improve its performance in remote sensing image segmentation tasks.

[0128] In some implementations, the above-described feature refinement processing of the fifth input in the spatial dimension to obtain a spatially refined feature map may include:

[0129] A depthwise separable convolution operation is performed on the fifth input to obtain a spatial attention map; an activation operation is performed on the spatial attention map to obtain an activated spatial attention map; and a spatial refinement feature map is obtained based on the fifth input and the activated spatial attention map.

[0130] In this embodiment, firstly, a depth-wise convolution is performed on the fifth input to generate a spatial attention map S∈R.H×W×1 Then, an activation operation is performed on the spatial attention map S to obtain an activated spatial attention map, introducing nonlinear changes that enable the SCFEMamba model to learn complex ground feature patterns. The activation function can be flexibly set according to the actual situation; for example, it can be the sigmoid function, but it is not limited to this. Finally, matrix multiplication is performed on the fifth input and the activated spatial attention map to obtain a spatially refined feature map. This allows for dynamic adjustment of ground feature features in the fifth input based on the spatial attention weights, enhancing the SCFEMamba model's sensitivity to key ground feature information.

[0131] As can be seen, this implementation introduces spatial paths into the feature refinement block, which can locate important ground features and effectively suppress the interference of background and noise on remote sensing image segmentation, thereby further improving the performance of the SCFEMamba model in remote sensing image segmentation tasks.

[0132] In some implementations, the above-described SCFEMamba model is also obtained by removing two stacked visual state space blocks from the last decoder of the UNetMamba model.

[0133] In this embodiment, in addition to introducing feature refinement blocks, feature enhancement blocks, and improved decoding blocks into the aforementioned SCFEMamba model, the two stacked visual state space blocks in the last decoder of the UNetMamba model are also removed, i.e., as shown below. Figure 1 As shown, with the input image at the top, the fourth decoder is arranged from the top down to the first decoder. The "VSS Block×2" in the fourth decoder is deleted to achieve lightweight deployment.

[0134] The performance of the SCFEMamba model (i.e., the SCFEMamba model) provided in the embodiments of this application will be verified below.

[0135] (I) Datasets: The publicly available datasets selected in this embodiment are the LoveDA land use and cover dataset, the ISPRS Vaihingen dataset, and the ISPRS Potsdam dataset. The specific information of these three datasets is shown in Table 1 below.

[0136] Table 1: Details of the dataset

[0137]

[0138] (II) Experimental Setup: All models in this experiment were implemented using the PyTorch framework on four NVIDIA RTX 3060 GPUs. For the LoveDA, Vaihinge, and Potsdam datasets, images were randomly cropped to a size of 512×512. During training, data augmentation techniques such as random scaling (specifically [0.5, 0.75, 1.0, 1.25, 1.5]), random vertical flipping, random horizontal flipping, and random rotation were employed. The number of training iterations was set to 100, and the batch size was 6. During the testing phase, multi-scale and random flipping augmentations were used. To achieve fast convergence, the AdamW optimizer was deployed to train all models in the experiment. The base learning rate was set to 6e-4, and a cosine learning strategy was used to adjust the learning rate.

[0139] The loss function used in this embodiment consists of a pixel-level classification cross-entropy loss function and a Dice loss function focused on the boundary region. The goal of the cross-entropy loss function is to minimize the difference between the negative log-likelihood of the one-hot encoded class label and the predicted probability. The cross-entropy loss function is defined as shown in the following formula (4): (4); In equation (4), This refers to the cross-entropy loss function. and These represent the number of samples and the number of categories, respectively. and Indicates the first The first data sample The label of each category and the model prediction of the corresponding sample's first category. The confidence level of each category.

[0142] Dice loss improves segmentation accuracy by optimizing the overlap between the predicted and real regions, and is very suitable for imbalanced datasets. The definition of the Dice loss function is shown in the following formula (5):

[0143]

[0144] In equation (5), L dice This refers to the Dice loss function.

[0145] In addition, this embodiment also incorporates a cross-entropy auxiliary loss function to learn semantic features from the auxiliary head. The auxiliary head extracts features from each decoder and then performs upsampling and summation operations to obtain an intermediate semantic segmentation map. The definition of the auxiliary loss function is shown in the following formula (6):

[0146]

[0147] In equation (6), L aux Denotes the auxiliary loss function. Let represent the semantic segmentation map from the intermediate decoder. The overall loss function is a weighted sum of the above loss functions, as shown in the following formula (7):

[0148] L total =L ce +L dice +αL aux (7);

[0149] In equation (7), L total α is the overall loss function; α is the weight of the auxiliary loss function, for example, it is set to 0.4, but is not limited to this.

[0150] (III) Performance metrics: Overall accuracy (OA), F1 score, and mean intersection over union (mIoU) were selected as metrics for evaluating model performance.

[0151] (IV) Comparative Experiments: To verify the effectiveness of the SCFEMamba model proposed in this embodiment, eight sets of comparative experiments were conducted. The comparative models included current mainstream semantic segmentation methods, the baseline model of this embodiment, and the improved model proposed in this embodiment. These included the UNet and ABCNet models based on the CNN architecture; the BANet, UNetFormer, and DCSWin models based on the Transformer architecture; the PyramidMamba model based on the Mamba architecture; plus the baseline model UNetMamba and the improved model SCFEMamba of this embodiment, totaling eight sets of comparative experiments. These six comparative experimental models were tested on the three publicly available datasets mentioned above and compared with the UNetMamba and improved models SCFEMamba of this embodiment, as shown in Tables 2 to 4.

[0152] Table 2: Performance of all models on the LoveDA dataset

[0153]

[0154] As shown in Table 2, the SCFEMamba model performs exceptionally well on the LoveDA dataset, especially in terms of mIoU, achieving 54.14%, significantly higher than other models. It improves upon the baseline model UNetMamba by 2.13%, surpasses the best CNN-based model ABCNet by 5.05%, and exceeds the best Transformer-based model BANet by 3.47%. This indicates that SCFEMamba has a significant advantage in semantic segmentation accuracy, enabling more accurate identification and segmentation of different semantic regions in remote sensing images. Furthermore, SCFEMamba also boasts the highest F1 score and OA among all models, improving upon the baseline model by 2.1% and 2.35%, respectively. This demonstrates its excellent recall capability in high-resolution remote sensing image segmentation tasks, effectively identifying more correct segmentation results, and exhibiting better overall classification accuracy across all categories. In summary, the SCFEMamba model achieved the highest scores on all three key metrics, demonstrating its significant advantages in remote sensing image semantic segmentation tasks. Its superiority stems from its unique Mamba-based network structure, which enables it to better capture and utilize multi-scale features and contextual information in remote sensing images.

[0155] Table 3: Performance of all models on the Vaihingen dataset

[0156]

[0157] As shown in Table 3, SCFEMamba performs exceptionally well on the Vaihingen dataset, particularly in mIoU and the benchmark, achieving 77.14%, higher than all other models and 0.32% higher than the second-place baseline model. The improved mIoU indicates that SCFEMamba's segmentation results are closer to the true labels, demonstrating its high accuracy in semantic segmentation. SCFEMamba also achieved the highest F1 score in the table, surpassing the baseline model UNetMamba, further proving its superiority in semantic segmentation. SCFEMamba's overall accuracy is 87.91%, very close to the first-place model's 87.98%, differing by only 0.07%, maintaining high accuracy. Although the advantage in this metric is not significant, SCFEMamba still maintains a high overall performance.

[0158] Table 4: Performance of all models on the Potsdam dataset

[0159]

[0160] As shown in Table 4, on the Potsdam dataset, the SEFEMamba model has the highest F1 score of 87.42%, significantly higher than the baseline model UNetMamba's 86.89%. This indicates that SCFEMamba performs better in balancing precision and recall. SCFEMamba's OA is 85.89%, slightly lower than UNetMamba's 86.79%. This is because the SCFEMamba model replaced the pure SSM architecture with an SS-Conv-SSM architecture in the encoder. Compared to the pure SSM architecture, SS-Conv-SSM introduces local perception capabilities, which slightly weakens the global modeling ability, resulting in a decrease in OA. However, the model's mIoU and F1 score are improved, proving that the SCFEMamba model has stronger overall performance and performs better than UNetMamba. Furthermore, compared to Transformer-based models, SCFEMamba's F1 score is higher than all Transformer models in the table, but its operational accuracy (OA) is lower. This is because SCFEMamba's global modeling ability is slightly weaker than that of Transformer-based models. However, due to the introduction of the SSM architecture, SCFEMamba's computational complexity is much lower than that of Transformer models, and its mIoU is higher than all other models. This indicates that SCFEMamba is more accurate in identifying target regions and delineating boundaries in segmentation tasks. Finally, compared to CNN-based models, SCFEMamba's F1 score, OA, and mIoU are all higher. This is because the SCFEMamba model has better global modeling capabilities than CNN models.

[0161] In summary, the SCFEMamba model performs exceptionally well in remote sensing image semantic segmentation tasks, particularly in F1 score and mIoU, significantly outperforming the baseline model UNetMamba and other comparative models. This demonstrates SCFEMamba's clear advantages in balancing accuracy and recall, target region identification, and boundary delineation, thus proving its potential in practical applications.

[0162] (V) Ablation Experiment:

[0163] To further verify the contribution of each module in the SCFEMamba model to the overall model, ablation experiments will be conducted on each module: SS-Conv-SSM, SS2D-CS, E-FFN, and FRH. Eight ablation experiments were designed on the LoveDA dataset. The UNetMamba model was used as the baseline for these experiments. The results are shown in Table 5. Eight different configurations were designed for this ablation experiment, each corresponding to a row in Table 5. By adding or removing specific modules, the changes in the model's F1 score, OA, and mIoU on the LoveDA dataset for semantic segmentation were observed.

[0164] Table 5: Ablation experimental results of each module in the SCFEMamba model on LoveDA data

[0165]

[0166] In Table 5: The baseline model is exp_1, which retains its basic structure without any modules. At this point, the F1 score is 67.71%, OA is 70.01%, and mIoU is 52.01%, providing a performance foundation for subsequent experiments. exp_2 is an experiment where the SS-Conv-SSM module is added alone to the baseline model. The F1 score improves to 68.19%, OA slightly improves by 0.61%, and mIoU improves by 0.62%, indicating that the SS-Conv-SSM module has a positive contribution to model performance. exp_3 is an experiment where the FRH module is added alone to the baseline model. Its F1 score improves to 68.34%, OA improves to 70.51%, and mIoU improves to 52.73%. exp_4 is an experiment where the SS2D-CS module is added alone. Its F1 score, OA, and mIoU all improve, increasing by 0.52%, 0.89%, and 0.79%, respectively. exp_5 adds the E-FFN module to the baseline model, which shows the highest improvement in F1 score compared to other modules. exp_5 demonstrates that all the improved modules proposed in this application improve model performance. Next, a module stacking experiment is conducted to verify the effectiveness of module combination on model performance. exp_6 adds the FRH module to exp_2, further improving the F1 score to 68.72%, OA to 71.72%, and mIoU to 53.18%, representing a significant improvement over the baseline model. exp_7 introduces the SS2D-CS module to exp_6, improving the F1 score by 1.31% compared to the baseline model, while also improving the OA by 1.9% and the mIoU by 1.61%. exp_8 incorporates all modules into the baseline model, resulting in the SCFEMamba model. Compared to the baseline model, its F1 score, OA, and mIoU values ​​are improved by 1.9%, 2.35%, and 2.13%, respectively. The experimental results show that each module in the SCFEMamba model contributes positively to the overall performance. This indicates that these modules play a crucial role in semantic segmentation tasks, effectively improving the model's recognition accuracy and robustness. Furthermore, the experiments reveal synergistic effects between modules. For example, the combination of the SS-Conv-SSM and FRH modules outperforms either module alone, while adding the SS2D-CS module further enhances performance.

[0167] (vi) Results Visualization:

[0168] On the LoveDA dataset test set, this application embodiment used representative models for semantic segmentation result visualization, namely the U-Net model based on CNN architecture, the UNetFormer model based on Transformer architecture, the UNetMamba model based on Mamba architecture, and the SCFEMamba model proposed in this application embodiment. The semantic segmentation result visualization is as follows: Figure 7 As shown, Figure 7 In this context, "Nirrg images" refers to remote sensing images, "Building" refers to buildings, "Road" refers to roads, "Water" refers to rivers, "Barren" refers to wasteland, "Forest" refers to forests, "Agriculture" refers to farmland, "Background" refers to the background, and "GT" refers to standard outline maps.

[0169] Based on the classification of the LoveDA dataset, semantic segmentation experiments were conducted on the two main categories of rural and urban areas in the figure. The first column represents the original remote sensing image patch, the second column represents the corresponding ground truth label, and columns 4-6 represent the corresponding model prediction results. These models show little difference in segmenting large objects such as water bodies and roads; the performance difference mainly lies in segmenting small targets and obfuscated objects. The following section uses two typical segmented regions as examples to illustrate and analyze the visualization results.

[0170] In the rural segmentation results, the building prediction class is circled by the dashed box. In the U-Net model, the prediction results are blurry, the edge information is unclear, and there are missed detections. In the UNetFormer model, this phenomenon is improved, and the edge information of the detected buildings is clearer, but there are still missed detections. In the UNetMamba model, the edge information of the detected buildings at the same location is not much different from the prediction results of the previous model. Until the SCFEMamba model proposed in this application embodiment, the edge information of the detected buildings at the same location is clearly visible, and the missed detections are greatly reduced. In the semantic segmentation results, the detection of the areas where open land, forest, and water meet is more difficult, as shown by the circles in the figure. SCFEMamba has the best segmentation results in this part, restoring the label map to the greatest extent. In contrast, the baseline model UNetMamba has a large number of missed detections in the segmentation results at the same location.

[0171] In the segmentation results of the Urban series, buildings are the main components of the city, occupying the largest area. Therefore, for semantic segmentation of the city, building segmentation accounts for the vast majority. As shown in the box in the figure, the segmentation effect of buildings in the UNet model is extremely poor, especially the segmentation of the edges between buildings, which has almost no segmentation effect. In the UNetFormer and UNetMamba models, the segmentation results at this position are not much different from U-Net, with almost no improvement. Until the SCFEMamba model proposed in the embodiments of this application, the label map is restored to the greatest extent, and each building is very well segmented at the building segmentation point, with the highest segmentation accuracy.

[0172] In summary, this application provides a remote sensing image segmentation method based on the SCFEMamba model, aiming to address numerous challenges in semantic segmentation of high-resolution remote sensing images. Addressing the shortcomings of traditional methods, such as the difficulty of CNN architecture in extracting global contextual information, the high computational complexity and large number of parameters of Transformer architecture, and the deficiencies of Mamba architecture in feature extraction and fusion in complex scenes, this application makes a series of improvements to the existing UnetMamba model. It introduces the SS-Conv-SSM module to replace the VSS module in the decoder, which combines the local feature extraction capability of the convolutional branch with the global modeling capability of the SSM branch, improving the comprehensiveness of features. The SS-Conv-SSM module retains the SS2D mechanism, adding a branch combining channel attention and spatial attention to enhance the model's ability to capture key features. An E-FFN module is introduced at the output of each layer of the decoder to effectively fuse features at different scales, enhancing feature representation capabilities. Finally, an FRH module is introduced at the last layer of the model to fully fuse shallow spatial information and deep semantic information, improving segmentation accuracy.

[0173] Experimental results on the publicly available datasets LoveDA, ISPRS Vaihingen, and ISPRS Potsdam demonstrate that the SCFEMamba model performs exceptionally well in semantic segmentation tasks. Compared to mainstream contrasting models, it significantly outperforms the baseline model UNetMamba and other models in terms of F1 score and mIoU. On the LoveDA dataset, SCFEMamba achieves an mIoU of 54.14%, a 2.13% improvement over UNetMamba; on the Vaihingen dataset, its mIoU reaches 77.14%, higher than all other models; and on the Potsdam dataset, its F1 score reaches 87.42%, also higher than the baseline model and other contrasting models. This fully demonstrates that SCFEMamba has significant advantages in balancing accuracy and recall, target region identification, and boundary segmentation.

[0174] Ablation experiments further validated the positive contributions of each module—SS-Conv-SSM, SS2D-CS, E-FFN, and FRH—to the overall performance, and revealed a synergistic effect among the modules, collectively improving the semantic segmentation accuracy of the model. Visual analysis of the experimental results showed that SCFEMamba performed exceptionally well in segmenting small and obfuscated objects, clearly restoring object edge information and reducing missed detections, demonstrating higher segmentation accuracy compared to other models.

[0175] Furthermore, embodiments of this application also provide a remote sensing image segmentation apparatus, which may include:

[0176] The acquisition module is used to acquire remote sensing images;

[0177] The segmentation module is used to segment remote sensing images using the SCFEMamba model to obtain the outline map of the ground feature area;

[0178] The SCFEMamba model is obtained by replacing the two stacked visual state space blocks in the first three decoders of the UNetMamba model with feature enhancement blocks and at least one sequentially connected improved decoding block. The feature enhancement block is used to perform feature enhancement processing on the output of the last improved decoding block to obtain the decoder output. The improved decoding block includes:

[0179] The channel splitting layer is used to split the input of the improved decoding block to obtain the first input and the second input;

[0180] The convolutional branch is used to perform convolution processing on the first input to obtain the convolutional branch feature map;

[0181] The state branch is used to perform visual state space processing on the second input to obtain the state branch feature map;

[0182] The pre-output module is used to obtain the output of the improved decoding block based on the convolution branch feature map, the state branch feature map, and the input of the improved decoding block.

[0183] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0184] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0185] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A remote sensing image segmentation method, characterized in that, Includes the following steps: Acquire remote sensing images; The remote sensing image is segmented using the SCFEMamba model to obtain the outline map of the ground feature area; The SCFEMamba model is obtained by replacing two stacked visual state space blocks in the first three decoders of the UNetMamba model with feature enhancement blocks and at least one sequentially connected improved decoding block. The feature enhancement block is used to perform feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder. The improved decoding block includes: A channel segmentation layer is used to segment the input of the improved decoding block to obtain a first input and a second input; The convolutional branch is used to perform convolution processing on the first input to obtain the convolutional branch feature map; The state branch is used to perform visual state space processing on the second input to obtain the state branch feature map; The pre-output module is used to obtain the output of the improved decoding block based on the convolutional branch feature map, the state branch feature map, and the input of the improved decoding block.

2. The method according to claim 1, characterized in that, The step of performing convolution processing on the first input to obtain a convolutional branch feature map includes: The first input is batch normalized to obtain the batch normalized first input; Perform a regular convolution operation on the first input after batch normalization to obtain the first convolution feature map; Perform pointwise convolution on the first convolutional feature map to obtain the convolutional branch feature map.

3. The method according to claim 1, characterized in that, The step of performing visual state space processing on the second input to obtain a state branch feature map includes: The second input is segmented to obtain the third and fourth inputs; Perform linear embedding and activation operations on the third input to obtain a side branch feature map; The fourth input is subjected to depthwise separable convolution and scanning attention operations to obtain the main branch feature map; The state branch feature map is obtained based on the side branch feature map and the main branch feature map.

4. The method according to claim 3, characterized in that, The scanning attention operation includes: A two-dimensional selective scanning operation is performed on the input of the scanning attention operation to obtain a scanning feature map; An attention operation is performed on the input of the scanning attention operation to obtain an attention weight matrix; The output of the scanning attention operation is obtained based on the scan feature map and the attention weight matrix.

5. The method according to claim 1, characterized in that, The step of performing feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder includes: The output of the last improved decoding block is subjected to channel adjustment processing to obtain an adjusted feature map; Parallel depthwise separable convolution operations are performed on the adjusted feature maps to obtain multiple second convolutional feature maps; The output of the decoder is obtained based on multiple second convolutional feature maps.

6. The method according to claim 1, characterized in that, The SCFEMamba model is further obtained by replacing the last layer of the UNetMamba model with a feature refinement block, which includes: A refined input module is used to obtain the fifth input based on the output of the first encoder and the output of the last decoder; The channel path is used to refine the features of the fifth input in the channel dimension to obtain a channel-refined feature map. The spatial path is used to refine the features of the fifth input in the spatial dimension to obtain a spatially refined feature map. The refinement output module is used to obtain the outline map of the land feature area based on the channel refinement feature map, the spatial refinement feature map, and the fifth input.

7. The method according to claim 6, characterized in that, The step of refining the features of the fifth input along the channel dimension to obtain a channel-refined feature map includes: The fifth input is pooled to obtain a channel attention map; The channel attention map is subjected to channel adjustment processing to obtain the adjusted channel attention map; The adjusted channel attention map is activated to obtain the activated channel attention map. The channel refinement feature map is obtained based on the fifth input and the activated channel attention map.

8. The method according to claim 6, characterized in that, The step of refining the features of the fifth input in the spatial dimension to obtain a spatially refined feature map includes: Perform a depthwise separable convolution on the fifth input to obtain a spatial attention map; The spatial attention map is activated to obtain the activated spatial attention map. The spatial refinement feature map is obtained based on the fifth input and the activated spatial attention map.

9. The method according to claim 1, characterized in that, The SCFEMamba model is also obtained by removing the two stacked visual state space blocks in the last decoder of the UNetMamba model.

10. A remote sensing image segmentation device, characterized in that, include: The acquisition module is used to acquire remote sensing images; The segmentation module is used to segment the remote sensing image using the SCFEMamba model to obtain the outline map of the ground feature area; The SCFEMamba model is obtained by replacing two stacked visual state space blocks in the first three decoders of the UNetMamba model with feature enhancement blocks and at least one sequentially connected improved decoding block. The feature enhancement block is used to perform feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder. The improved decoding block includes: A channel segmentation layer is used to segment the input of the improved decoding block to obtain a first input and a second input; The convolutional branch is used to perform convolution processing on the first input to obtain the convolutional branch feature map; The state branch is used to perform visual state space processing on the second input to obtain the state branch feature map; The pre-output module is used to obtain the output of the improved decoding block based on the convolutional branch feature map, the state branch feature map, and the input of the improved decoding block.

Images