Remote sensing image segmentation method and device

Abstract

The application discloses a remote sensing image segmentation method and device, which is applied to the technical field of remote sensing image segmentation and comprises the following steps: performing segmentation processing on a remote sensing image by using an SCFEMamba model to obtain an outline map of a ground object area, wherein the model is obtained by replacing two visual state space blocks stacked in the first three decoders of a UNetMamba model with a feature enhancement block and at least one improved decoding block connected in sequence; in the improved decoding block, input of the improved decoding block is divided into a first input and a second input, convolution processing is performed on the first input to obtain a convolution branch feature map, visual state space processing is performed on the second input to obtain a state branch feature map, and output of the improved decoding block is obtained according to the convolution branch feature map, the state branch feature map and the input of the improved decoding block; and the feature enhancement block performs feature enhancement processing on output of the last improved decoding block. The application effectively improves the segmentation precision of the remote sensing image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image segmentation, and particularly relates to a remote sensing image segmentation method and device. BACKGROUND

[0002] With the rapid development of remote sensing technology, high-resolution remote sensing images have become the main source of obtaining rich ground object information on the earth's surface. Semantic segmentation of ground objects in these high-resolution remote sensing images has practical significance and plays a crucial role in multiple downstream tasks. However, the gradual enrichment of detailed information in high-resolution images leads to more complex appearance characteristics of ground objects, which brings major challenges to segmentation accuracy and segmentation efficiency. Generally, the unique challenges of remote sensing images come from the diversity of targets and the variability of scales under high-viewing-angle shooting conditions, including low recognition of objects with similar color and shape, widespread existence of small-size targets, complex and variable background, and complex edge details.

[0003] With the development of computer vision technology, a variety of computer vision basic models have emerged, such as models based on convolutional neural network (CNN) architecture and models based on Transformer architecture, which have enabled high-resolution remote sensing images to flourish in the field of semantic segmentation.

[0004] Among them, in the model based on CNN architecture, the U-Net model is a milestone in semantic segmentation, which establishes a basic architecture for subsequent work, i.e., an encoder-decoder architecture with skip connections. In addition, the DeepLab model introduces atrous convolution and spatial pyramid pooling, which improves segmentation accuracy and speed; the PSPNet model uses a pyramid pooling module to obtain semantic information at different scales, which helps to accurately divide object boundaries. Although CNNs show significant advantages in local feature extraction, their limited receptive field limits the ability to model global context, which can lead to spatial information loss when processing high-resolution remote sensing images, thereby affecting segmentation accuracy.

[0005] The model based on the Transformer architecture introduces a self-attention mechanism, which brings breakthroughs in the fields of natural language processing and computer vision, and exhibits significant advantages in modeling long-distance dependencies and capturing global semantic associations, effectively overcoming the limitations of the local receptive field of traditional CNNs. Some related technologies innovatively embed Transformer modules into U-shaped network frameworks, which greatly improves the accuracy of high-resolution remote sensing images in semantic segmentation. Inspired by this, subsequent research gradually shifted to the Transformer-based technical route. However, the intrinsic quadratic computational complexity and massive parameter quantity of Transformer pose a challenge to its efficiency in processing high-resolution remote sensing images, which is an important challenge to its practical application.

[0006] Recently, the Mamba architecture has become a new hotspot, with its linear time complexity and strong long-distance dependency modeling capability, attracting widespread attention in the field of remote sensing image semantic segmentation. The Mamba architecture successfully solves the quadratic computational complexity problem of traditional Transformer models, while ensuring accuracy through its competitive long-distance dependency modeling capability. With the continuous development of Mamba technology in the field of computer vision, models based on the Mamba architecture have emerged in the field of remote sensing image semantic segmentation, such as the UNetMamba model obtained by embedding the Mamba architecture into the U-shaped network framework. Although the UNetMamba model has made significant progress in performing remote sensing image segmentation tasks, it has deficiencies in feature extraction and fusion of diverse features in complex scenes, making it difficult to capture the details and key feature differences between different feature targets, resulting in blurred boundaries and class confusion for some features during segmentation, thus leading to low segmentation accuracy of the UNetMamba model for remote sensing images. SUMMARY

[0007] The embodiments of the present application provide a remote sensing image segmentation method and device for improving the segmentation accuracy of remote sensing images.

[0008] In one aspect, the embodiments of the present application provide a remote sensing image segmentation method, comprising the following steps:

[0009] obtaining a remote sensing image;

[0010] segmenting the remote sensing image using the SCFEMamba model to obtain a feature region contour map;

[0011] The SCFEMamba model is obtained by replacing two visual state space blocks stacked in the first three decoders of the UNetMamba model with a feature enhancement block and at least one improved decoding block connected in sequence, the feature enhancement block is used for feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder, and the improved decoding block includes:

[0012] A channel split layer is configured to split the input of the improved decoding block to obtain a first input and a second input.

[0013] A convolution branch is configured to perform convolution processing on the first input to obtain a convolution branch feature map.

[0014] A state branch is configured to perform visual state space processing on the second input to obtain a state branch feature map.

[0015] A pre-output module is configured to obtain the output of the improved decoding block according to the convolution branch feature map, the state branch feature map, and the input of the improved decoding block.

[0016] In another aspect, an embodiment of the present application provides a remote sensing image segmentation device, which includes:

[0017] An acquisition module is configured to acquire a remote sensing image.

[0018] A segmentation module is configured to perform segmentation processing on the remote sensing image by using a SCFEMamba model to obtain a contour map of a ground object region.

[0019] The SCFEMamba model is obtained by replacing two visual state space blocks stacked in the first three decoders of the UNetMamba model with a feature enhancement block and at least one improved decoding block connected in sequence, the feature enhancement block is used for feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder, and the improved decoding block includes:

[0020] A channel split layer is configured to split the input of the improved decoding block to obtain a first input and a second input.

[0021] A convolution branch is configured to perform convolution processing on the first input to obtain a convolution branch feature map.

[0022] A state branch is configured to perform visual state space processing on the second input to obtain a state branch feature map.

[0023] A pre-output module is configured to obtain the output of the improved decoding block according to the convolution branch feature map, the state branch feature map, and the input of the improved decoding block.

[0024] According to the remote sensing image segmentation method and device provided by the embodiment of the application, first, a remote sensing image is obtained; then, the remote sensing image is segmented by using an SCFEMamba model to obtain a contour map of a ground feature area; wherein the SCFEMamba model is obtained by replacing two visual state space blocks stacked in the first three decoders of a UNetMamba model with a feature enhancement block and at least one improved decoding block connected in sequence; in the improved decoding block, the input of the improved decoding block is divided into a first input and a second input, the first input is subjected to convolution processing to obtain a convolution branch feature map, the second input is subjected to visual state space processing to obtain a state branch feature map, and the output of the improved decoding block is obtained according to the convolution branch feature map, the state branch feature map and the input of the improved decoding block; the feature enhancement block is used for performing feature enhancement processing on the output of the last improved decoding block. The application effectively improves the segmentation accuracy of the remote sensing image.

[0025] Other features and advantages of the application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by means of the instrumentalities particularly pointed out in the description and claims, and the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a structural diagram of an existing UnetMamba model;

[0027] Figure 2 is a structural diagram of an SCFEMamba model provided by the application;

[0028] Figure 3 is a structural diagram of an SS-Conv-SSM block provided by the application;

[0029] Figure 4 is a principle diagram of an SS2D-CS mechanism provided by the application;

[0030] Figure 5 is a structural diagram of an E-FFN block provided by the application;

[0031] Figure 6 is a structural diagram of an FRH block provided by the application;

[0032] Figure 7 is a result visualization comparison diagram of the SCFEMamba model and other models provided by the application. DETAILED DESCRIPTION

[0033] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0034] The present application will be further described below in conjunction with the drawings and specific embodiments. The described embodiments should not be considered as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0035] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0037] In view of the limitations of segmentation models based on architectures such as CNN, Transformer, UnetMamba, etc. in the remote sensing image segmentation task, the embodiments of the present application propose a remote sensing image segmentation method and device, which improves the existing UNetMamba model to obtain the SCFEMamba model. The model introduces the Mamba architecture, combines the local feature extraction capability of CNN and the global context modeling capability of Transformer, and improves the image segmentation accuracy and efficiency of remote sensing images.

[0038] First, the structure of the existing Mamba model is introduced. The UNetMamba model is a UNet-like semantic segmentation model based on the Mamba architecture, and its overall architecture is similar to U-Net, aiming to efficiently process high-resolution remote sensing images. Referring to Figure 1 , Figure 1 In the above, "Input" refers to the remote sensing image to be segmented, "Output" refers to the outline map of the ground object region segmented from the remote sensing image, "ResT Stage1-4" respectively refers to the first encoder to the fourth encoder in the encoder structure, "H" refers to the height of the feature map, "W" refers to the width of the feature map, "Sum" refers to the summation operation (such as direct summation or weighted summation), "Principal Loss" and "Auxiliary Loss" respectively refer to different loss functions. The model consists of three components, which are:

[0039] A ResT Backbone is provided with four encoders (ResT Stage1-4) for encoding the input image in stages.

[0040] A Mamba Segmentation Decoder (MSD) can effectively decode complex semantic information in high-resolution images. Specifically, the Mamba architecture has demonstrated excellent capabilities in the field of computer vision, thanks to the VMamba model. In the field of semantic segmentation, the VMamba model is widely used thanks to its long sequence modeling capabilities. The MSD uses the VSS block in the VMamba model to construct a decoder structure for the UNetMamba model in 4 layers. Specifically, each layer structure of the MSD is implemented by an up-sampling module (Patch Expanding) to implement up-sampling and two stacked VSS blocks (VSS Block x 2) to implement semantic information extraction. The specific function of the up-sampling module is to compress the channel dimension by 2 while up-sampling the spatial resolution by 2. The VSS block is provided with a 2D selective scan (2D Selective Scan, SS2D) mechanism. By decoding multi-scale features through the MSD, the semantic segmentation result is finally output through a 1x1 convolution (1x1 Conv).

[0041] A Local Supervision Module (LSM) is provided with three LSM blocks (LSM Block), which correspond to the first three layers of the decoder, for significantly enhancing the perception of local information. Specifically, since the VSS block in the MSD can extract a larger receptive field in the feature map and is good at capturing long-distance dependencies, it ignores local information, which contains key factors related to the accuracy of the semantic segmentation task. Therefore, the introduction of LSM can well enhance the perception of local detail information. It should be noted that LSM only plays a role in model training and does not participate in real-time applications.

[0042] Secondly, the improvements made by the embodiments of the present application to the existing UNetMamba model are briefly introduced. Referring to Figure 2 The UNetMamba model is improved as follows in the embodiments of the present application:

[0043] (I) A new improved decoding block (SS-Conv-SSM block) is introduced in the MSD to replace the original two stacked VSS blocks, which integrates a convolution (Conv) architecture for extracting local features and a state space model (SSM) architecture for capturing global dependencies.

[0044] (II) A feature enhancement block (E-FFN block) is integrated at the output of the decoder to enhance the feature representation capability.

[0045] (III) The improved decoding block retains the SS2D mechanism and introduces an attention branch based on the existing SS2D mechanism, forming a scanning attention mechanism (SS2D-CS mechanism), which enhances the model's ability to capture key features and improves the model's robustness.

[0046] (IV) A fusion output module (FRH block) is introduced at the end of the UNetMamba model to replace the original 1x1 convolution layer, which combines with the spatial information in the shallow features to improve the segmentation accuracy of remote sensing images.

[0047] Based on the above, a remote sensing image segmentation method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0048] The remote sensing image segmentation method provided by the embodiments of the present application can be applied in a terminal, a server, or software running in a terminal or a server. The terminal can be a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc. basic cloud computing services. In addition, the server can also be a node server in a blockchain network, but is not limited thereto. The blockchain is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. computer technology.

[0049] Referring to Figure 2 and Figure 3 The remote sensing image segmentation method can include the following steps S101-S102:

[0050] S101, acquiring a remote sensing image;

[0051] S102, segment the remote sensing image by using the SCFEMamba model to obtain a contour map of the ground object region.

[0052] The SCFEMamba model is obtained by replacing two visual state space blocks stacked in the first three decoders of the UNetMamba model with a feature enhancement block and at least one sequentially connected improved decoding block, the feature enhancement block is used for feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder, and the improved decoding block includes:

[0053] The channel segmentation layer is configured to segment the input of the improved decoding block to obtain a first input and a second input.

[0054] The convolution branch is configured to perform convolution processing on the first input to obtain a convolution branch feature map.

[0055] The state branch is configured to perform visual state space processing on the second input to obtain a state branch feature map.

[0056] The pre-output module is configured to obtain the output of the improved decoding block according to the convolution branch feature map, the state branch feature map, and the input of the improved decoding block.

[0057] In the embodiment of the application, first, a remote sensing image to be segmented is obtained, which contains a ground object region, i.e., a region of interest in the remote sensing image segmentation task. Then, the remote sensing image is input into the SCFEMamba model for image segmentation processing. The SCFEMamba model is a neural network model improved on the basis of the existing UNetMamba model, and is trained by a plurality of preset remote sensing image samples and label information corresponding to each remote sensing image sample, wherein the label information corresponding to the remote sensing image sample refers to the ground object region contained in the remote sensing image sample. The contour map of the ground object region can be segmented from the target remote sensing image by the SCFEMamba model.

[0058] Specifically, as shown in Figure 1 In the existing UNetMamba model, the MSD is used as a decoder structure, which is provided with four decoders, each of which contains two stacked VSS blocks, i.e., "VSS Blockx2" as shown in Figure 1 Accordingly, as shown in Figure 2 Taking the input image as the top, the SCFEMamba model of the present application retains the up-sampling module in the first three decoders of the MSD, i.e., retains "Up Sample" as shown in Figure 1“Patch Expanding,” and replace the two VSS blocks stacked in the first three decoders of the MSD with a feature enhancement block and at least one sequentially connected improved decoding block, i.e., replace the “VSS Block” shown in FIG. 2A with the “Patch Expanding” shown in FIG. 2B. Figure 1 “VSS Block” shown in FIG. 2A with the “SS-Conv-SSM” and “E-FFN” shown in FIG. 2C, the improved decoding block is designed to perform convolution processing and visual state space processing on its input, and the feature enhancement block is used to further enhance the feature expression of the ground objects output by the last improved decoding block. Figure 2 “VSS Block” shown in FIG. 2A with the “SS-Conv-SSM” and “E-FFN” shown in FIG. 2C, the improved decoding block is designed to perform convolution processing and visual state space processing on its input, and the feature enhancement block is used to further enhance the feature expression of the ground objects output by the last improved decoding block.

[0059] It is worth noting that the number of improved decoding blocks in each decoder is not specifically limited in the embodiments of the present application, and can be flexibly set according to actual conditions. Exemplarily, it is verified through experiments that the performance of the SCFEMamba model is best when the number of improved decoding blocks in the first decoder and the third decoder is 1, and the number of improved decoding blocks in the second decoder is 2. It should be understood that if two improved decoding blocks are stacked in the decoder, the output of the first improved decoding block is taken as the input of the second improved decoding block, and the output of the second improved decoding block is taken as the input of the feature enhancement block.

[0060] In the SCFEMamba model, the data flow is as follows: four multi-scale features F ResT1 , F ResT2 , F ResT3 and F ResT4 are extracted through the ResT-based segmentation encoder structure, and then the four features are sequentially input into the MSD. In the MSD, the multi-scale feature F ResT4 is input into the first decoder. In the first decoder, the up-sampling module is first used for up-sampling processing, and then at least one sequentially connected improved decoding block is used for convolution processing and visual state space processing, and finally the feature enhancement block is used for feature refinement on the output of the last improved decoding block, to obtain the output F SCD4 of the first decoder. Then, the output F SCD4 of the first decoder is input into the second decoder after being spliced with the multi-scale feature F ResT3 in the channel, and the second decoder performs the same data flow as the first decoder. In this way, four outputs F SCD4 , F SCD3 , F SCD2 and F SCD1 of the MSD are obtained.

[0061] More specifically, grouped convolution uses a group of convolutions (multiple kernels per layer) to facilitate the neural network model to learn various high-level features and various low-level features, which was first proposed in AlexNet. In recent years, some researches have found that this module can be used to improve the performance of the neural network model while reducing the parameter quantity and complexity of the neural network model. In order to make the SCFEMamba model more effectively model the ground object region in the remote sensing image, under the inspiration of grouped convolution, the improved decoding block is introduced as the main component of the MSD of the SCFEMamba model. As shown in Figure 3 , the improved decoding block is a lightweight dual-branch structure, which includes a channel splitting layer, a convolution branch (Conv-Branch), a state branch (SSM-Branch), and a pre-output module, and the data flow is as follows:

[0062] Firstly, the operation of channel splitting (Split) is performed in the channel splitting layer, that is, the input X of the improved decoding block is split into two groups of feature maps in the channel dimension, which are the first input H×W×C and the second input , respectively. Here, C is the number of channels, the first input X1 is taken as the input of the convolution branch, and the second input X2 is taken as the input of the visual state space path.

[0063] Then, in the convolution branch, the first input X1 is processed by convolution to obtain the convolution branch feature map Here, the convolution branch can effectively capture the local features of the ground object region and enhance the extraction ability of the SCFEMamba model for local ground object features. At the same time, in the state branch, the second input X2 is processed by visual state space to obtain the state branch feature map Here, the state branch can effectively capture the long-distance global dependency relationship of the ground object features and enhance the global modeling ability of the SCFEMamba model. In this way, the global information and local information associated with the ground object region in each group of features can be extracted respectively.

[0064] Finally, the pre-output module is used to integrate the convolution branch feature map the state branch feature map and the input X of the improved decoding block, aiming to integrate the three into the output X' of the improved decoding block. Here, the input X of the improved decoding block represents the original ground object features at the shallow layer, the convolution branch feature map and the state branch feature map The deep global feature and the local feature are represented, the shallow feature information and the deep feature information are effectively integrated through the pre-output operation, more effective information associated with the feature region is comprehensively captured, the information loss between channels caused by the grouped convolution is avoided, the convergence of the SCFEMamba model is accelerated by using the residual connection, the comprehensiveness of the feature is improved, and the calculation complexity of the SCFEMamba model is reduced.

[0065] After obtaining the output of the last improved decoding block, the output of the last improved decoding block is processed by the feature enhancement block to obtain the output of the decoder In this way, more rich and representative feature representations of ground objects can be generated, so as to improve the performance of the SCFEMamba model in the remote sensing image segmentation task.

[0066] Therefore, the SCFEMamba model is obtained by improving the existing UNetMamba model, and the specific improvements include replacing the two VSS blocks stacked in the first three decoders of the MSD with the improved decoding block and the feature enhancement block. In the improved decoding block, the input thereof is divided into the inputs of the convolution branch and the state branch by the grouped convolution operation of the channel segmentation layer, and parallel feature processing is performed, the convolution branch can extract rich ground object detail information in the remote sensing image, such as local texture, and enhance the expression of the ground object feature, and the state branch can extract global ground object features in the remote sensing image and capture long-distance dependencies, so as to fully exert the high efficiency of the convolutional neural network in local feature extraction and fully utilize the strong ability of the visual state space model in global modeling. Then, the outputs of the two branches are pre-output with the initial input of the improved decoding block by the pre-output module, so as to avoid the information loss between channels caused by the grouped convolution, accelerate the convergence of the SCFEMamba model by using the residual connection, improve the comprehensiveness of the feature, and reduce the calculation complexity of the SCFEMamba model. On the basis of the above, in order to generate more rich and representative feature representations of ground objects, the feature enhancement block is introduced to perform feature enhancement on the output of the last improved decoding block to generate more rich and representative feature representations of ground objects and enhance the expression of the ground object feature.

[0067] In summary, on the one hand, the improved decoding block introduced in the embodiments of the present application enhances the capturing ability of the SCFEMamba model for global features and long-distance dependencies while enhancing the perception ability of the SCFEMamba model for local feature characteristics, which helps the SCFEMamba model to capture the semantic features of complex features such as small target features in remote sensing images, and enhances the comprehensiveness of semantic information; on the other hand, the feature enhancement block introduced in the embodiments of the present application enhances the extraction and channel fusion of the SCFEMamba model for the expression of features in space, which reduces the parameter amount and calculation amount of the SCFEMamba model while enhancing the expression of feature characteristics. In this way, the embodiments of the present application can effectively improve the performance of the SCFEMamba model in feature extraction and fusion of diversified features in complex scenes, enhance its perception ability for detailed features and key features of feature regions, especially small target features, thereby effectively improving the segmentation accuracy of remote sensing images.

[0068] The above will be described below in combination with Figure 3 The improved decoding block is further described.

[0069] In some embodiments, the convolution processing of the first input to obtain the convolution branch feature map can include:

[0070] The batch normalization processing is performed on the first input to obtain the batch-normalized first input; the ordinary convolution operation is performed on the batch-normalized first input to obtain the first convolution feature map; and the point convolution operation is performed on the first convolution feature map to obtain the convolution branch feature map.

[0071] In the convolution branch, first, the batch normalization processing (BN) is performed on the first input X1 to obtain the batch-normalized first input X1', which aims to ensure the stability of the input distribution. Then, the ordinary convolution operation is performed on the batch-normalized first input X1', which aims to extract local feature characteristics, specifically, the first ordinary convolution operation is performed on the batch-normalized first input X1' to obtain the first ordinary convolution first input x'1, and then the second ordinary convolution operation is performed on the first convolution first input x'1 to obtain the second ordinary convolution first input x'2, which is the first convolution feature map. The ordinary convolution operation is sequentially provided with convolution (Conv), batch normalization (BN), and activation, and the convolution kernel size and the activation function can be flexibly set according to actual conditions, for example, the convolution can be a 3x3 convolution kernel, and the activation function can be a ReLU activation function, but is not limited thereto. Finally, the point-by-point convolution operation is performed on the first convolution feature map x'2 to obtain the convolution branch feature map The point-wise convolution operation is sequentially provided with a point-wise convolution (PWConv) and an activation, and the activation function can be flexibly set according to actual conditions, for example, the activation function can be a ReLU activation function, but is not limited thereto.

[0072] For ease of understanding, assuming that the activation functions of the convolution branches are all ReLU activation functions, the data flow of the convolution branch can be expressed as formula (1) as follows:

[0073] x'1 = ReLU (BN (Conv (BN (X1))), x'2 = ReLU (BN (Conv (x'1))),

[0074]

[0075] It can be seen that the embodiment introduces a convolution branch in the improved decoding block, and through two consecutive ordinary convolution operations, rich ground feature detail information in a remote sensing image, such as local texture, can be effectively extracted, and through a point-wise convolution operation, channel information can be further integrated to enhance the expression of ground feature, thereby enhancing the perception ability of the SCFEMamba model to local ground features, which helps the SCFEMamba model to semantic features of complex ground features such as small target ground features in a remote sensing image, and enhances the comprehensiveness of semantic information.

[0076] In some embodiments, the above-mentioned visual state space processing of the second input to obtain the state branch feature map can include:

[0077] The second input is segmented to obtain a third input and a fourth input; the third input is subjected to a linear embedding operation and an activation operation to obtain a side branch feature map; the fourth input is subjected to a depth separable convolution operation and a scanning attention operation to obtain a main branch feature map; and the state branch feature map is obtained according to the side branch feature map and the main branch feature map.

[0078] In the state branch, first, the second input is segmented, specifically, the second input X2 is subjected to layer normalization (LN) to improve the robustness of the SCFEMamba model to input changes and adapt to lightweight design, and then a channel splitting (Split) operation is performed, that is, the second input after layer normalization is split into a third input x1 and a fourth input x2, the third input x1 is used as the input of the side branch, and the fourth input x2 is used as the input of the main branch.

[0079] Then, in the side branch, the third input x1 is first subjected to a linear embedding operation (Linear) and then subjected to an activation operation to obtain a side branch feature map x"1, aiming to complete feature mapping through linear transformation and break the linear restriction through the activation operation, so that the SCF EMamba model has the ability to fit complex patterns. Among them, the activation function can be flexibly set according to the actual situation, for example, the activation function can be a SiLU activation function, but is not limited thereto.

[0080] Meanwhile, in the main branch, the fourth input x2 is first subjected to a linear embedding operation (Linear) aiming to realize feature mapping, then subjected to a depth separable convolution operation after linear embedding, and finally subjected to a scanning attention operation after depth separable convolution to obtain a main branch feature map x"2. In this way, global ground feature characteristics and long-distance dependency relationships can be effectively captured. Among them, the depth separable convolution operation is sequentially provided with a depth separable convolution (DwConv) and an activation, and the convolution kernel size and the activation function can be flexibly set according to the actual situation, for example, the depth separable convolution is a 3x3 convolution kernel, and the activation function can be a SiLU activation function, but is not limited thereto. In addition, the scanning attention operation is sequentially provided with an SS2D operation (SS2D) based on an attention mechanism and a layer normalization (LN).

[0081] Finally, the side branch feature map x"1 output by the side branch and the main branch feature map x"2 output by the main branch are multiplied element by element, and the multiplication result is subjected to linear mapping (Linear) to obtain a state branch feature map Aiming to interact and linearly adjust the processing results of the two branches, to enhance the ability of the SCFEMamba model to screen, fuse and express key features of ground object regions.

[0082] For ease of understanding, assuming that the activation function in the state branch is a SiLU activation function and the convolution kernel of the depth separable convolution is 3x3, the data flow of the state branch can be represented as the following formula (2):

[0083] x1, x2 = Split (LN (X2)), x"1 = SiLU (Linear (x1)),

[0084] x"2 = LN (SS2D (SiLU (DWConv (Linear (x2)))),

[0085] Therefore, the embodiment introduces a state branch in the improved decoding block, in which a series of processes such as linear embedding, depth separable convolution, scanning attention operation and activation are performed to capture global object features and long-distance dependencies, thereby enhancing the capturing ability of the SCFEMamba model for global object features and long-distance dependencies, helping the SCFEMamba model to capture the semantic features of complex objects such as small target objects in remote sensing images, and enhancing the comprehensiveness of semantic information.

[0086] In some embodiments, with reference to Figure 4 The scanning attention operation can include:

[0087] The input of the scanning attention operation is subjected to a two-dimensional selective scanning operation to obtain a scanning feature map, and the input of the scanning attention operation is subjected to an attention operation to obtain an attention weight matrix, and the output of the scanning attention operation is obtained according to the scanning feature map and the attention weight matrix.

[0088] In the embodiment, the original SS2D mechanism comes from the VMamba, aiming to solve the contradiction between the non-sequential nature of visual data and the sequential nature in natural language processing tasks. The SS2D mechanism traverses the image through four scanning paths (from the four corners of the image to the relative position), ensuring that each image block only obtains context information through the compressed hidden state calculated along the scanning path, thereby reducing the computational complexity from quadratic to linear. Considering that the existing UNetMamba model lacks attention to important object feature information in the channel dimension and the spatial dimension, the embodiment introduces a scanning attention operation in the state branch of the improved decoding block, which is an SS2D operation based on an attention mechanism, referred to as an SS2D-CS mechanism.

[0089] Specifically, the output of the depth separable convolution operation is input to the scanning attention operation, the input of the scanning attention operation is subjected to a normal SS2D operation to obtain a scanning feature map, and the input of the scanning attention operation is subjected to an attention operation to obtain an attention weight matrix, and then the scanning feature map and the attention weight matrix are multiplied element by element to fuse the information of the two branches, thereby introducing an attention mechanism for the original SS2D mechanism and enhancing the expression ability of object features. Compared with simple concatenation, element-by-element multiplication can more finely fuse object feature information and avoid object information mixing, so that the SCFEMamba model focuses more on key object features and improves the pertinence and accuracy of object feature expression.

[0090] More specifically, the SS2D operation is the prior art. In simple terms, first, a cross-scan operation is performed, that is, the input (Input Patches) of the SS2D module is scanned in four directions, i.e., up, down, left and right, each image block (Patch) in the input is flattened into a one-dimensional vector, and four initial feature sequences in different directions are obtained. This design enables each image block to obtain global context information from different directions, while avoiding the computational bottleneck problem caused by quadratic complexity in traditional self-attention mechanisms. Then, an independent S6 block is set for each direction, and the S6 block extracts features from the initial feature sequence corresponding to the S6 block to obtain the feature sequence of the S6 block, i.e., the feature sequence of the ground object in four directions. The S6 block retains feature information related to the ground object area through a selection mechanism and filters feature information unrelated to the ground object area. Finally, the feature sequences of the ground object in the four directions are recombined (Cross-merge) to obtain a complete, two-dimensional scan feature map (Output Patches), thereby restoring the spatial structure of the feature map.

[0091] In the branch of attention operation (Channel-Spatial Branch), the channel attention mechanism (Channel Attention, CA) and the spatial attention mechanism (Spatial Attention, SA) are introduced. The channel attention operation is first performed on the input of the attention module to obtain a channel attention feature map, and then the spatial attention operation is performed on the channel attention feature map to obtain an attention weight matrix.

[0092] Here, the channel attention mechanism dynamically calibrates the weights by quantifying the contribution of each channel, enabling the SCFEMamba model to adaptively assign high weights to key channels and low weights to secondary channels, thereby accurately learning the differentiated contributions of different channels. This design captures the dependency between channels and automatically adjusts the response of feature channels, significantly improving the feature representation capability of the SCFEMamba model. The process of the channel attention mechanism is as follows: first, the input is convoluted, and then the feature map is compressed through global average pooling operation to calculate the global average value of each channel. This average value reflects the global feature response of each channel. Next, the average values are processed through an activation function (such as ReLU) to generate the weight of each channel. Finally, the weights are multiplied with the original features to obtain the channel attention feature map, thereby completing the dynamic weighting of the feature channels. This process not only helps the SCFEMamba model to better understand the data, but also highlights important features, thereby improving the overall performance and generalization ability of the SCFEMamba model.

[0093] The core idea of the spatial attention mechanism is to let the SCFEMamba model learn which part of the spatial region in the feature map is important to the current task, thereby improving the performance and generalization ability of the SCFEMamba model. The process of the spatial attention mechanism is as follows: first, perform global pooling operation on the input in the channel dimension, and the pooling method is usually maximum pooling and average pooling, to obtain two feature maps with a channel number of 1. Second, the two channel number 1 feature maps after pooling are spliced in the channel dimension to fuse the advantages of the two pooling methods, and then the spliced feature map is converted to a 1-channel feature map through a convolution layer. Finally, the feature map is sent to the activation function to be normalized to the range [0, 1] to obtain the weight matrix of the final spatial attention mechanism, that is, the attention weight matrix.

[0094] It is worth noting that the spatial attention mechanism is executed after the channel attention mechanism, which is derived from the logical order of feature processing. The channel attention mechanism can first filter out key feature information in the channel dimension, providing a more high-quality input for the spatial attention mechanism, so that the spatial attention mechanism can more accurately locate the important spatial region. The application of this hierarchical attention mechanism can greatly enhance the SCFEMamba model's understanding of remote sensing image vision and improve its performance in remote sensing image segmentation tasks.

[0095] As can be seen, the scanning attention operation provided by the embodiment introduces a channel attention mechanism and a spatial attention mechanism on the basis of the SS2D mechanism, and the results of the attention operation and the SS2D operation are multiplied element by element and fused into the final output. This way introduces an attention mechanism for the original SS2D mechanism, significantly enhancing the expression ability of the feature. The former realizes efficient context information extraction through four-way scanning and state space model, and the latter enhances the extraction and fusion ability of the feature through the attention mechanism. This combination not only improves the computational efficiency and robustness of the SCFEMamba model, but also further enhances the model's ability to capture key features, improving the original UNetMamba model's attention to important information in the channel and spatial dimensions, thereby improving its performance in remote sensing image segmentation tasks.

[0096] In some embodiments, the above obtaining the output of the improved decoding block according to the convolution branch feature map, the state branch feature map, and the input of the improved decoding block can include:

[0097] The convolution branch feature map and the state branch feature map are fused to obtain a spliced feature map; the spliced feature map is subjected to a channel shuffling operation to obtain a shuffled spliced feature map; and the shuffled spliced feature map and the input of the improved decoding block are fused to obtain the output of the improved decoding block.

[0098] In the pre-output module, first, the convolution branch feature map and the state branch feature map are spliced in the channel dimension to obtain a spliced feature map, aiming to preliminarily interact and fuse the outputs of the convolution branch and the state branch. Then, the spliced feature map is subjected to a channel shuffle operation to obtain a shuffled spliced feature map. The channel shuffle operation promotes the flow of information between different groups of channels by rearranging the channel order, thereby improving the feature expression capability of the SCFEMamba model without significantly increasing the amount of calculation, and solving the problem of information isolation between channels caused by grouped convolution. Finally, the shuffled spliced feature map and the input X of the improved decoding block are added in residual connection to obtain the output X' of the improved decoding block. The input X of the improved decoding block represents the original feature information of the ground object, and the shuffled spliced feature map represents the feature information of the ground object after deep feature extraction. By fusing the two, the shallow ground feature information and the deep ground feature information can be effectively integrated, and more effective information associated with the ground object region can be captured comprehensively.

[0099] As can be seen, the pre-output module is introduced in the improved decoding block in the embodiment, which can not only avoid the loss of information between channels caused by grouped convolution, but also accelerate the convergence of the SCFEMamba model by using residual connection, improve the comprehensiveness of the ground feature, and reduce the computational complexity of the SCFEMamba model, thereby improving the performance of the SCFEMamba model in the task of remote sensing image segmentation.

[0100] The feature enhancement block will be further described below. Figure 2 and Figure 5 .

[0101] In some embodiments, referring to Figure 5 , the feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder can include:

[0102] channel adjustment processing on the output of the last improved decoding block to obtain an adjusted feature map; parallel depth separable convolution operations on the adjusted feature map to obtain a plurality of second convolution feature maps; and obtaining the output of the decoder according to the plurality of second convolution feature maps.

[0103] In the embodiment, the purpose of the feature enhancement block is to fuse ground features of different sizes to generate more rich and representative ground feature representations, thereby improving the performance of the SCFEMamba model in the task of remote sensing image segmentation. Specifically, the output of the last improved decoding block is represented as a feature map F Figure 5 in the embodiment. uFirst, a 1x1 convolutional layer is used to compress the channel dimension of the feature map F u to obtain an adjusted feature map F u ', which helps to reduce the computational complexity and parameter quantity of the SCFEMamba model while retaining important feature information of ground objects. Then, two parallel depthwise separable convolutions (DW-Conv) are performed on the adjusted feature map F u ' to obtain two second convolutional feature maps. The depthwise separable convolution kernel can be flexibly set according to actual conditions, for example, it can be a 3x3 depthwise separable convolution and a 5x5 depthwise separable convolution, but is not limited thereto. Here, the depthwise separable convolution is used to extract ground object features of different scales, and using the depthwise separable convolution instead of the ordinary convolution can help the SCFEMamba model to effectively reduce the computational complexity while maintaining good feature extraction capability. Finally, the two second convolutional feature maps are added to obtain an added feature F u ", and a 1x1 convolutional layer is used to adjust the channels of the added feature F u " to the number of channels of the original input to obtain the output of the decoder

[0104] For ease of understanding, the data flow of the above feature enhancement block can be represented as formula (3) as follows:

[0105] F u ' = Conv 1×1 (F u ), F u " = DWConv(F u ') + DWConv(F u '),

[0106]

[0107] As can be seen, the feature enhancement block is introduced into the SCFEMamba model in the embodiment, and compared with the traditional feature fusion method, the feature enhancement block further optimizes the architecture of the SCFEMamba model through depthwise separable convolution, 1x1 convolution and feature fusion operation, so that the SCFEMamba model can effectively capture ground object features of different scales, generate more comprehensive and robust ground object feature representations, and further enhance the representation capability of the ground object features. In this way, the SCFEMamba model's ability to capture detailed information and global information of ground objects, as well as its ability to extract spatial features and fuse channels, is effectively enhanced, and the computational complexity and parameter quantity are compressed, thereby significantly improving the performance of the SCFEMamba model in the remote sensing image segmentation task, especially the inference efficiency and inference accuracy.

[0108] The following will be combined with Figure 2 andFigure 6 Other improvements of the SCFEMamba model are introduced.

[0109] In some embodiments, referring to Figure 6 The SCFEMamba model is obtained by replacing the last layer of the UNetMamba model with a feature refinement block, and the feature refinement block comprises:

[0110] a refinement input module configured to obtain a fifth input according to the output of the first encoder and the output of the last decoder;

[0111] a channel path configured to perform feature refinement processing on the fifth input in the channel dimension to obtain a channel-refined feature map;

[0112] a spatial path configured to perform feature refinement processing on the fifth input in the spatial dimension to obtain a spatial-refined feature map;

[0113] a refinement output module configured to obtain the contour map of the feature region according to the channel-refined feature map, the spatial-refined feature map, and the fifth input.

[0114] In the present embodiment, in order to enhance the refinement degree of feature mapping, in addition to introducing the improved decoding block and the feature enhancement block into the original UNetMamba model, the present embodiment also introduces a feature refinement block into the UNetMamba model. Specifically, as shown in Figure 1 , the last layer of the original UNetMamba model is a 1x1 convolution layer, which is a "1x1Conv" as shown in Figure 1 , which is used for dimension reduction and outputs the final segmentation result. Accordingly, the present embodiment replaces the 1x1 convolution layer with a feature refinement block, i.e., replaces the "1x1Conv" as shown in Figure 1 with "FRH" as shown in Figure 2 , and the feature refinement block aims to optimize features by bridging spatial details and semantic information.

[0115] More specifically, the feature refinement block comprises a refinement input module, a channel path, a spatial path, and a refinement output module, and the data flow is as follows:

[0116] In the refinement input module, the output F ResT1 of the first encoder (i.e., the feature of the feature region in the shallow layer) and the output F SCD1The fifth input is input into the channel path and the spatial path respectively.

[0117] In the channel path, the fifth input is processed in the channel dimension to obtain a channel refined feature map, aiming to adjust the importance of each channel in the fifth input, filter key channel information, enhance the semantic expression ability of the feature of ground objects, reduce noise and redundant feature information of ground objects, and thus enhance the perception ability of the SCFEMamba model to key features of ground objects.

[0118] The channel refined feature map, the spatial refined feature map and the fifth input are input into the refined output module. In the refined output module, first, the channel refined feature map and the spatial refined feature map are fused into a target refined feature map through summation operation (Sum), aiming to preliminarily interact the ground object features refined in the channel dimension with the ground object features refined in the spatial dimension, and integrate the important information of the ground object region in the spatial dimension and the channel dimension.

[0119] Then, the target refined feature map is sequentially subjected to depth-wise convolution (Depth-wise Conv), batch normalization (Batch Norm) and channel adjustment (Conv 1x1) to obtain an adjusted target refined feature map, aiming to further refine and reduce the dimension of the feature of ground objects. The channel adjustment uses a 1x1 convolution layer, and the size of the convolution kernel of the depth-wise convolution can be flexibly set according to actual conditions, for example, the convolution kernel of the depth-wise convolution can be 3x3, but is not limited thereto.

[0120] After that, the fifth input is connected to the adjusted target refined feature map through residual connection (Residual Connection), and the two are further fused through summation operation (Sum) to obtain a fused feature map, which not only can alleviate the problem of gradient disappearance and prevent the SCFEMamba model from degrading, but also can improve the stability of the SCFEMamba model, so that it can more stably learn and express complex semantic features of ground objects.

[0121] Finally, the fused feature map is sequentially activated, channel adjusted (Conv 1x1), and upsampled (Upsample) to obtain the contour map (Output map) of the feature region. In this way, the output dimension can be reduced through the final processing, and the segmentation accuracy of the remote sensing image and the stability of the SCFEMamba model can be ensured. Among them, the activation function and the number of upsampling can be flexibly set according to actual conditions, for example, the activation function can be a ReLU activation function, and the number of upsampling can be four times, but it is not limited thereto.

[0122] Therefore, compared with the existing UNetMamba model, the feature refinement block has a significant advantage in balancing the semantic and spatial information when fusing feature regions. Through the parallel spatial path and channel path, the feature refinement block can respectively strengthen the expression of feature regions from the spatial dimension and the channel dimension. The former accurately locates important feature regions, and the latter filters the key channel information of the feature regions. The two can make up for the shortcomings of single-dimensional processing and comprehensively improve the quality of feature regions. In subsequent processing, through the first summation operation and the depth separable convolution, batch normalization, and channel adjustment after that, the features of the two paths can be effectively fused and refined, thereby integrating the important information of feature regions in the spatial dimension and the channel dimension. Through the second summation operation and the activation, channel adjustment, and up-sampling after that, on the one hand, the gradient vanishing problem can be alleviated, the SCFEMamba model can be prevented from degrading, and the stability of the SCFEMamba model can be improved, so that the SCFEMamba model can more stably learn and express complex feature semantic features. On the other hand, the output dimension of the contour map can be reduced, and the segmentation accuracy of the remote sensing image and the stability of the SCFEMamba model can be ensured.

[0123] In this way, the shallow features generated by the first encoder contain rich spatial detail information but lack semantic representation ability, while the global-local fused features output by the last decoder have deep semantic expression ability but lose spatial resolution. Although directly fusing the two kinds of features has computational advantages, it may lead to spatial-semantic information imbalance. To this end, the feature refinement block can effectively compensate for the semantic gap between shallow feature regions and deep feature regions through the adaptive feature calibration mechanism, so as to significantly improve the segmentation accuracy of the SCFEMamba model while maintaining computational efficiency.

[0124] In some embodiments, the feature refinement processing on the fifth input in the channel dimension to obtain the channel refined feature map can include:

[0125] The fifth input is subjected to a pooling operation to obtain a channel attention map; the channel attention map is subjected to channel adjustment processing to obtain an adjusted channel attention map; the adjusted channel attention map is subjected to an activation operation to obtain a channel attention weight; and a channel refinement feature map is obtained according to the fifth input and the activated channel attention map.

[0126] In the channel path, first, the fifth input is subjected to a global average pooling operation (Global Average Pool) to generate a channel attention map C' ∈ R 1×1×C Then, the channel attention map C' is subjected to channel adjustment processing to obtain an adjusted channel attention map, which specifically includes a dimension reduction operation (Reduce) to reduce the number of channels by four times to extract key feature information of ground objects by compressing the number of channels, and an expansion operation (Expand) to restore the number of channels to C to enhance the expression of feature information of ground objects by restoring the number of channels. Subsequently, the adjusted channel attention map is subjected to an activation operation to obtain an activated channel attention map to introduce nonlinear changes, so that the SCFEMamba model can learn complex feature patterns of ground objects. The activation function can be flexibly set according to actual conditions, for example, the activation function can be a Sigmoid function, but is not limited thereto. Finally, the fifth input and the activated channel attention map are subjected to matrix multiplication (Matrix Multiplication) to obtain a channel refinement feature map, which can dynamically adjust the feature information of ground objects in the fifth input according to the channel attention weight, and enhance the sensitivity of the SCFEMamba model to key feature information of ground objects.

[0127] As can be seen, the present embodiment introduces a channel path in the feature refinement block, which dynamically adjusts the importance of each channel by assigning different weights to each channel, filters important channel information, thereby enhancing the semantic expression ability of feature information of ground objects, reducing noise and redundant information, thereby improving the perception ability of the SCFEMamba model to key feature information of ground objects, and helping to improve its performance in the task of remote sensing image segmentation.

[0128] In some embodiments, the above-mentioned feature refinement processing of the fifth input in the spatial dimension to obtain a spatial refinement feature map can include:

[0129] The fifth input is subjected to a depth separable convolution operation to obtain a spatial attention map; the spatial attention map is subjected to an activation operation to obtain an activated spatial attention map; and a spatial refinement feature map is obtained according to the fifth input and the activated spatial attention map.

[0130] In the present embodiment, first, the fifth input is subjected to a depth separable convolution (Depth-wise Conv) to generate a spatial attention map S ∈ RH×W×1 Then, the spatial attention map S is subjected to an activation operation to obtain an activated spatial attention map, so as to introduce a nonlinear change, so that the SCFEMamba model can learn complex feature patterns of ground objects. The activation function can be flexibly set according to actual conditions, for example, the activation function can be a Sigmoid function, but is not limited thereto. Finally, the fifth input and the activated spatial attention map are subjected to matrix multiplication to obtain a spatial refined feature map, so that the ground object features in the fifth input can be dynamically adjusted according to the spatial attention weight, and the sensitivity of the SCFEMamba model to key ground object information is enhanced.

[0131] Therefore, the embodiment introduces a spatial path in the feature refinement block, which can locate important ground object regions, effectively suppress the interference of background and noise on remote sensing image segmentation, and further improve the performance of the SCFEMamba model in the remote sensing image segmentation task.

[0132] In some embodiments, the above-mentioned SCFEMamba model is also obtained by deleting two visual state space blocks stacked in the last decoder of the UNetMamba model.

[0133] In the embodiment, in addition to introducing the feature refinement block, the feature enhancement block and the improved decoding block in the above-mentioned SCFEMamba model, two visual state space blocks stacked in the last decoder of the UNetMamba model are also deleted, that is, as shown in Figure 1 the input image is taken as the top, and the fourth decoder to the first decoder are sequentially arranged from top to bottom, and the "VSS Blockx2" in the fourth decoder is deleted to realize lightweight deployment.

[0134] The performance of the SCFEMamba model (i.e., the SCFEMamba model) provided by the embodiments of the present application will be verified below.

[0135] (I) Dataset: The public datasets of LoveDA land use and cover dataset, ISPRSVaihingen dataset and ISPRS Potsdam dataset are selected in this embodiment, and the specific information of the three datasets is shown in Table 1.

[0136] Table 1: Details of the dataset

[0137]

[0138] (B) Experimental setup: All models in this example experiment are implemented using the PyTorch framework on 4 NVIDIA RTX 3060 GPUs. For the LoveDA, Vaihinge, and Potsdam datasets, the images are randomly cropped to a size of 512x512. During training, data augmentation techniques such as random scaling (specifically [0.5, 0.75, 1.0, 1.25, 1.5]), random vertical flipping, random horizontal flipping, and random rotation are used. The number of training iterations is set to 100, and the batch size is 6. In the testing phase, multi-scale and random flip augmentation are used. To achieve fast convergence, the AdamW optimizer is deployed to train all models in the experiment. The base learning rate is set to 6e-4, and the cosine learning rate adjustment strategy is used.

[0139] The loss function used in this example consists of a cross-entropy loss function for pixel-level classification and a Dice loss function that focuses on the boundary region. The goal of the cross-entropy loss function is to minimize the difference between the one-hot encoded class labels and the negative log-likelihood of the predicted probabilities. The cross-entropy loss function is defined as shown in the following equation (4):

[0140] (4);

[0141] In equation (4), Lceis the cross-entropy loss function, N and C represent the number of samples and the number of classes, respectively, yikand pkikrepresent the label of the i-th class of the k-th data sample and the confidence of the model predicting the i-th class of the k-th sample, respectively.

[0142] The Dice loss improves segmentation accuracy by optimizing the overlap between predicted regions and true regions, and is very suitable for unbalanced datasets. The Dice loss function is defined as shown in the following equation (5):

[0143]

[0144] In equation (5), L dice is the Dice loss function.

[0145] In addition, this example also designs a cross-entropy auxiliary loss function in the loss function to learn semantic features from the auxiliary head. The auxiliary head extracts features from each decoder, performs upsampling and summation operations to obtain an intermediate semantic segmentation map, and the definition of the auxiliary loss function is shown in the following equation (6):

[0146] ​​​​​

[0147] In formula (6), L aux represents an auxiliary loss function, represents a semantic segmentation map from the intermediate decoder, and the overall loss function is a weighted sum of the above loss functions, and the specific formula is shown in the following formula (7):

[0148] L total = L ce + L dice + aL aux (7).

[0149] In formula (7), L total is the overall loss function; a is the weight of the auxiliary loss function, for example, it is set to 0.4, but is not limited thereto.

[0150] (Three) Performance indicators: select overall accuracy (OA), F1 score and mean intersection over union (mIoU) as indicators for evaluating the performance of the model.

[0151] (Four) Comparative experiment: in order to verify the effectiveness of the SCFEMamba model proposed in the embodiment, the embodiment makes 8 groups of comparative experiments, wherein the comparative models include the mainstream semantic segmentation method at present, the baseline model of the embodiment and the improved model of the embodiment, which are respectively UNet model, ABCNet model based on CNN architecture; BANet model, UNetFormer model and DCSWin model based on Transformer architecture; PyramidMamba model based on Mamba architecture; plus the baseline model UNetMamba model of the embodiment and the improved model SCFEMamba model, a total of 8 groups of comparative experiments. The six kinds of comparative experimental models are respectively tested on the three public data sets mentioned above, and compared with the UNetMamba model and the improved model SCFEMamba model of the embodiment, as shown in Tables 2 to 4.

[0152] Table 2: Performance of all models on LoveDA dataset

[0153]

[0154] From Table 2, it can be seen that the SCFEMamba model performs very well on the LoveDA dataset, especially in the mIoU indicator, reaching 54.14%, which is significantly higher than other models. Compared with the baseline model UNetMamba, it improves by 2.13%, which is 5.05% higher than the best model ABCNet in the CNN-based model, and 3.47% higher than the best model BANet in the Transformer-based model. This shows that SCFEMamba has a significant advantage in semantic segmentation accuracy, and can more accurately identify and segment different semantic regions in remote sensing images. In addition, the F1 score and OA of SCFEMamba are also the highest among all models, which are 2.1% and 2.35% higher than the baseline model, respectively. This shows that it has good recall ability in high-resolution remote sensing image segmentation tasks, and can effectively identify more correct segmentation results, and the overall classification accuracy of the model in all classes is better. In summary, SCFEMamba model has the highest score in three key indicators, which shows that the model has a significant advantage in remote sensing image semantic segmentation tasks, and its superiority is due to its unique network structure based on Mamba, which can better capture and utilize multi-scale features and context information in remote sensing images.

[0155] Table 3: Performance of all models on the Vaihingen dataset

[0156]

[0157] From Table 3, it can be seen that on the Vaihingen dataset, SCFEMamba performs well, especially in the mIoU and indicators, reaching 77.14%, which is higher than all other models, and 0.32% higher than the second place baseline model. The improvement in mIoU indicates that the segmentation results of SCFEMamba model are closer to the true labels, thereby indicating the high accuracy of the model in semantic segmentation tasks. SCFEMamba also achieves the highest score in the F1 score table, which is higher than the baseline model UNetMamba, which further proves its superiority in semantic segmentation tasks. The overall accuracy of SCFEMamba is 87.91%, which is very close to the first place 87.98%, only 0.07% difference, still maintaining a high accuracy, although the advantage is not obvious in this indicator, but SCFEMamba still maintains a high overall level.

[0158] Table 4: Performance of all models on the Potsdam dataset

[0159]

[0160] From Table 4, it can be seen that on the Potsdam dataset, the F1 score of SCFEMamba model is the highest, reaching 87.42%, which is significantly higher than the 86.89% of the baseline model UNetMamba, indicating that SCFEMamba performs better in balancing precision and recall; the OA of SCFEMamba is 85.89%, which is slightly lower than the 86.79% of UNetMamba, which is caused by the replacement of the pure SSM architecture in the encoder with the SS-Conv-SSM architecture in the SCFEMamba model. Compared with the pure SSM architecture, the SS-Conv-SSM has slightly weakened global modeling ability due to the introduction of local perception ability, resulting in a decrease in OA, but the mIoU and F1 score of the model have improved, proving that the comprehensive performance of SCFEMamba model is stronger, and the overall performance is better than UNetMamba. In addition, compared with the models based on Transformer architecture, the F1 score of SCFEMamba is higher than all the Transformer models in the table, but the OA has decreased, which is because the global modeling ability of SCFEMamba is slightly worse than the models based on Transformer architecture, but SCFEMamba introduces SSM architecture, and its computational complexity is much lower than the Transformer architecture model, and the mIoU of SCFEMamba is higher than all the models, which shows that SCFEMamba is more accurate in target region recognition and boundary division in the segmentation task. Finally, compared with the models based on CNN architecture, the F1 score, OA and mIoU of SCFEMamba are all higher than the CNN architecture models. This is because SCFEMamba model has better global modeling ability than CNN model.

[0161] In summary, SCFEMamba model performs well in remote sensing image semantic segmentation task, especially in F1 score and mIoU indicators, significantly better than the baseline model UNetMamba and other comparative models. This shows that SCFEMamba has obvious advantages in balancing precision and recall, target region recognition and boundary division, etc., thus proving its potential in practical applications.

[0162] (Five) Ablation experiment:

[0163] To further verify the contribution of each module in the SCFEMamba model to the overall model, the following will conduct an ablation experiment on each module, which is SS-Conv-SSM, SS2D-CS, E-FFN and FRH. The ablation experiment is conducted on the LoveDA dataset, and a total of 8 ablation experiments are designed. Among them, the UNetMamba model is taken as the baseline of the ablation experiment. The ablation experiment results are shown in Table 5. The ablation experiment designs eight different configurations, which respectively correspond to a row of Table 5, and observes the changes of F1 score, OA and mIoU of the model in the semantic segmentation task on the LoveDA dataset by adding or deleting a specific module.

[0164] Table 5: Ablation experiment results of each module in the SCFEMamba model on the LoveDA data

[0165]

[0166] In Table 5: the baseline model is exp_1, that is, without adding any module, the model retains the basic structure, at this time the F1 score is 67.71%, the OA is 70.01%, and the mIoU is 52.01%, which provides a performance basis for subsequent experiments. exp_2 is a model experiment of adding a SS-Conv-SSM module alone on the basis of the baseline model, the F1 score is increased to 68.19%, the OA is slightly increased by 0.61%, and the mIoU is increased by 0.62%, which shows that the SS-Conv-SSM module has a positive contribution to the model performance. exp_3 is a model experiment of adding a FRH module alone to the baseline model, the F1 score is increased to 68.34%, the OA value is increased to 70.51%, and the mIoU value is increased to 52.73%. exp_4 is an experiment of adding a SS2D-CS module alone, and the F1 score, OA value and mIoU value are all increased, which are increased by 0.52%, 0.89% and 0.79% respectively. exp_5 is to add an E-FFN module alone on the basis of the baseline model, and the improvement of the F1 score of this module is the highest compared with other modules. From exp_5, it can be concluded that all the improved modules proposed in the embodiments of the present application improve the model performance, and then the module stacking experiment is carried out to verify the effectiveness of the module combination on the model performance. exp_6 adds a FRH module on the basis of exp_2, the F1 score is further increased to 68.72%, the OA is increased to 71.72%, and the mIoU is increased to 53.18%, which has a greater improvement compared with the baseline model. exp_7 introduces a SS2D-CS module on the basis of exp_6, and the F1 value is increased by 1.31% compared with the baseline model, at the same time, the OA value is increased by 1.9%, and the mIoU value is increased by 1.61%. exp_8 introduces all the modules into the baseline model to obtain the SCFEMamba model, compared with the baseline model, the F1 value, the OA value and the mIoU value are increased by 1.9%, 2.35% and 2.13% respectively. From the experimental results, it can be seen that each module in the SCFEMamba model has a positive contribution to the overall performance. This shows that these modules have an important role in processing the semantic segmentation task and can effectively improve the recognition accuracy and robustness of the model. In addition, the experiment also reveals the synergistic effect between the modules. For example, the combination effect of SS-Conv-SSM and FRH modules is better than using any one of the modules alone, and further adding a SS2D-CS module can further improve the performance.

[0167] (VI) Result visualization:

[0168] On the test set of the LoveDA dataset, the application embodiment uses representative models to visualize the semantic segmentation results, which are respectively the U-Net model based on the CNN architecture, the UNetFormer model based on the Transformer architecture and the UNetMamba model based on the Mamba architecture, and the SCFEMamba model proposed in the application embodiment. The semantic segmentation result visualization is as shown in Figure 7 Figure 7 In the figure, “Nirrg images” refers to remote sensing images, “Building” refers to buildings, “Road” refers to roads, “Water” refers to rivers, “Barren” refers to barren land, “Forest” refers to forests, “Agriculture” refers to farmland, “Background” refers to background, and “GT” refers to standard contour maps.

[0169] According to the classification of the LoveDA dataset, the semantic segmentation experiments are respectively performed on the two categories of rural and urban in the figure, wherein the first column is the original remote sensing image block, the second column is the corresponding real label, and the fourth to sixth columns are the corresponding model prediction results. The difference between the series of models in the segmentation of large-area objects such as water and roads is not large, and the model segmentation results are mainly reflected in small target objects and confused objects. Next, two typical segmentation areas are taken as an introduction to analyze the visualization results.

[0170] In the rural (Rural) series of segmentation results, the building prediction class is circled by the dashed box in the figure, in the U-Net model, the prediction result is fuzzy, the edge information is not clear, and there is a missing detection phenomenon; in the UNetFormer model, this phenomenon is improved, the edge information of the building detection is relatively clear, but there is still a missing detection phenomenon; in the UNetMamba model, the edge information of the building detection at the same position is not much different from the prediction result of the previous model; until the SCFEMamba model proposed in the application embodiment, in the same position building detection part, the edge information is clear and visible, and the missing detection part is greatly reduced. In the semantic segmentation result, the detection of the part where the empty land and the forest and the water area are connected is difficult, as shown in the circle in the figure, the SCFEMamba has the best segmentation result in this part, which restores the label map to the greatest extent, and compared with the baseline model UNetMamba, the segmentation result of the same position of the baseline model UNetMamba has a great missing detection problem.

[0171] ​In the Urban series segmentation result map, buildings are the main component of the city, and their area proportion is the largest, so for the semantic segmentation of the city, building segmentation accounts for the majority. As circled in the figure, in the UNet model, the segmentation effect of buildings is very poor, especially for the edge segmentation between buildings, which has almost no segmentation effect; in the UNetFormer and UNetMamba models, the segmentation result of this position is not much different from that of U-Net, and there is almost no improvement, until the SCFEMamba model proposed in the embodiments of the present application, which restores the label map to the greatest extent and very well separates each building in the building segmentation, with the highest segmentation accuracy.

[0172] In summary, the embodiments of the present application provide a remote sensing image segmentation method, which is implemented based on the SCFEMamba model, aiming to solve many challenges in high-resolution remote sensing image semantic segmentation. In view of the shortcomings of traditional methods, such as the difficulty of CNN architecture to extract global context information, the high computational complexity and large parameter quantity of Transformer architecture, and the defects of Mamba architecture in complex scene feature extraction and fusion, the embodiments of the present application make a series of improvements to the existing UnetMamba model, replace the VSS module in the decoder with the SS-Conv-SSM module, which combines the local feature extraction capability of the convolution branch and the global modeling capability of the SSM branch to improve the comprehensiveness of the features; the SS-Conv-SSM module retains the SS2D mechanism, and increases the branch combining channel attention and spatial attention in the SS2D mechanism to enhance the model's ability to capture key features; an E-FFN module is introduced at the output of each layer of the decoder to effectively fuse features of different scales and enhance the feature expression capability; an FRH module is introduced at the last layer of the model to fully fuse shallow spatial information and deep semantic information and improve the segmentation accuracy.

[0173] The experimental results of the SCFEMamba model on the public data sets LoveDA, ISPRS Vaihingen and ISPRSPotsdam show that the SCFEMamba model performs well in the semantic segmentation task. Compared with the mainstream comparative models, it significantly outperforms the baseline model UNetMamba and other models in F1 score and mIoU indicators. On the LoveDA data set, the mIoU of SCFEMamba reaches 54.14%, which is 2.13% higher than that of UNetMamba; on the Vaihingen data set, the mIoU reaches 77.14%, which is higher than that of all other models; on the Potsdam data set, the F1 score reaches 87.42%, which is also higher than that of the baseline model and other comparative models. This fully proves that SCFEMamba has obvious advantages in balancing precision and recall rate, target region recognition and boundary division, etc.

[0174] The ablation experiment further verifies that each module of the SS-Conv-SSM, the SS2D-CS, the E-FFN and the FRH positively contributes to the overall performance, and there is a synergistic effect between the modules, which collectively improves the semantic segmentation accuracy of the model. The visual analysis of the experimental results shows that the SCFEMamba performs excellently in the segmentation of small target objects and confused objects, can clearly restore the edge information of the objects, and reduces the missed detection phenomenon, and has higher segmentation accuracy than other models.

[0175] In addition, the embodiment of the present application also provides a remote sensing image segmentation device, which can include:

[0176] The acquisition module is configured to acquire the remote sensing image.

[0177] The segmentation module is configured to segment the remote sensing image by using the SCFEMamba model to obtain a contour map of the ground object region.

[0178] The SCFEMamba model is obtained by replacing two visual state space blocks stacked in the first three decoders of the UNetMamba model with a feature enhancement block and at least one sequentially connected improved decoding block, the feature enhancement block is configured to perform feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder, and the improved decoding block includes:

[0179] The channel segmentation layer is configured to segment the input of the improved decoding block to obtain a first input and a second input.

[0180] The convolution branch is configured to perform convolution processing on the first input to obtain a convolution branch feature map.

[0181] The state branch is configured to perform visual state space processing on the second input to obtain a state branch feature map.

[0182] The pre-output module is configured to obtain the output of the improved decoding block according to the convolution branch feature map, the state branch feature map and the input of the improved decoding block.

[0183] The contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0184] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and purposes of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0185] The above describes the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A remote sensing image segmentation method, characterized in that, Includes the following steps: Acquire remote sensing images; The remote sensing image is segmented using the SCFEMamba model to obtain the outline map of the ground feature area; The SCFEMamba model is obtained by replacing two stacked visual state space blocks in the first three decoders of the UNetMamba model with feature enhancement blocks and at least one sequentially connected improved decoding block. The feature enhancement block is used to perform feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder. The improved decoding block includes: A channel segmentation layer is used to segment the input of the improved decoding block to obtain a first input and a second input; The convolutional branch is used to perform convolution processing on the first input to obtain the convolutional branch feature map; The state branch is used to perform visual state space processing on the second input to obtain the state branch feature map; The pre-output module is used to obtain the output of the improved decoding block based on the convolutional branch feature map, the state branch feature map, and the input of the improved decoding block; The step of performing visual state space processing on the second input to obtain a state branch feature map includes: The second input is segmented to obtain the third and fourth inputs; Perform linear embedding and activation operations on the third input to obtain a side branch feature map; The fourth input is subjected to depthwise separable convolution and scanning attention operations to obtain the main branch feature map; The state branch feature map is obtained based on the side branch feature map and the main branch feature map; The scanning attention operation includes: A two-dimensional selective scanning operation is performed on the input of the scanning attention operation to obtain a scanning feature map; An attention operation is performed on the input of the scanning attention operation to obtain an attention weight matrix; The output of the scanning attention operation is obtained based on the scan feature map and the attention weight matrix; The step of performing feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder includes: The output of the last improved decoding block is subjected to channel adjustment processing to obtain an adjusted feature map; Parallel depthwise separable convolution operations are performed on the adjusted feature maps to obtain multiple second convolutional feature maps; The decoder output is obtained based on multiple second convolutional feature maps; wherein, the SCFEMamba model is further obtained by replacing the last layer of the UNetMamba model with a feature refinement block, the feature refinement block comprising: A refined input module is used to obtain the fifth input based on the output of the first encoder and the output of the last decoder; The channel path is used to refine the features of the fifth input in the channel dimension to obtain a channel-refined feature map. The spatial path is used to refine the features of the fifth input in the spatial dimension to obtain a spatially refined feature map. The refinement output module is used to obtain the outline map of the land feature area based on the channel refinement feature map, the spatial refinement feature map, and the fifth input.

2. The method according to claim 1, characterized in that, The step of performing convolution processing on the first input to obtain a convolutional branch feature map includes: The first input is batch normalized to obtain the batch normalized first input; Perform a regular convolution operation on the first input after batch normalization to obtain the first convolution feature map; Perform pointwise convolution on the first convolutional feature map to obtain the convolutional branch feature map.

3. The method according to claim 1, characterized in that, The step of refining the features of the fifth input along the channel dimension to obtain a channel-refined feature map includes: The fifth input is pooled to obtain a channel attention map; The channel attention map is subjected to channel adjustment processing to obtain the adjusted channel attention map; The adjusted channel attention map is activated to obtain the activated channel attention map. The channel refinement feature map is obtained based on the fifth input and the activated channel attention map.

4. The method according to claim 1, characterized in that, The step of refining the features of the fifth input in the spatial dimension to obtain a spatially refined feature map includes: Perform a depthwise separable convolution on the fifth input to obtain a spatial attention map; The spatial attention map is activated to obtain the activated spatial attention map. The spatial refinement feature map is obtained based on the fifth input and the activated spatial attention map.

5. The method according to claim 1, characterized in that, The SCFEMamba model is also obtained by removing the two stacked visual state space blocks in the last decoder of the UNetMamba model.

6. A remote sensing image segmentation device, characterized in that, include: The acquisition module is used to acquire remote sensing images; The segmentation module is used to segment the remote sensing image using the SCFEMamba model to obtain the outline map of the ground feature area; The SCFEMamba model is obtained by replacing two stacked visual state space blocks in the first three decoders of the UNetMamba model with feature enhancement blocks and at least one sequentially connected improved decoding block. The feature enhancement block is used to perform feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder. The improved decoding block includes: A channel segmentation layer is used to segment the input of the improved decoding block to obtain a first input and a second input; The convolutional branch is used to perform convolution processing on the first input to obtain the convolutional branch feature map; The state branch is used to perform visual state space processing on the second input to obtain the state branch feature map; The pre-output module is used to obtain the output of the improved decoding block based on the convolutional branch feature map, the state branch feature map, and the input of the improved decoding block; The step of performing visual state space processing on the second input to obtain a state branch feature map includes: The second input is segmented to obtain the third and fourth inputs; Perform linear embedding and activation operations on the third input to obtain a side branch feature map; The fourth input is subjected to depthwise separable convolution and scanning attention operations to obtain the main branch feature map; The state branch feature map is obtained based on the side branch feature map and the main branch feature map; The scanning attention operation includes: A two-dimensional selective scanning operation is performed on the input of the scanning attention operation to obtain a scanning feature map; An attention operation is performed on the input of the scanning attention operation to obtain an attention weight matrix; The output of the scanning attention operation is obtained based on the scan feature map and the attention weight matrix; The step of performing feature enhancement processing on the output of the last improved decoding block to obtain the output of the decoder includes: The output of the last improved decoding block is subjected to channel adjustment processing to obtain an adjusted feature map; Parallel depthwise separable convolution operations are performed on the adjusted feature maps to obtain multiple second convolutional feature maps; The output of the decoder is obtained based on multiple second convolutional feature maps; The SCFEMamba model is further obtained by replacing the last layer of the UNetMamba model with a feature refinement block, which includes: A refined input module is used to obtain the fifth input based on the output of the first encoder and the output of the last decoder; The channel path is used to refine the features of the fifth input in the channel dimension to obtain a channel-refined feature map. The spatial path is used to refine the features of the fifth input in the spatial dimension to obtain a spatially refined feature map. The refinement output module is used to obtain the outline map of the land feature area based on the channel refinement feature map, the spatial refinement feature map, and the fifth input.

Images