A dual-branch remote sensing image semantic segmentation method and system based on Mamba and adaptive convolution

Abstract

This invention relates to a dual-branch remote sensing image semantic segmentation method and system based on Mamba and adaptive convolution, belonging to the fields of remote sensing image semantic segmentation and artificial intelligence technology. This method aims to solve the problems of existing technologies, such as difficulty in coordinating global dependencies and local texture information, low segmentation accuracy at multiple scales, poor multi-band adaptability, and low training efficiency. The invention includes using a multispectral adaptive processor for spectral grouping and attention enhancement, inputting the data into a dual-path backbone network after downsampling through a convolutional backbone. The global path uses a Mamba module to achieve linear complexity global modeling, while the local path uses adaptive convolution to dynamically adjust weights to capture details. A multi-head cross-attention fusion module then enables bidirectional interaction of multi-scale features, and a feature pyramid network performs cross-scale optimization and resolution restoration. This invention significantly improves segmentation accuracy, reduces computational burden, and adapts to the real-time application requirements of high-resolution remote sensing images.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image semantic segmentation and artificial intelligence technology, and relates to a two-branch remote sensing image semantic segmentation method and system based on Mamba and adaptive convolution. Background Technology

[0002] Semantic segmentation of remote sensing images is a core technology in fields such as land cover classification, urban planning, and disaster assessment. Its main task is to accurately label pixel-level land cover categories, such as buildings, roads, and water bodies. With the widespread availability of high-resolution and multi-band remote sensing data, early machine learning methods such as Support Vector Machines (SVMs) have become insufficient to meet the demands of current complex applications due to their over-reliance on manually designed features and poor generalization ability.

[0003] In the era of deep learning, Convolutional Neural Networks (CNNs) have become the mainstream solution due to their excellent local feature extraction capabilities. However, CNNs are limited by the size of their local receptive fields, making them unable to effectively capture long-range dependencies in images. To address this issue, VisionTransformers (ViTs) implement global modeling based on a self-attention mechanism. However, ViTs suffer from quadratic computational complexity, resulting in extremely high hardware costs when processing high-resolution remote sensing images.

[0004] The Mamba model, proposed in recent years, achieves global feature capture with linear complexity through a State Space Model (SSM), providing a new research direction for remote sensing image segmentation. Nevertheless, existing Mamba models still have certain limitations in the field of remote sensing.

[0005] First, existing integration solutions often simply replace network components, failing to effectively address the compatibility issue between the causal logic of the Mamba model and the non-causal features of visual data. Second, some dual-branch models directly stack global and local features, easily leading to information redundancy and low fusion efficiency. Furthermore, remote sensing images contain targets at various scales, ranging from narrow roads to large forests, and traditional fixed convolutional kernels struggle to adapt to such drastic scale changes. While existing technologies employ architectures using dilated pooling or detail-aware modules, a highly efficient fusion mechanism that simultaneously handles multi-directional global scanning and adaptive local perception is still lacking. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a dual-branch remote sensing image semantic segmentation method and system based on Mamba and adaptive convolution, aiming to solve the problems of difficulty in coordinating global dependency and local texture information, low multi-scale target segmentation accuracy, poor multi-band adaptability, and low training efficiency in existing technologies. By constructing a dual-path backbone network and combining OmnidirectionalSS2D multi-directional scanning technology and AdaptiveConv dynamic convolution technology, global information modeling and local detail capture are strengthened respectively; by using the MHCAFM bidirectional attention fusion module and FPN feature pyramid network, efficient fusion of global and local features is achieved, improving segmentation accuracy; at the same time, the model structure is optimized to support cropping and quantization operations, improve training efficiency, and enable it to be adapted for edge deployment, meeting the needs of engineering implementation.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A two-branch remote sensing image semantic segmentation method based on Mamba and adaptive convolution includes the following steps: The remote sensing image to be segmented is acquired, and the remote sensing image is subjected to spectral grouping and attention enhancement processing using a multi-spectral adaptive processor (MSP). The processed remote sensing image is input into the convolutional backbone (Conv Stem) for feature extraction and downsampling to obtain the basic feature map at the first scale; The basic feature maps are input into the dual-path backbone network in parallel. The global path extracts global dependency features at multiple scales through multiple Mamba stages in series, while the local path extracts local detail features at multiple scales through multiple adaptive convolution stages in series. At each scale, the global dependency features and the local detail features are fused bidirectionally using a Multi-Head Cross-AttentionFusion Module (MHCAFM) to obtain a multi-scale fused feature set; The multi-scale fused feature set is optimized and its resolution restored using a Feature Pyramid Network (FPN) to output a semantic segmentation result consistent with the resolution of the remote sensing image.

[0008] Furthermore, the step of performing spectral grouping and attention enhancement processing on the remote sensing image using a multispectral adaptive processor (MSP) includes: The remote sensing image is divided into multiple spectral feature groups according to the channel dimension; Channel statistics for each of the spectral feature groups are extracted using global average pooling. The weight coefficients of each spectral feature group are calculated through a shared fully connected layer, and the weight coefficients are multiplied element-wise with the corresponding spectral feature group.

[0009] Furthermore, the convolutional backbone includes two consecutive convolutional layers with a stride of 2, and each convolutional layer is followed by a layer of Layer Norm (LN) and a layer of Rectified Linear Unit (ReLU) activation function.

[0010] Furthermore, the process of extracting global dependency features at multiple scales from the global path includes: Adjust the dimensional order of the input feature maps to Format, in which, B Indicates batch size. H Indicates altitude, W Indicates width, C The number of channels is represented; the feature map after dimension adjustment is standardized using a Layer Normalization (LN) layer. Flatten the standardized feature map as follows The format is then input into the Mamba module for global feature modeling; The characteristics output by the Mamba module were readjusted. Format and output.

[0011] Furthermore, the local path extracts local detail features at multiple scales using an adaptive convolution (AdaptiveConvolution) module.

[0012] Furthermore, the calculation formula for the adaptive convolution module is expressed as follows:

[0013]

[0014]

[0015] in, x Indicates input features, express Fixed convolution operation, Represents basic local features. This indicates an adaptive average pooling operation. express Convolution operation, This represents the activation function. Indicates dynamic weights, This represents the enhanced local features of the output.

[0016] Furthermore, the fusion process of the multi-head cross-attention fusion module MHCAFM includes: use Convolutions are applied to the global dependency features respectively. g and the aforementioned local detail features l Perform channel projection to obtain projection features and ; The projection feature and Flatten and standardize using the layer normalization LN layer.

[0017] Furthermore, with the aforementioned projection features as query vector Q ,by as a key vector K Sum value vector V Attention response values ​​are calculated through a multi-head cross-attention mechanism. Its calculation formula is expressed as:

[0018] in, d Representing feature dimension, Softmax Represents the normalized exponential function; the attention response value After being enhanced by a feed-forward network (FFN), the features are residually connected to the original global dependency features to obtain fused features.

[0019] Furthermore, the processing flow of the Feature Pyramid Network (FPN) includes: The fusion features at the three scales were respectively obtained through Lateral convolution unifies the output channel dimension; Starting with the highest-scale fused features, bilinear interpolation is used for upsampling, and then element-wise addition is performed with the adjacent low-scale lateral features to achieve the transfer of high-level semantic features to low-level features. Perform the summation of features at each scale Convolutional enhancement processing is performed to obtain a complete feature pyramid; the lowest-scale features in the feature pyramid are finally convolved and mapped to a class probability map, and then output at the original resolution of the remote sensing image.

[0020] A two-branch remote sensing image semantic segmentation system based on Mamba and adaptive convolution, the system comprising: The Multispectral Adaptive Processor (MSP) is used to perform spectral grouping and attention enhancement processing on acquired remote sensing images. The convolutional backbone, whose input is connected to the output of the multispectral adaptive processor MSP, is used to extract the basic feature map at the first scale. The dual-path backbone network has its input connected to the output of the convolutional backbone and includes parallel global and local paths. The global path is used to extract global dependency features at multiple scales, and the local path is used to extract local detail features at multiple scales. Multiple multi-head cross-attention fusion modules (MHCAFM) are connected to the corresponding scale outputs of the global path and the local path, respectively, to fuse global dependency features and local detail features at the same scale and output a multi-scale fused feature set. The Feature Pyramid Network (FPN) is connected to the output of each of the Multi-Head Cross-Attention Fusion Modules (MHCAFM), and is used to perform cross-scale optimization on the multi-scale fused feature set and output semantic segmentation results.

[0021] The beneficial effects of this invention are as follows: (1) This invention introduces the Mamba module to construct a global path and takes advantage of the linear computational complexity of the state space model to achieve accurate capture of large-scale spatial context information of remote sensing images without significantly increasing the computational burden, thus overcoming the memory bottleneck of traditional visual transformers when processing high-resolution remote sensing data.

[0022] (2) The local path designed in this invention adopts the AdaptiveConv module, which can dynamically adjust the perception weight according to the statistical characteristics of the input features, thereby effectively dealing with the problem of drastic changes in target scale in remote sensing images and improving the extraction accuracy of small roads and complex ground features edges.

[0023] (3) The multi-head cross-attention fusion module MHCAFM used in this invention realizes the deep interaction between global dependent features and local detailed features, avoids information redundancy caused by simple feature superposition, enhances the model's ability to identify ground features in complex backgrounds, and significantly improves the average intersection-union ratio of remote sensing image semantic segmentation.

[0024] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 A schematic diagram of the overall architecture of a dual-branch remote sensing image semantic segmentation system based on Mamba and adaptive convolution provided for an embodiment of the present invention; Figure 2 This is a schematic diagram of the internal structure of the Mamba Stage of the global path in the dual-path backbone network in this embodiment of the invention. Figure 3 This is a schematic diagram of the structure of the AdaptiveConv module for local paths in the dual-path backbone network in this embodiment of the invention; Figure 4 This is a schematic diagram of the feature fusion logic of the Multi-Head Cross-AttentionFusion Module (MHCAFM) in an embodiment of the present invention; Figure 5 This is a schematic diagram of the cross-scale feature enhancement structure of the Feature Pyramid Network (FPN) in an embodiment of the present invention. Figure 6 This is a comparison chart of the Vaihingen data results. Detailed Implementation

[0026] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0027] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0028] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0029] This invention proposes a dual-branch remote sensing image semantic segmentation neural network architecture (DBMACNet) based on Mamba and adaptive convolution. With a dual-path backbone network at its core, it innovatively integrates modules such as multispectral adaptive processing, multi-scale bidirectional attention fusion, and feature pyramid output. Through a complete workflow of "preprocessing - dual-path feature extraction - multi-scale feature fusion - multi-scale optimized output," it achieves pixel-level accurate semantic segmentation of remote sensing images. The specific technical solution is as follows: (I) Overall Structure of the Model 1. Overall structure of the model like Figure 1 As shown, the network architecture of this invention follows a closed-loop logic of "targeted preprocessing - path feature extraction - multi-scale intelligent fusion - multi-scale output," and its core comprises five functional modules. Each module works in concert according to the data flow direction to form an end-to-end segmented link: Input layer: Receives multi-band high-resolution remote sensing image data (in H×W×3 format or multispectral extended channels), covering different spectral types such as visible light, near-infrared, and short-wave infrared; Preprocessing layer: Composed of a multispectral adaptive processor, which performs grouping processing and attention enhancement for multi-band spectral heterogeneity; Feature extraction layer: namely, dual-path backbone network, which includes Mamba global branch and adaptive convolutional local branch, extracts global dependent features and local detailed features in parallel, and outputs intermediate features simultaneously at three scales: 1 / 2, 1 / 4 and 1 / 8. Feature fusion layer: Through the MHCAFM bidirectional attention fusion module, adaptive weighted fusion of global and local features is achieved at three corresponding scales to form a multi-scale fusion feature set; Output layer: The FPN feature pyramid network is used to perform cross-scale feature optimization and resolution restoration, and finally outputs a semantic segmentation result with the same resolution as the input image.

[0030] 2. Overall data flow The input remote sensing image is first processed by a multispectral adaptive processor to perform spectral grouping, channel enhancement, and noise suppression. Then, preliminary feature extraction and downsampling (stride=2) are performed through the convolutional backbone (Conv Stem) to obtain basic features at the 1 / 2 scale. The basic features are fed into the Mamba global branch and the adaptive convolutional local branch, respectively. The two branches are processed in multiple stages (3 stages each) to generate feature maps simultaneously at the 1 / 2, 1 / 4, and 1 / 8 scales. The global and local features at each scale are fused by bidirectional attention through the MHCAFM module. Finally, the multi-scale fused features are input into the FPN feature pyramid network. Through top-down upsampling and bottom-up feature complementation, a complete feature pyramid is constructed, which is then mapped to the final semantic segmentation result through the segmentation head (SegHead).

[0031] (II) Detailed Design of Core Modules 1. Dual-path backbone network A dual-branch parallel structure is adopted, focusing on global long-range dependency modeling and local detail feature capture respectively, and outputting features simultaneously at three scales to provide data support for multi-scale fusion: (1) Global path (Mamba branch) Core functionality: Leveraging the linear complexity advantage of the Mamba model and combining omnidirectional feature extraction, it captures the global spatial dependence and long-range correlation of remote sensing images. Module composition: It consists of 3 Mamba Stages connected in series. Each Stage contains 1 MambaBlock, and scale compression between Stages is achieved through a downsampling module (global_down1 / 2). Key component: MambaBlock structure as follows Figure 2 As shown.

[0032] Dimension transformation: Permute the input feature map (B×C×H×W) to B×H×W×C format to adapt to Mamba input requirements; Layer normalization (LayerNorm): Standardizes the feature distribution to improve training stability; Feature flattening and Mamba processing: Flatten the features into B×(H×W)×C, input them into the Mamba module to achieve global feature modeling; Dimension restoration: The processed features are reshaped to B×H×W×C, and then permute back to B×C×H×W format for output; The input features are processed by Conv Stem to obtain 1 / 2 scale features, which are then processed by Mamba Stage 1 to output 1 / 2 scale global features. The features are then downsampled to 1 / 4 scale by global_down1 (3×3 convolution, stride=2) and output by Mamba Stage 2. Finally, the features are downsampled to 1 / 8 scale by global_down2 and output by Mamba Stage 3. Core advantages: It achieves global modeling of linear complexity through Mamba's State Space Model (SSM), and with multi-stage scale evolution, it balances global consistency and scale adaptability, solving the problems of high computational complexity in traditional ViT and loss of directional information in Mamba's unidirectional processing.

[0033] (2) Local Path (Adaptive Convolutional Branch) Core function: Based on the dynamic convolution kernel adjustment mechanism, it accurately captures local details in remote sensing images (such as narrow roads, building edges, and scattered vegetation) and reduces the omission rate of small targets; Module composition: It consists of 3 Adapt Stages connected in series. Each Stage contains 1 AdaptiveConv module, and the scale synchronization between Stages is achieved through the downsampling module (local_down1 / 2). Key component: AdaptiveConv structure as follows Figure 3 As shown.

[0034] Basic feature extraction: Basic local features (base_feat) are extracted using a 3×3 fixed convolution kernel (stride=1, padding=1). Dynamic weight generation: The input features are subjected to adaptive average pooling (AdaptiveAvgPool2d(1)), followed by 1×1 convolution and Sigmoid activation to generate dynamic weights with the same dimensions as the basic features. Feature weighted fusion: Basic features are multiplied element-wise with dynamic weights to achieve dynamic enhancement of local features; Scale evolution: Synchronized with the global path, starting from 1 / 2 scale features, 1 / 2 scale local features are output through Adapt Stage 1; downsampled to 1 / 4 scale through local_down1(AdaptiveConv, stride=2), 1 / 4 scale local features are output by Adapt Stage 2; downsampled to 1 / 8 scale through local_down2, 1 / 8 scale local features are output by Adapt Stage 3; Core advantage: The dynamic weight generation mechanism enables the convolution kernel to adapt to the local texture distribution, which can capture detailed features more accurately than fixed convolution kernels, solving the problem of insufficient flexibility in local feature extraction in traditional CNNs.

[0035] 3. MHCAFM Bidirectional Attention Fusion Module To achieve deep complementarity between global and local features, a multi-scale cross-attention fusion module is designed to perform feature fusion at three scales: 1 / 2, 1 / 4, and 1 / 8. Core functionality: Through a multi-head cross-attention mechanism, it adaptively learns the fusion weights of global and local features, highlighting key features (such as land cover boundaries and category distinguishing features) and achieving complementary advantages between the two types of features; The implementation process is as follows Figure 4 As shown.

[0036] Feature projection: Global features (g) and local features (l) are projected into channels (g_proj, l_proj) through 1×1 convolutions to unify feature dimensions; Dimensional transformation and normalization: The projected features are flattened into B×(H×W)×C format and normalized by layer normalization (global_norm, local_norm); Multi-head attention computation: Using g_proj as the query (Q) and l_proj as the key (K) and value (V), the attention response (g_attn) of global features to local features is calculated through the multi-head attention mechanism. Feature reshaping and enhancement: The attention response is reshaped into B×C×H×W format, enhanced by a feedforward network (FFN: 1×1 convolution-GELU-1×1 convolution), and finally connected with the original global feature residuals to output fused features; Multi-scale adaptation: Independent MHCAFM modules are deployed at three scales: 1 / 2, 1 / 4, and 1 / 8. The fusion strategy is adaptively adjusted according to the characteristics of different scales (such as focusing on structure at large scales and focusing on details at small scales). Core advantages: Cross-attention mechanism enables precise association between global and local features, residual connection and FFN enhance feature representation, solving the problems of feature conflict and information redundancy in traditional fusion methods.

[0037] 4. Feature Pyramid Network (FPN) To address the segmentation needs of multi-scale land features (such as large forests and small buildings, wide rivers and narrow canals) in remote sensing images, a cross-scale feature optimization module is constructed: Core function: Integrate features from three scales to construct a complete feature pyramid, achieving complementarity between high-level semantic features and low-level detail features, thereby improving the accuracy of multi-scale target segmentation; Module composition: Includes lateral convolution (lateral_convs), top-down upsampling, bottom-up feature fusion, and output convolution (output_convs, final_conv). The implementation process is as follows Figure 5 As shown.

[0038] Lateral feature extraction: The fused features at three scales (1 / 8 high scale, 1 / 4 medium scale, and 1 / 2 low scale) are processed by 1×1 lateral convolutions to unify the output channel dimension; Top-down upsampling: Starting from the highest scale (1 / 8) features, upsampling is performed to the adjacent lower scale (1 / 4) through bilinear interpolation, and then added to the lateral features at that scale to achieve the transfer of high-level semantic features to the lower level; this process is repeated to the lowest scale (1 / 2). Feature enhancement: After summing at each scale, 3×3 convolutions (output_convs) are used to enhance feature smoothness and expressive power; Final prediction: The lowest-scale features are mapped to a class probability map through final convolution (final_conv), and then restored to the input image resolution through 4x bilinear interpolation to output the semantic segmentation result; Core advantages: Cross-scale feature fusion takes into account both global semantics and local details, solving the problem that single-scale features are difficult to adapt to multi-scale features, and ensuring the segmentation accuracy of features of different sizes.

[0039] I. Network Overall Architecture Implementation Example This embodiment provides a dual-branch remote sensing image semantic segmentation deep learning network structure (DBMACNet) based on Mamba and adaptive convolution. It adopts a complete architecture process of "dual-path feature extraction - multi-scale bidirectional attention fusion - feature pyramid output," and the parameter settings, processing logic, and output specifications for each stage are clearly defined as follows: Main feature extraction stage 1. Convolutional Stem Settings: First convolutional layer: 3×3 convolutional kernel, stride=2, padding=1, number of input channels=6, number of output channels=32, no bias, BatchNorm2d layer followed by GELU activation, output feature map resolution 512×512 (1 / 2 scale). The second convolutional layer has a 3×3 kernel, stride=2, padding=1, input channels=32, output channels=64, and no bias. The BatchNorm2d layer is followed by GELU activation, and the final output is a 1 / 4 scale basic feature map (B×64×256×256).

[0040] Dual-path feature extraction stage (1) Global path (Mamba branch) Core parameter configuration: Mamba module d_model=64 (base channel count), d_state=16, expand=2, layer normalization (LayerNorm) ε=1e-5; downsampling module uses 3×3 convolution, stride=2, padding=1, no bias, paired with BatchNorm2d and GELU activation.

[0041] Multi-stage processing flow: Stage 1 (1 / 4 scale): Input basic feature map (B×64×256×256), processed by MambaBlock—first converted to B×256×256×64 format by permute, then flattened to B×(256×256)×64 after LayerNorm normalization, input to Mamba module to capture global dependencies, output reshaped to B×256×256×64, then permute back to B×64×256×256 format, to obtain 1 / 4 scale global features; Downsampling to 1 / 8 scale: Through the global_down1 module (3×3 convolution, 64 input channels, 128 output channels), the output feature map resolution is 128×128 (B×128×128×128). Stage 2 (1 / 8 scale): Connect to MambaBlock (d_model=128, other parameters are the same as before), and output 1 / 8 scale global features (B×128×128×128). Downsampling to 1 / 16 scale: Through the global_down2 module (3×3 convolution, 128 input channels, 256 output channels), the output feature map resolution is 64×64 (B×256×64×64). Stage 3 (1 / 16 scale): Connect to MambaBlock (d_model=256, other parameters are the same as before), and finally output the global features at the 1 / 16 scale (B×256×64×64).

[0042] (2) Local Path (Adaptive Convolutional Branch) Core parameter configuration: AdaptiveConv convolution kernel size 3×3, padding=1, dynamic weight generator includes AdaptiveAvgPool2d(1), 1×1 convolution (number of input channels = number of current channels, number of output channels = number of current channels) and Sigmoid activation; downsampling module is implemented using AdaptiveConv, stride=2, padding=1, with BatchNorm2d and GELU activation.

[0043] Multi-stage processing flow: Stage 1 (1 / 4 scale): Input basic feature map (B×64×256×256), processed by AdaptiveConv (64 input channels, 64 output channels, stride=1) - 3×3 fixed convolution extracts basic local features, dynamic weight generator generates weight mask based on local texture (such as road edges, building corners), the two are multiplied element-wise to obtain enhanced local features, and then activated by BatchNorm2d and GELU, outputting 1 / 4 scale local features (B×64×256×256). Downsampling to 1 / 8 scale: Through the local_down1 module (AdaptiveConv, 64 input channels, 128 output channels, stride=2), the output feature map resolution is 128×128 (B×128×128×128). Stage 2 (1 / 8 scale): Input AdaptiveConv (128 channels for input, 128 channels for output, stride=1), processing flow is the same as Stage 1, output 1 / 8 scale local features (B×128×128×128). Downsampling to 1 / 16 scale: Through the local_down2 module (AdaptiveConv, 128 input channels, 256 output channels, stride=2), the output feature map resolution is 64×64 (B×256×64×64). Stage 3 (1 / 16 scale): Input AdaptiveConv (256 channels for input, 256 channels for output, stride=1), and finally output 1 / 16 scale local features (B×256×64×64).

[0044] Multi-scale bidirectional attention fusion process MHCAFM module parameters: num_heads=4 (number of multi-head attention heads), number of channels per head = number of current channels / / 4, number of intermediate channels in FFN = number of current channels × 2, activation function is GELU.

[0045] Three-scale fusion process: 1 / 4 Scale Fusion: Input global path Stage1 features (B×64×256×256) and local path Stage1 features (B×64×256×256), after 1×1 convolutional projection (64 input channels, 64 output channels), flattening (B×(256×256)×64), and LayerNorm normalization, multi-head cross-attention calculation is performed with global features as Q and local features as K / V. The output attention response is reshaped to B×64×256×256, and after FFN enhancement and residual connection (fusion feature + original global feature), a 1 / 4 scale fused feature (B×64×256×256) is obtained. 1 / 8 scale fusion: Input global path Stage2 features (B×128×128×128) and local path Stage2 features (B×128×128×128), the process is the same as 1 / 4 scale, and output 1 / 8 scale fused features (B×128×128×128). 1 / 16 scale fusion: Input global path Stage3 features (B×256×64×64) and local path Stage3 features (B×256×64×64), follow the same process as before, and output 1 / 16 scale fused features (B×256×64×64).

[0046] Feature Pyramid Output Stage FPN module parameters: Lateral convolution (1×1) with no bias, output channel number is uniformly 128; Output convolution (3×3) padding=1, no bias, with BatchNorm2d and ReLU activation; Upsampling uses bilinear interpolation, align_corners=False; Final convolution (1×1) output channel number = 8 (number of land cover categories).

[0047] Processing flow: Lateral feature extraction: 1 / 16 scale fusion features (256 channels) are mapped to 128 channels through 1×1 convolution; 1 / 8 scale fusion features (128 channels) are directly maintained at 128 channels through 1×1 convolution; 1 / 4 scale fusion features (64 channels) are mapped to 128 channels through 1×1 convolution. Top-down upsampling fusion: The 1 / 16 scale lateral features are upsampled to the 1 / 8 scale (128×128) by bilinear interpolation, and then added element-wise with the 1 / 8 scale lateral features. Finally, the output is enhanced by a 3×3 output convolution to obtain the 1 / 8 scale optimized features. The 1 / 8 scale optimized features are upsampled to the 1 / 4 scale (256×256) by bilinear interpolation, and then added element-wise with the 1 / 4 scale lateral features. After 3×3 output convolution enhancement, the 1 / 4 scale optimized features are obtained. Final prediction: The 1 / 4 scale optimized features are upsampled to the original resolution (1024×1024) by 4x bilinear interpolation, mapped to an 8-channel class probability map by 1×1 convolution, and then activated by Softmax to obtain the class prediction results for each pixel. The output format is B×8×1024×1024.

[0048] II. Verification of Technical Effects 1. Comparative experimental setup Comparison Models: Existing mainstream remote sensing image segmentation models were selected, including MambaUNet, RS3Mamba, U-Net (CNN benchmark model), and ViT-UNet (Transformer benchmark model). All models were trained in the same experimental environment and on the same dataset, with consistent training parameters (batch size = 8, initial learning rate = 1e-4, AdamW optimizer used, weight decay = 1e-5, training epochs = 100, and the learning rate decayed to 1 / 10 of its original value at epochs 50 and 80).

[0049] Evaluation metrics: The average intersection-union ratio (mIoU) is mainly used to evaluate segmentation accuracy. At the same time, the training time per epoch, the number of model parameters (params), and the inference speed (processing time for a single 1024×1024 image) are also statistically analyzed.

[0050] 2. Experimental Results and Analysis The present invention was compared with some classic models on the Potsdam and LoveDa datasets, and the results are shown in Tables 1 and 2.

[0051] Table 1 Potsdam dataset

[0052] Table 2 LoveDa dataset

[0053] Figure 6 This is the result of the Vaihingen data.

[0054] Segmentation accuracy advantage: The present invention achieves the highest mIoU on both datasets, improving upon the best comparison model RS3Mamba by 3.6%-3.9%. Among them, the IoU of small targets such as fine roads and scattered vegetation is improved by 4.6%-5.2%, thanks to the AdaptiveConv dynamic capture of local details. In multispectral data processing, due to the spectral grouping and attention mechanism of the multispectral adaptive processor, the improvement is 3.5%-4.3% compared with the traditional model, verifying the multi-band adaptability.

[0055] Efficiency and lightweight advantages: The model has only 35.2M parameters, which is 20%-32% less than the comparison model; the training time per round is 23%-27% shorter than RS3Mamba, and the inference speed is 30%-40% faster, meeting the real-time processing requirements of edge devices, which proves the effectiveness of structural pruning and parallel design.

[0056] Scene adaptability advantage: It maintains stable and high performance in both urban (LoveDa) and general (Potsdam) scenarios, indicating that the dual-path fusion and multi-scale optimization architecture has strong generalization ability and can be adapted to different remote sensing image segmentation scenarios.

[0057] Example 1 This embodiment provides a two-branch remote sensing image semantic segmentation method and system based on Mamba and adaptive convolution, aiming to solve the problem of difficulty in simultaneously taking into account global long-range dependencies and local fine details in remote sensing images.

[0058] The method described in this embodiment includes the following workflow: First, the system acquires the remote sensing image to be segmented. This remote sensing image typically contains multiple spectral bands. A Multi-Spectral Processor (MSP) is used to perform spectral grouping and attention enhancement processing on the remote sensing image. In the MSP, the input image is divided according to the channel dimension. N Each spectral feature group is divided into several subgroups. Global average pooling is used to extract channel statistics for each subgroup, and weight coefficients for each subgroup are calculated using a shared fully connected layer. Finally, these weight coefficients are multiplied element-wise with the corresponding spectral feature group to enhance the feature representation of important spectral bands.

[0059] Next, the processed remote sensing image is input into a convolutional backbone (Conv Stem) for feature extraction and downsampling. The convolutional backbone consists of two consecutive convolutional layers with a stride of 2. Each convolutional layer is followed by a layer of Layer Norm (LN) activation and a layer of Rectified Linear Unit (ReLU) activation function. After processing by the convolutional backbone, a base feature map at a first-scale resolution, one-quarter that of the original image, is output.

[0060] Then, the basic feature maps are input in parallel into a dual-path backbone network. The dual-path backbone network includes a global path and a local path. The global path extracts multi-scale global dependency features through three cascaded Mamba stages. The local path extracts multi-scale local detail features through three cascaded Adaptive Stages.

[0061] At each scale, the system utilizes a Multi-Head Cross-Attention Fusion Module (MHCAFM) to perform bidirectional attention fusion on the global dependency features and the local detail features, generating a multi-scale fused feature set.

[0062] Finally, the multi-scale fused feature set is optimized across scales using a Feature Pyramid Network (FPN). The FPN restores the image resolution through lateral connections and top-down upsampling operations, ultimately outputting a semantic segmentation result with the same resolution as the original remote sensing image.

[0063] Example 2 This embodiment focuses on illustrating the specific implementation process of feature extraction in a dual-path backbone network, in order to support the technical solutions regarding the Mamba module and the Adaptive Convolution (AdaptiveConv) module in the claims.

[0064] In the global path, each Mamba Stage achieves linear complexity global modeling through a State Space Model (SSM). Its specific workflow is as follows: Adjust the dimensionality order of the input feature maps from channel-first format to... Format.

[0065] The feature map is standardized using a layer normalization (LN) layer.

[0066] Flatten the standardized feature map as follows Format.

[0067] Input the Mamba module to perform scanning and feature extraction in four directions.

[0068] Readjust the output features Format and output.

[0069] Within the local path, each Adapt Stage utilizes the AdaptiveConv module to capture details of ground features. The computational flow of the adaptive convolution module is as follows: Input features x First, go through a The fixed convolution operation yields the basic local features, expressed by the formula: Meanwhile, input features x After adaptive average pooling operation and The convolution operation is then passed through an activation function. The dynamic weights are generated using the following formula: Finally, the basic local features are multiplied element-wise with the dynamic weights to obtain the output features, expressed by the formula: The AdaptiveConv convolution can dynamically adjust the attention of the convolution kernel according to the content of the input image, thereby better adapting to buildings and roads of different shapes in remote sensing images.

[0070] Example 3 This embodiment focuses on illustrating the internal logic of the multi-head cross-attention fusion module MHCAFM and the feature pyramid network FPN to support the technical solutions regarding feature fusion and result output in the claims.

[0071] The workflow of the multi-head cross-attention fusion module MHCAFM is as follows: System utilization Convolutions are applied to the global dependency features respectively. g and the aforementioned local detail features l Perform channel projection to obtain projection features and The projected features are flattened and normalized using a layer normalization (LN) layer. as query vector Q ,by as a key vector K Sum value vector V Calculate attention response values ​​using a multi-head cross-attention mechanism. The calculation formula is expressed as: in, d Representing feature dimension, Softmax This represents the normalized exponential function. The attention response value... After being enhanced by a feed-forward network (FFN), it is residually connected to the original global dependency features.

[0072] The processing flow of the Feature Pyramid Network (FPN) is as follows: The fusion features at the three scales were respectively obtained through Lateral convolution unifies the number of channels.

[0073] Starting with the highest-scale fused features, upsampling is performed using bilinear interpolation.

[0074] The upsampled features are added element-wise with the adjacent low-scale lateral features to achieve the downward transmission of semantic information.

[0075] Perform on the features after addition Convolution is used to eliminate the aliasing effect caused by upsampling.

[0076] The lowest-scale feature map is mapped to a class probability map, and the final segmentation mask is output.

[0077] In the test experiments of this invention, the Potsdam remote sensing dataset was used for verification. The experimental results show that the method described in this embodiment achieves a mean Intersection over Union (mIoU) ratio of 86.29%, which is significantly better than traditional convolutional neural network or visual transformer architectures, demonstrating the advanced nature of the technical solution of this invention.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A two-branch remote sensing image semantic segmentation method based on Mamba and adaptive convolution, characterized in that: Includes the following steps: The remote sensing image to be segmented is acquired, and the remote sensing image is subjected to spectral grouping and attention enhancement processing using a multispectral adaptive processor (MSP). The processed remote sensing image is input into the Conv Stem convolutional backbone for feature extraction and downsampling to obtain the basic feature map at the first scale. The basic feature maps are input into the dual-path backbone network in parallel. The global path extracts global dependency features at multiple scales through multiple Mamba stages in series, and the local path extracts local detail features at multiple scales through multiple Adapt convolution stages in series. At each scale, the global dependency features and the local detail features are fused bidirectionally using the multi-head cross-attention fusion module MHCAFM to obtain a multi-scale fused feature set; The Feature Pyramid Network (FPN) is used to perform cross-scale feature optimization and resolution restoration on the multi-scale fused feature set, and outputs a semantic segmentation result with the same resolution as the remote sensing image.

2. The dual-branch remote sensing image semantic segmentation method based on Mamba and adaptive convolution as described in claim 1, characterized in that: The steps of performing spectral grouping and attention enhancement processing on the remote sensing image using a multispectral adaptive processor (MSP) include: The remote sensing image is divided into multiple spectral feature groups according to the channel dimension; Channel statistics for each of the spectral feature groups are extracted using global average pooling. The weight coefficients of each spectral feature group are calculated through a shared fully connected layer, and the weight coefficients are multiplied element-wise with the corresponding spectral feature group.

3. The semantic segmentation method for dual-branch remote sensing images based on Mamba and adaptive convolution according to claim 1, characterized in that: The convolutional backbone consists of two consecutive convolutional layers with a stride of 2, and each convolutional layer is followed by a normalized LN layer and a modified linear unit (ReLU) activation function.

4. The dual-branch remote sensing image semantic segmentation method based on Mamba and adaptive convolution according to claim 1, characterized in that: The process of extracting global dependency features at multiple scales using the global path includes: Adjust the dimensional order of the input feature maps to Format, in which, B Indicates batch size. H Indicates altitude, W Indicates width, C The number of channels is represented; the adjusted feature map is standardized using a Layer Normalization (LN) layer. Flatten the standardized feature map as follows The format is then input into the Mamba module for global feature modeling; The characteristics output by the Mamba module were readjusted. Format and output.

5. The dual-branch remote sensing image semantic segmentation method based on Mamba and adaptive convolution according to claim 1, characterized in that: The local path extracts local detail features at multiple scales using the AdaptiveConv module.

6. The dual-branch remote sensing image semantic segmentation method based on Mamba and adaptive convolution according to claim 1, characterized in that: The calculation formula for the adaptive convolution module is expressed as follows: in, x Indicates input features, express Fixed convolution operation, Represents basic local features. This indicates an adaptive average pooling operation. express Convolution operation, This represents the activation function. Indicates dynamic weights, This represents the enhanced local features of the output.

7. The dual-branch remote sensing image semantic segmentation method based on Mamba and adaptive convolution according to claim 1, characterized in that: The fusion process of the multi-head cross-attention fusion module MHCAFM includes: use Convolutions are applied to the global dependency features respectively. g and the aforementioned local detail features l Perform channel projection to obtain projection features and ; The projection feature and Flatten and standardize using the layer normalization LN layer.

8. The dual-branch remote sensing image semantic segmentation method based on Mamba and adaptive convolution according to claim 7, characterized in that: With the projection features as query vector Q ,by as a key vector K Sum value vector V Attention response values ​​are calculated through a multi-head cross-attention mechanism. Its calculation formula is expressed as: in, d Representing feature dimension, Softmax Represents the normalized exponential function; the attention response value After being enhanced by the feedforward network FFN, the fused features are residually connected with the original global dependency features to obtain the fused features.

9. The dual-branch remote sensing image semantic segmentation method based on Mamba and adaptive convolution according to claim 1, characterized in that: The processing flow of the Feature Pyramid Network (FPN) includes: The fusion features at the three scales were respectively obtained through Lateral convolution unifies the output channel dimension; Starting with the highest-scale fused features, bilinear interpolation is used for upsampling, and then element-wise addition is performed with the adjacent low-scale lateral features to achieve the transfer of high-level semantic features to low-level features. Perform the summation of features at each scale Convolutional enhancement processing is performed to obtain a complete feature pyramid; the lowest-scale features in the feature pyramid are finally convolved and mapped to a class probability map, and then output at the original resolution of the remote sensing image.

10. A two-branch remote sensing image semantic segmentation system based on Mamba and adaptive convolution, characterized in that: The system includes: The Multispectral Adaptive Processor (MSP) is used to perform spectral grouping and attention enhancement processing on acquired remote sensing images. The convolutional backbone, whose input is connected to the output of the multispectral adaptive processor MSP, is used to extract the basic feature map at the first scale. The dual-path backbone network has its input connected to the output of the convolutional backbone and includes parallel global and local paths. The global path is used to extract global dependency features at multiple scales, and the local path is used to extract local detail features at multiple scales. Multiple multi-head cross-attention fusion modules (MHCAFM) are connected to the corresponding scale outputs of the global path and the local path, respectively, to fuse global dependency features and local detail features at the same scale and output a multi-scale fused feature set. The Feature Pyramid Network (FPN) is connected to the output of each of the Multi-Head Cross-Attention Fusion Modules (MHCAFM), and is used to perform cross-scale optimization on the multi-scale fused feature set and output semantic segmentation results.

Images