Remote sensing image multi-scale segmentation method based on spectral information processing and mamba space modeling

Abstract

The application discloses a remote sensing image multi-scale segmentation method based on spectrum information processing and Mamba space modeling, and comprises the following steps: pre-processing a remote sensing image to obtain an input image; inputting the input image into a trained remote sensing image segmentation model; the remote sensing image segmentation model comprises an initial convolution layer, a spectrum domain information processing unit branch, a Mamba layer branch and a segmentation head; the initial convolution layer is used for feature extraction of the input image to obtain initial image features; the spectrum domain information processing unit branch and the Mamba layer branch are combined to extract features of the initial image features and obtain fusion features; the fusion features are up-sampled and decoded by the segmentation head, and pixel-level classification is performed to output a multi-scale segmentation result corresponding to the remote sensing image. The method combines the spectrum domain and the space domain to extract features, and solves the problems of common details blur, unclear boundaries and insufficient multi-scale target expression in remote sensing image segmentation.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and more specifically to a multi-scale segmentation method for remote sensing images based on spectral information processing and Mamba spatial modeling. Background Technology

[0002] Remote sensing image segmentation, as a key technology for intelligent interpretation of remote sensing images, has wide application value in fields such as land use classification, urban planning, ecological monitoring, and disaster assessment. Its core task is to divide input multi-source remote sensing images into several semantically meaningful land feature regions, such as buildings, roads, water bodies, farmland, and forests. However, due to the inherent characteristics of remote sensing images, such as high spatial resolution, rich spectral dimensions, diverse observation angles, and complex land feature types, the shapes of targets in the images are varied, with rich texture details and blurred boundaries, posing a significant challenge to traditional image segmentation methods.

[0003] In recent years, deep learning techniques, especially convolutional neural networks (CNNs) and Transformer architectures, have been widely applied to remote sensing image segmentation tasks, achieving significant progress. CNN-based methods typically employ an encoder-decoder structure, extracting semantic features from images through downsampling and then recovering spatial details through upsampling. However, pure CNN architectures still perform poorly when facing problems such as cross-scale target recognition, boundary continuity modeling, and texture preservation, as their receptive field is limited and they tend to overlook long-range dependencies. To address these issues, some studies have introduced Transformer architectures, using self-attention mechanisms to model global dependencies between distant pixels in an image, thereby improving overall semantic consistency. However, Transformer architectures have high computational complexity and insufficient ability to model details, making it difficult to effectively capture small targets or texture boundaries, especially exhibiting instability in remote sensing images.

[0004] Furthermore, existing methods are generally limited to spatial domain feature modeling, neglecting the important information contained in the frequency domain. In fact, the edge, texture, and shape information of ground objects in remote sensing images are more clearly and concentratedly expressed in the frequency domain. The spectral domain can effectively distinguish high-frequency details (such as edges and textures) from low-frequency components (such as background and structure) in images. If it can be jointly modeled with spatial domain information, it is expected to further improve the model's sensitivity to boundaries and small targets. However, most current image segmentation models have not yet fully explored the potential of the spectral domain in remote sensing image interpretation.

[0005] Therefore, how to combine the spectral domain and the spatial domain for feature extraction and solve common problems in remote sensing image segmentation such as blurred details, unclear boundaries, and insufficient representation of multi-scale targets is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the above problems, the present invention provides a multi-scale segmentation method for remote sensing images based on spectral information processing and Mamba spatial modeling, so as to at least solve some of the technical problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] This invention provides a multi-scale segmentation method for remote sensing images based on spectral information processing and Mamba spatial modeling, comprising the following steps:

[0009] The remote sensing images acquired from the target area are preprocessed to obtain the input image;

[0010] The input image is fed into a trained remote sensing image segmentation model; the remote sensing image segmentation model includes an initial convolutional layer, a spectral domain information processing unit branch, a Mamba layer branch, and a segmentation head;

[0011] The input image is used to extract features through the initial convolutional layer to obtain initial image features;

[0012] The spectral domain information processing unit branch and the Mamba layer branch are combined to extract features from the initial image features and obtain fused features.

[0013] The fused features are upsampled, decoded, and classified at the pixel level using a segmentation head, and the multi-scale segmentation result corresponding to the remote sensing image is output.

[0014] Furthermore, the preprocessing includes: radiometric correction, atmospheric correction, geometric correction, and noise suppression.

[0015] Furthermore, the initial convolutional layer consists of multiple first convolutional layers.

[0016] Furthermore, the step of combining the spectral domain information processing unit branch and the Mamba layer branch to extract features from the initial image features and obtain fused features specifically includes:

[0017] The initial image features are processed by the first spectral domain information processing unit and the first Mamba layer to obtain the corresponding first spectral features and first spatial features.

[0018] The first spectral feature and the first spatial feature are concatenated and then passed through a second convolutional layer to obtain the first fused feature;

[0019] The first fusion feature is processed by the second spectral domain information processing unit and the second Mamba layer to obtain the corresponding second spectral feature and second spatial feature.

[0020] The second spectral feature and the second spatial feature are concatenated and then passed through a third convolutional layer to obtain the second fused feature;

[0021] The second fusion feature is processed by the third spectral domain information processing unit and the third Mamba layer to obtain the corresponding third spectral feature and third spatial feature.

[0022] The third spectral feature, the third spatial feature, and the second spectral feature are concatenated and then passed through a fourth convolutional layer to obtain the third fused feature.

[0023] The third fusion feature is processed by the fourth spectral domain information processing unit and the fourth Mamba layer to obtain the corresponding fourth spectral feature and fourth spatial feature.

[0024] The fourth spectral feature, the fourth spatial feature, and the first spectral feature are concatenated and then passed through a fifth convolutional layer to obtain the fourth fused feature.

[0025] The fourth fusion feature is processed by the fifth spectral domain information processing unit and the fifth Mamba layer to obtain the corresponding fifth spectral feature and fifth spatial feature.

[0026] The fifth spectral feature, the fifth spatial feature, and the initial image feature are concatenated and then passed through a sixth convolutional layer to obtain the final fused feature.

[0027] Furthermore, the input features of each Mamba layer are denoted as Mamba input features, and the following operations are performed in each Mamba layer:

[0028] The channels of the Mamba input features are compressed and mapped;

[0029] The Mamba input features are rearranged according to the processed channel dimensions so that the Mamba layer can perform sequential scanning along the vertical direction of the image.

[0030] One-dimensional convolutional modeling is performed based on the forward state modeling mechanism to capture long-distance dependency features and obtain forward modeling results;

[0031] Flip the Mamba input features in the vertical direction;

[0032] One-dimensional convolution modeling is performed based on the backward state modeling mechanism to obtain contextual information extracted in the opposite direction and obtain the backward modeling result;

[0033] Flip the Mamba input features back to their original state in the vertical direction;

[0034] The forward modeling results and the backward modeling results are added element-wise to obtain the output features of the Mamba layer.

[0035] Furthermore, the input features of each spectrum domain information processing unit are denoted as spectrum domain input features, and the following operations are performed in each spectrum domain information processing unit:

[0036] The spectral domain input features are standardized by the first normalization layer to obtain normalized features;

[0037] The normalized features are used for preliminary feature extraction through the seventh convolutional layer to obtain preliminary features; at the same time, the normalized features are input into the fully connected layer to generate an index vector.

[0038] The preliminary features are simultaneously input into three parallel first sub-processing branches, which output the Q value, K value and V value respectively.

[0039] The Q value and K value are multiplied together and then passed through a first activation function, which is then multiplied by the V value to generate a spectral attention fusion feature;

[0040] After reshaping the spectral attention fusion features, a sparse hash mapping operation is performed on them according to the index vector to obtain real number domain features;

[0041] After the real-number domain features are converted into complex-number domain signals through the real-number-complex-number conversion layer, they are then passed through the first inverse Fourier transform layer and the eighth convolutional layer in sequence to obtain intermediate features.

[0042] The intermediate features are residually concatenated with the spectral domain input features to obtain gain fusion features;

[0043] The gain fusion features are input into two parallel second sub-processing branches through the second normalization layer and the ninth convolutional layer for spectral enhancement and fusion, respectively obtaining the upper branch output features and the lower branch output features.

[0044] After multiplying and fusing the upper branch output features and lower branch output features pixel by pixel, the combined features are obtained by passing them through the tenth convolutional layer.

[0045] The joint feature and the gain fusion feature are residually concatenated to obtain the output feature of the spectrum domain information processing unit.

[0046] Furthermore, the structures of each first sub-processing branch are consistent, each including a first depthwise separable convolutional layer, a first fast Fourier transform layer, and a complex number realization processing layer;

[0047] Local features of the preliminary features are extracted through the first depth separable convolutional layer;

[0048] The local features are transformed to the frequency domain through the first fast Fourier transform layer;

[0049] The complex number conversion layer converts the complex spectrum in the frequency domain into a real number vector.

[0050] Furthermore, the two parallel second sub-processing branches include an upper branch and a lower branch;

[0051] In the upper branch, the upper branch input features are sequentially passed through an expanded depthwise separable convolutional layer and a second fast Fourier transform layer to obtain the upper branch frequency domain representation; the upper branch frequency domain representation is multiplied pixel-by-pixel with a first frequency weight, and the product result is added to a first bias to obtain the upper branch frequency domain filtered features; the upper branch frequency domain filtered features are passed through a second inverse Fourier transform layer to recover the spatial domain, and after passing through a second activation function, the upper branch output features are obtained;

[0052] In the lower branch, the input features of the lower branch are sequentially passed through a second depthwise separable convolutional layer and a third fast Fourier transform layer to obtain the frequency domain representation of the lower branch; the frequency domain representation of the lower branch is multiplied pixel-by-pixel with the second frequency weight, and the product result is added to the second bias to obtain the frequency domain filtered features of the lower branch; the frequency domain filtered features of the lower branch are passed through a third inverse Fourier transform layer to recover the spatial domain to obtain the output features of the lower branch.

[0053] Furthermore, the step of upsampling and decoding the fused features using a segmentation head and performing pixel-level classification to output the multi-scale segmentation result corresponding to the remote sensing image specifically includes:

[0054] The fused features are progressively upsampled, with the spatial resolution doubled each time. This process is repeated until the size of the feature map is restored to the same size as the input image, resulting in the final upsampled feature map.

[0055] The upsampled feature map is concatenated with the initial image features to obtain a concatenated feature map;

[0056] The stitched feature map is then classified and mapped pixel by pixel through the eleventh convolutional layer, and the number of output channels of the eleventh convolutional layer is equal to the total number of categories.

[0057] The third activation function is used to normalize the scores of all categories at each pixel location to obtain the probability distribution of each category to which each pixel belongs. The pixel-level classification mask generated by the category with the highest probability is used as the final segmentation result.

[0058] Furthermore, the loss function of the remote sensing image segmentation model is a weighted combination of pixel-level cross-entropy loss and Dice loss.

[0059] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a multi-scale segmentation method for remote sensing images based on spectral information processing and Mamba spatial modeling, which has the following beneficial effects:

[0060] This invention takes a complementary approach from the frequency domain and spatial domain, jointly modeling the structural information, spectral features, and contextual dependencies in remote sensing images. It overcomes the technical bottlenecks of existing methods, such as blurred details, unclear boundaries, and poor multi-scale adaptability, and provides an efficient, stable, and scalable solution for high-precision segmentation of remote sensing images.

[0061] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0063] Figure 1 This is a schematic diagram illustrating the workflow of the remote sensing image segmentation model provided in an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of the workflow of the spectrum domain information processing unit provided in an embodiment of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] This invention discloses a multi-scale segmentation method for remote sensing images based on spectral information processing and Mamba spatial modeling, comprising the following steps:

[0067] The remote sensing images acquired from the target area are preprocessed to obtain the input image;

[0068] The input image is fed into the trained remote sensing image segmentation model; the remote sensing image segmentation model includes an initial convolutional layer, a spectral domain information processing unit branch, a Mamba layer branch, and a segmentation head;

[0069] The initial image features are obtained by extracting features from the input image through the initial convolutional layer;

[0070] By combining the spectral domain information processing unit branch and the Mamba layer branch, feature extraction is performed on the initial image features to obtain fused features;

[0071] The segmentation head performs upsampling decoding and pixel-level classification on the fused features, outputting multi-scale segmentation results corresponding to the remote sensing image.

[0072] The aforementioned preprocessing of remote sensing images specifically includes performing the following steps sequentially: radiometric correction, atmospheric correction, geometric correction, and noise suppression. Specifically, radiometric and atmospheric correction can eliminate amplitude deviations introduced by sensor and atmospheric factors; geometric correction can perform pixel-level alignment of multi-source images; and noise suppression can specifically employ Wiener / Lee filtering combined with edge preservation techniques to preserve ground feature boundaries.

[0073] Next, we will provide a detailed explanation of each component in the aforementioned remote sensing image segmentation model. For more details, please refer to [link / reference needed]. Figure 1 As shown.

[0074] 1. Initial convolutional layer:

[0075] This initial convolutional layer consists of multiple first convolutional layers, used to extract initial features from the standardized input image I generated after the above preprocessing, thereby obtaining initial image features. In this embodiment of the invention, four first convolutional layers are specifically included.

[0076] 2. Spectrum Domain Information Processing Unit Branch and Mamba Layer Branch:

[0077] In this embodiment of the invention, a multi-layer spectral domain information processing unit and a multi-scale Mamba spatial coding layer are integrated into the network structure to achieve deep coupling between the frequency and spatial domains. In each feature extraction stage, dual-domain features are integrated through cross-domain concatenation and residual fusion, and a highly expressive intermediate feature representation is constructed through multi-level convolutional operations. Finally, in the segmentation and decoding stage, multi-scale upsampling, skip connections, and a joint loss function are introduced to further improve the model's accuracy and stability for pixel-level classification. It exhibits significant advantages, particularly in small target recognition, boundary detail preservation, and multi-class ground feature discrimination. Specifically, the initial image features obtained above... The input is fed into the spectral domain information processing unit branch and the Mamba layer branch for in-depth feature extraction to obtain fused features. Specifically, it includes:

[0078] Initial image features The corresponding first spectral features are obtained through the first spectral domain information processing unit and the first Mamba layer, respectively. and first spatial features

[0079] The first spectral feature and first spatial features After feature concatenation, the first fused feature is obtained through a second convolutional layer.

[0080] The first fusion feature The corresponding second spectral features are obtained through the second spectral domain information processing unit and the second Mamba layer, respectively. Second spatial features

[0081] The second spectral feature Second spatial features After feature concatenation, the second fused feature is obtained through a third convolutional layer.

[0082] The second fusion feature The corresponding third spectral features are obtained through the third spectral domain information processing unit and the third Mamba layer, respectively. and third space features

[0083] The third spectral feature Third Space Features Second spectral features After feature concatenation, the third fused feature is obtained through the fourth convolutional layer.

[0084] The third fusion feature The corresponding fourth spectral features are obtained through the fourth spectral domain information processing unit and the fourth Mamba layer, respectively. and fourth space features

[0085] The fourth spectral feature Fourth Space Features and first spectral features After feature concatenation, the fourth fused feature is obtained through the fifth convolutional layer.

[0086] The fourth fusion feature The corresponding fifth spectral features are obtained through the fifth spectral domain information processing unit and the fifth Mamba layer, respectively. and the characteristics of the fifth space

[0087] The fifth spectral feature Fifth Space Characteristics and initial image features After feature concatenation, the final fused features are obtained through a sixth convolutional layer.

[0088] The aforementioned multi-level, cross-domain feature fusion process can not only ensure the complete connectivity of ground objects in remote sensing image segmentation, but also enhance the detection and segmentation capabilities of complex details and small targets; thus providing a solid multi-scale, multi-modal feature foundation for the final refined segmentation.

[0089] Next, each of the above Mamba layers and spectrum domain information processing units will be described in detail.

[0090] (1)Mamba layer:

[0091] To fully extract multi-scale spatial structure features and spectral joint patterns from remote sensing images, a multimodal attention enhancement module, called the Mamba layer, is proposed for remote sensing image segmentation tasks. This Mamba layer enhances the edge perception and region consistency representation capabilities of complex targets (such as buildings, roads, woodlands, and water bodies) in remote sensing images. Specifically, the input features of each Mamba layer are denoted as Mamba input features, and the following operations are performed in each Mamba layer:

[0092] 1) Construction of input features for the Mamba layer:

[0093] The Mamba input features are represented as Where B represents the batch size; C represents the number of channels; H and W represent the spatial dimensions of the remote sensing image; this Mamba input feature contains edge, texture, and structural line information extracted from the original remote sensing image;

[0094] 2) Channel compression and mapping:

[0095] By compressing and re-encoding the Mamba input features using 1×1 convolutions, we can highlight the response to spatially salient target regions while reducing redundant feature interference. This can be represented as follows:

[0096]

[0097] In this formula, This represents the result after channel compression and mapping of the Mamba input features; While maintaining the spatial dimensions, the channel dimensions can be dynamically adjusted, which is beneficial for subsequent attention mechanisms and state modeling modules; Conv 1×1This represents the 1×1 convolution used to compress and map the channels;

[0098] 3) Forward and Backward State Modeling (SSM):

[0099] To enhance the model's ability to model structures with continuity or directionality in remote sensing images (such as roads, water bodies, river boundaries, and land parcel boundaries), a forward and backward one-dimensional state space modeling (SSM) mechanism is designed in the Mamba layer. This mechanism can dynamically extract and integrate spatial sequence information from remote sensing images along the spatial dimension, thereby improving the segmentation model's ability to perceive linear structures and occluded targets. The specific processing flow is as follows:

[0100] First, the Mamba input features are rearranged according to the processed channel dimensions, enabling the Mamba layer to perform sequential scanning along the vertical direction (i.e., longitudinal direction) of the image. Next, one-dimensional convolutional modeling is performed based on the forward state modeling mechanism to obtain the forward modeling result. This forward one-dimensional convolutional modeling is used to simulate the propagation path of continuous targets in the image, thereby capturing long-distance dependency features, which helps to maintain the overall coherence of slender targets such as roads and water systems.

[0101] Subsequently, in order to enhance the Mamba layer's ability to recover complex situations such as occlusion, breaks, and shadows in images, a backward state modeling mechanism was added. This backward state modeling mechanism first flips the Mamba input features in the vertical direction, and then performs the same one-dimensional convolution modeling as the forward direction to obtain the context information extracted in the opposite direction and obtain the backward modeling result; then the Mamba input features are flipped back to their original state in the vertical direction.

[0102] Finally, the forward modeling results and the backward modeling results are added element by element to obtain a joint feature representation that incorporates bidirectional spatial dependency information, which is used as the output feature of the Mamba layer.

[0103] This forward and backward state modeling strategy can not only model short-term local changes in images, but also characterize long-distance structural relationships, which can significantly improve the accurate segmentation of structures such as field boundaries, roads, and canals in remote sensing images.

[0104] 4) Enhanced spatial attention:

[0105] In this embodiment of the invention, to enable the model to focus more on spatially salient regions in remote sensing images, such as dense building clusters, isolated small targets, water edges, or vegetation transition zones, a spatial attention mechanism is introduced in the Mamba layer. This spatial attention mechanism extracts local spatial structure features from the input features of the Mamba layer through a 3×3 convolution operation and generates a spatial weight map based on these local spatial structure features. This spatial weight map reflects the spatial importance of each location in the image. The Mamba layer weights its input features based on this spatial weight map, thereby enhancing the response to key regions and suppressing the influence of background or interference regions. This spatial attention mechanism helps improve the ability to distinguish target boundaries in remote sensing image segmentation and reduces missed detections on small targets, and is particularly suitable for the accurate segmentation of fine structures such as road intersections and building corners.

[0106] 5) Enhanced channel attention:

[0107] In this embodiment of the invention, considering the multi-spectral and multi-channel imaging characteristics of remote sensing images, such as infrared, visible light, panchromatic, SAR radar, NDVI, and other different band information that may carry different ground features, a channel attention mechanism is introduced in the Mamba layer. This channel attention mechanism first analyzes the contribution of each channel to the overall image understanding through a 1×1 convolution operation, and then generates a set of channel weight coefficients. These channel weight coefficients are then multiplied with the input features of the Mamba layer channel by channel, thereby realizing dynamic weighting of features of different bands or channels. The introduction of this channel attention mechanism enables the Mamba layer to automatically focus on the band that is most discriminative to the current ground feature category, improving the segmentation accuracy of complex targets (such as the boundary between farmland and forest, water bodies and shadow areas) under the condition of multi-source remote sensing data fusion.

[0108] 6) Nonlinear activation and output:

[0109] After the spatial and channel attention mechanisms mentioned above, the input features of the Mamba layer already possess strong region focusing and band discrimination capabilities. To further enhance the model's nonlinear expressive power, a nonlinear activation function (ReLU) is added to the output stage of the Mamba layer. This activation function processes the features point by point, suppressing invalid negative responses and enhancing the expressive power of key features, thus improving the final output features. This not only includes the joint modeling results of spatial structure, spectral information, and contextual semantics, but also lays a solid foundation for subsequent fusion with the output of the spectral domain information processing module. This step effectively improves the clarity of target boundaries and the consistency of regional structure in remote sensing images.

[0110] (2) Spectrum domain information processing unit:

[0111] See Figure 2 As shown in the figure, the spectrum domain information processing unit is mainly used for frequency domain analysis and fusion of input features to highlight the detailed information of the edges and textures of ground objects in the remote sensing image. This unit, as shown in the attached figure, combines spatial and frequency domain features through multi-path frequency domain transformation and attention mechanisms to enhance high-frequency details and extract global structure. Specifically, the input features of each spectrum domain information processing unit are denoted as spectrum domain input features φ, and the following operations are performed in each spectrum domain information processing unit:

[0112] 1) Data preprocessing and index vector calculation:

[0113] The input features φ in the spectral domain are standardized by the first normalization layer to obtain normalized features;

[0114] The normalized features are used for preliminary feature extraction through the seventh convolutional layer to obtain preliminary feature φ. o ; indicates as:

[0115] φ o =Conv 7 (Norm(φ))

[0116] In this formula, Conv 7 This indicates the seventh convolutional layer; Norm represents the normalization layer. This normalization + convolution processing can stabilize the feature distribution and extract basic spatial information.

[0117] At the same time, the normalized features are input into the fully connected layer to generate an index vector. Represented as:

[0118]

[0119] Here, FC represents a fully connected layer. This index vector... This is used for subsequent sparse mapping of features, indicating which feature channels or spatial locations should be prioritized in frequency domain processing (e.g., selecting important frequency bands through hash mapping) to reduce computational complexity and highlight key frequency information.

[0120] 2) Multi-branch spectral feature extraction:

[0121] Preliminary feature φ o Simultaneously, the first sub-processing branch, which operates in parallel with the input, outputs the Q, K, and V values ​​respectively. Here, Q represents the query matrix, indicating which spectral locations the current feature is interested in; K represents the key matrix, representing the feature description of each spectral location; and V represents the value matrix, representing the spectral content actually used for weighted fusion.

[0122] The structures of the aforementioned first sub-processing branches are consistent, each including a first depthwise separable convolutional layer, a first fast Fourier transform layer, and a complex number realization processing layer; specifically:

[0123] ① By using the first depthwise separable convolutional layer, local features of the preliminary features are extracted; represented as:

[0124] φ (i) =DSConv(φ o ), i∈{q,k,v}

[0125] In this formula, DSConv represents the first depthwise separable convolutional layer, which is used to preserve spatial structure and reduce computational complexity, and is suitable for high-resolution remote sensing images;

[0126] ② The local features are transformed to the frequency domain through the first Fast Fourier Transform layer; expressed as:

[0127]

[0128] In this formula, φ (i) Represents the i-th local feature; This indicates that local features are transformed into φ. (i) The result after switching to the frequency domain; FFT 1 This represents the first Fast Fourier Transform (FFT) layer; this first FFT layer transforms the input from the spatial domain to the frequency domain, enabling the model to capture long-range dependencies in the frequency dimension;

[0129] ③ By using a complex-to-real-number conversion layer (e.g., concatenating the real and imaginary parts or taking the modulus), the complex spectrum in the frequency domain is converted into a real vector for network processing; represented as:

[0130]

[0131] In this formula, Indicates will The result of converting to a real number vector; ComplexToReal represents the realization of complex numbers; because the network module cannot directly process complex numbers, it is necessary to map the complex number spectrum representation to a real number field tensor for subsequent calculations.

[0132] After obtaining the Q, K, and V values, a standard self-attention calculation method is used. This involves multiplying the Q and K values ​​and applying the result to a first activation function to calculate the correlation between spectral locations. This correlation is then used as weights to weight and combine the spectral features to generate the spectral attention fusion feature φ. 1 ; indicates as:

[0133] α = Softmax(QK) T ),φ 1 =αV

[0134] In this formula, α represents the relevance weight; Softmax represents the first activation function.

[0135] 3) Sparse mapping and spatial structure restoration:

[0136] Spectral attention fusion feature φ 1 After reshaping the shape, based on the index vector A sparse hash mapping operation is performed on it to obtain real-number domain features; specifically, in the reshaped feature tensor, the features are obtained through indexing. Selecting or rearranging a small number of key channel and spatial coefficients is equivalent to selecting important frequencies on the spectral plane for processing. This sparse mapping effectively reduces the computational complexity of the subsequent inverse Fourier transform while preserving the expressive power for the main frequency bands. Mathematically, it can be expressed as:

[0137]

[0138] In this formula, SparseMap(·) represents the operation of selecting spectral coefficients based on the index vector;

[0139] After completing the sparse mapping, the features of the real number field are processed through a real-to-complex transformation layer. Convert to the complex domain signal, and then perform the first inverse Fourier transform (iFFT) layer to recover its spatial structure: Represented as:

[0140] φ spatial =iFFT 1 (φ -1 )

[0141] Where, φ spatial Indicates the characteristics of the real number field The result after restoring its spatial structure; iFFT 1 This represents the first inverse Fourier transform layer; this process is equivalent to reconstructing the corresponding spatial feature map from the selected spectral coefficients, ensuring that high-frequency details (such as edges) are represented.

[0142] Then, the information is further integrated through the eighth convolutional layer to obtain the intermediate feature φ. 2 ; indicates as:

[0143] φ 2 =Conv 8 (φ spatial )

[0144] Among them, Conv 8 Indicates the eighth convolutional layer;

[0145] Finally, the intermediate feature φ 2 By performing a residual connection with the spectral domain input feature φ, the gain fusion feature φ is obtained.3 ; indicates as:

[0146] φ 3 =φ+φ 2

[0147] This residual connection preserves the original spatial information while introducing detailed features obtained from frequency domain recovery, which helps to alleviate gradient vanishing and maintain feature integrity.

[0148] 4) Spectrum enhancement and dual-branch fusion:

[0149] Gain fusion feature φ 3 The inputs from the second normalization layer and the ninth convolutional layer are fed into two parallel second sub-processing branches for spectral enhancement and fusion, respectively obtaining the output features of the upper branch and the output features of the lower branch.

[0150] The two parallel second sub-processing branches mentioned above include an upper branch and a lower branch; wherein:

[0151] ① In the upper branch, the input features are first passed through a dilated depthwise separable convolutional layer (DilatedDepthwiseSeparableConv) to expand the receptive field and capture a wider range of spatial context information. This dilated depthwise separable convolutional layer increases the receptive range of the convolutional kernel while maintaining the feature map size, making it very suitable for extracting features that span large objects (such as road networks or rivers) in remote sensing images. Then, the output of the dilated depthwise separable convolutional layer is transformed to the frequency domain through a second fast Fourier transform layer to obtain the upper branch frequency domain representation. The upper branch frequency domain representation is multiplied pixel-wise (at the same frequency position) by the first frequency weight w, and the product is added to the first bias B1, which is equivalent to weighted modulation of different frequency channels in the spectral domain to obtain the upper branch frequency domain filtered features. The upper branch frequency domain filtered features are then passed through a second inverse Fourier transform layer to recover the spatial domain, and after passing through a second activation function, the upper branch output features X1 are obtained; represented as:

[0152] X1 = ReLU 2 (iFFT 2 (w1·FFT 2 (Conv dilated (φ 3 ))+B1))

[0153] In this formula, ReLU 2 Indicates the second activation function; iFFT 2 Indicates the second inverse Fourier transform layer; FFT 2 Indicates the second Fast Fourier Transform layer; Conv dilated This indicates a depth-separable convolutional layer;

[0154] This process enables the network to emphasize high-frequency textures and edges at the frequency domain scale (by adjusting the frequency response corresponding to w1), while suppressing unimportant low-frequency components, thus enhancing sensitivity to textures and details. As related studies have shown, frequency domain processing helps separate edge information from noise, thereby improving the saliency of boundaries.

[0155] ② In the lower branch, the input features of the lower branch are first processed by the second depthwise separable convolutional layer to extract local information, and then transformed to the frequency domain by the third fast Fourier transform layer to obtain the frequency domain representation of the lower branch; the frequency domain representation of the lower branch is multiplied by the second frequency weights pixel by pixel w2, and the product is added to the second bias B2 to obtain the frequency domain filtered features of the lower branch; the frequency domain filtered features of the lower branch are processed by the third inverse Fourier transform layer to recover the spatial domain to obtain the output features X2 of the lower branch; expressed as:

[0156] X2 = iFFT 3 (w2·FFT 3 (Conv 2 (φ 3 ))+B2)

[0157] In this formula, iFFT 3 Indicates the third inverse Fourier transform layer; FFT 3 Indicates the third Fast Fourier Transform layer; Conv 2 This indicates a second-depth separable convolutional layer;

[0158] The goal of the lower branch is to perform another type of filtering on the frequency domain features, which complements the dilated convolution output of the upper branch, enabling the network to process spectral information from different perspectives. The upper and lower branches learn their respective spectral responses through different convolution kernels and frequency weights, ensuring that while enhancing details, the overall semantics are also taken into account.

[0159] After multiplying and fusing the upper branch output feature X1 and the lower branch output feature X2 pixel by pixel (dot multiplication is equivalent to exchanging spectral information), the result is passed through the tenth convolutional layer Conv. 10 To obtain joint features;

[0160] Combined features with gain feature φ 3 Perform residual connections to obtain the output feature φ of the spectral domain information processing unit. + ; indicates as:

[0161] φ + =φ 3 +Conv 10 (X1·X2)

[0162] This fusion method is equivalent to gated combination of frequency domain features: only when both branches are strongly represented, such as at real edges or textures, are X1 and X2 simultaneously activated, does the output achieve a larger response, thus further emphasizing the features at these locations. Adding the residuals ensures the preservation of fundamental features and avoids information loss. Overall, φ + By combining information from the spatial and frequency domains and enhancing the frequency response through multiplication operations, significant enhancements were achieved in edge and texture details.

[0163] In summary, this method innovatively constructs a dual-branch architecture: on the one hand, a spectral domain information processing unit is designed to map spatial features to the frequency domain through Fast Fourier Transform (FFT) and perform operations such as spectral attention calculation, sparse mapping, and frequency domain filtering to enhance high-frequency details and edge information in the image; on the other hand, a Mamba spatial modeling layer is introduced, which performs multi-scale fine modeling of spatial targets (such as roads, water bodies, and buildings) with continuity, directionality, and hierarchical structure through state-space modeling (SSM) and spatial-channel attention enhancement mechanisms, thereby enhancing the model's context awareness and boundary discrimination capabilities.

[0164] 3. Segmentation Head:

[0165] The segmentation head is used to decode the fused features and ultimately output the multi-scale segmentation result corresponding to the remote sensing image. The entire process mainly includes three steps:

[0166] First, the fused features are progressively upsampled, with the spatial resolution doubled each time. This process is repeated until the size of the feature map is restored to the same size as the input image of the remote sensing image segmentation model, resulting in the final upsampled feature map.

[0167] Then, the upsampled feature map is concatenated with the initial image features to obtain a concatenated feature map. The concatenation operation here is used to introduce the original spatial information while restoring the spatial resolution, thereby improving the boundary recognition and detail preservation capabilities of pixel-level classification. It aims to supplement the spatial details and boundary information contained in the shallow features, which helps to improve the accuracy of the segmentation results at the target boundary.

[0168] Finally, the concatenated feature map is input into the eleventh convolutional layer for pixel-by-pixel classification mapping. The number of output channels of the eleventh convolutional layer is equal to the total number of categories, and the convolutional layer uses a kernel size of 1×1, which is only used to convert the feature vector of each pixel position into the prediction score of the corresponding category. Then, the third activation function (softmax function) is used to normalize the scores of all categories at each pixel position to obtain the probability distribution of each category to which each pixel belongs. The pixel-level classification mask generated by the category corresponding to the highest probability is used as the final segmentation result.

[0169] In this embodiment of the invention, to balance class imbalance and boundary accuracy, a weighted combination of pixel-level cross-entropy loss and Dice loss is introduced, defining the loss function of the remote sensing image segmentation model as follows:

[0170]

[0171] in, Represents the loss function; This represents pixel-level cross-entropy loss; λ represents the Dice loss; λ represents the weighting factor.

[0172] During the training of the remote sensing image segmentation model, the above loss function Backpropagation updates network parameters on all training samples; Inference phase: Dropout / BN training mode is removed, forward computation is performed directly on the input image, and the final segmentation mask is output. By designing a segmentation head and joint loss, the boundaries and regional coherence of different land features in remote sensing images can be effectively captured, improving segmentation accuracy and robustness.

[0173] In summary, this invention provides a multi-scale segmentation method for remote sensing images based on spectral information processing and Mamba spatial modeling. Unlike traditional remote sensing image segmentation methods that only perform convolution or attention modeling in the spatial domain, this invention innovatively introduces a spectral domain information processing unit for joint spectral-spatial modeling. Spatial features are mapped to the frequency domain via Fast Fourier Transform (FFT), and attention mechanisms, filtering enhancement, and sparse reconstruction are performed in the frequency domain, effectively capturing high-frequency details and low-frequency structures in the image. By combining inverse Fourier transform and residual mechanisms, the fusion of frequency domain details and spatial backbone features is achieved, enhancing edge and texture representation capabilities and effectively improving the clarity of segmentation boundaries and the accuracy of ground feature identification.

[0174] Furthermore, the Mamba layer designed in this invention integrates forward and backward state modeling (SSM) with a spatial-channel dual attention mechanism, simulating continuous and directional structural features (such as roads, rivers, and building edges) in remote sensing images within the spatial domain. Through multi-scale compression mapping, bidirectional information transfer, and a salient region attention mechanism, it effectively enhances the recognition capabilities for occluded, small, and complex-shaped targets. Compared to existing convolutional or Transformer methods, the Mamba layer is more adaptable to the sparse structure and uneven object distribution characteristics of remote sensing images, thereby improving overall segmentation accuracy while maintaining the consistency and boundary integrity of target regions, achieving large-scale context awareness and boundary preservation.

[0175] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0176] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-scale segmentation method for remote sensing images based on spectral information processing and Mamba spatial modeling, characterized in that, Includes the following steps: The remote sensing images acquired from the target area are preprocessed to obtain the input image; The input image is fed into a trained remote sensing image segmentation model; the remote sensing image segmentation model includes an initial convolutional layer, a spectral domain information processing unit branch, a Mamba layer branch, and a segmentation head; The input image is used to extract features through the initial convolutional layer to obtain initial image features; The spectral domain information processing unit branch and the Mamba layer branch are combined to extract features from the initial image features and obtain fused features. The fused features are upsampled, decoded, and classified at the pixel level by the segmentation head, and the multi-scale segmentation result corresponding to the remote sensing image is output. The step of combining the spectral domain information processing unit branch and the Mamba layer branch to extract features from the initial image features and obtain fused features specifically includes: The initial image features are processed by the first spectral domain information processing unit and the first Mamba layer to obtain the corresponding first spectral features and first spatial features. The first spectral feature and the first spatial feature are concatenated and then passed through a second convolutional layer to obtain the first fused feature; The first fusion feature is processed by the second spectral domain information processing unit and the second Mamba layer to obtain the corresponding second spectral feature and second spatial feature. The second spectral feature and the second spatial feature are concatenated and then passed through a third convolutional layer to obtain the second fused feature; The second fusion feature is processed by the third spectral domain information processing unit and the third Mamba layer to obtain the corresponding third spectral feature and third spatial feature. The third spectral feature, the third spatial feature, and the second spectral feature are concatenated and then passed through a fourth convolutional layer to obtain the third fused feature. The third fusion feature is processed by the fourth spectral domain information processing unit and the fourth Mamba layer to obtain the corresponding fourth spectral feature and fourth spatial feature. The fourth spectral feature, the fourth spatial feature, and the first spectral feature are concatenated and then passed through a fifth convolutional layer to obtain the fourth fused feature. The fourth fusion feature is processed by the fifth spectral domain information processing unit and the fifth Mamba layer to obtain the corresponding fifth spectral feature and fifth spatial feature. The fifth spectral feature, the fifth spatial feature, and the initial image feature are concatenated and then passed through a sixth convolutional layer to obtain the final fused feature.

2. The remote sensing image multi-scale segmentation method based on spectral information processing and Mamba spatial modeling according to claim 1, characterized in that, The preprocessing includes: radiometric correction, atmospheric correction, geometric correction, and noise suppression.

3. The remote sensing image multi-scale segmentation method based on spectral information processing and Mamba spatial modeling according to claim 1, characterized in that, The initial convolutional layer consists of multiple first convolutional layers.

4. The remote sensing image multi-scale segmentation method based on spectral information processing and Mamba spatial modeling according to claim 1, characterized in that, The input features of each Mamba layer are denoted as Mamba input features, and the following operations are performed in each Mamba layer: The channels of the Mamba input features are compressed and mapped; The Mamba input features are rearranged according to the processed channel dimensions so that the Mamba layer can perform sequential scanning along the vertical direction of the image. One-dimensional convolutional modeling is performed based on the forward state modeling mechanism to capture long-distance dependency features and obtain forward modeling results; Flip the Mamba input features in the vertical direction; One-dimensional convolution modeling is performed based on the backward state modeling mechanism to obtain contextual information extracted in the opposite direction and obtain the backward modeling result; Flip the Mamba input features back to their original state in the vertical direction; The forward modeling results and the backward modeling results are added element-wise to obtain the output features of the Mamba layer.

5. The remote sensing image multi-scale segmentation method based on spectral information processing and Mamba spatial modeling according to claim 1, characterized in that, The input features of each spectrum domain information processing unit are denoted as spectrum domain input features, and the following operations are performed in each spectrum domain information processing unit: The spectral domain input features are standardized by the first normalization layer to obtain normalized features; The normalized features are used for preliminary feature extraction through the seventh convolutional layer to obtain preliminary features; at the same time, the normalized features are input into the fully connected layer to generate an index vector. The preliminary features are simultaneously input into three parallel first sub-processing branches, which output the Q value, K value and V value respectively. The Q value and K value are multiplied together and then passed through a first activation function, which is then multiplied by the V value to generate a spectral attention fusion feature; After reshaping the spectral attention fusion features, a sparse hash mapping operation is performed on them according to the index vector to obtain real number domain features; After the real-number domain features are converted into complex-number domain signals through the real-number-complex-number conversion layer, they are then passed through the first inverse Fourier transform layer and the eighth convolutional layer in sequence to obtain intermediate features. The intermediate features are residually concatenated with the spectral domain input features to obtain gain fusion features; The gain fusion features are input into two parallel second sub-processing branches through the second normalization layer and the ninth convolutional layer for spectral enhancement and fusion, respectively obtaining the upper branch output features and the lower branch output features. After multiplying and fusing the upper branch output features and lower branch output features pixel by pixel, the combined features are obtained by passing them through the tenth convolutional layer. The joint feature and the gain fusion feature are residually concatenated to obtain the output feature of the spectrum domain information processing unit.

6. The remote sensing image multi-scale segmentation method based on spectral information processing and Mamba spatial modeling according to claim 5, characterized in that, Each of the first sub-processing branches has the same structure, including a first depthwise separable convolutional layer, a first fast Fourier transform layer, and a complex number realization processing layer; Local features of the preliminary features are extracted through the first depth separable convolutional layer; The local features are transformed to the frequency domain through the first fast Fourier transform layer; The complex number conversion layer converts the complex spectrum in the frequency domain into a real number vector.

7. The remote sensing image multi-scale segmentation method based on spectral information processing and Mamba spatial modeling according to claim 5, characterized in that, The two parallel second sub-processing branches include an upper branch and a lower branch; In the upper branch, the upper branch input features are sequentially passed through an expanded depthwise separable convolutional layer and a second fast Fourier transform layer to obtain the upper branch frequency domain representation; The upper branch frequency domain representation is multiplied pixel-by-pixel by the first frequency weight, and the product is added to the first bias to obtain the upper branch frequency domain filtering feature; the upper branch frequency domain filtering feature is restored to the spatial domain by the second inverse Fourier transform layer, and after passing through the second activation function, the upper branch output feature is obtained. In the lower branch, the input features of the lower branch are sequentially passed through a second depthwise separable convolutional layer and a third fast Fourier transform layer to obtain the frequency domain representation of the lower branch; The lower branch frequency domain representation is multiplied pixel by pixel with the second frequency weight, and the product result is added to the second bias to obtain the lower branch frequency domain filtering feature; The lower branch frequency domain filtering features are then processed through a third inverse Fourier transform layer to recover the spatial domain, thereby obtaining the lower branch output features.

8. The remote sensing image multi-scale segmentation method based on spectral information processing and Mamba spatial modeling according to claim 1, characterized in that, The step involves upsampling and decoding the fused features using a segmentation head, performing pixel-level classification, and outputting a multi-scale segmentation result corresponding to the remote sensing image; specifically including: The fused features are progressively upsampled, with the spatial resolution doubled each time. This process is repeated until the size of the feature map is restored to the same size as the input image, resulting in the final upsampled feature map. The upsampled feature map is concatenated with the initial image features to obtain a concatenated feature map; The stitched feature map is then classified and mapped pixel by pixel through the eleventh convolutional layer, and the number of output channels of the eleventh convolutional layer is equal to the total number of categories. The third activation function is used to normalize the scores of all categories at each pixel location to obtain the probability distribution of each category to which each pixel belongs. The pixel-level classification mask generated by the category with the highest probability is used as the final segmentation result.

9. The remote sensing image multi-scale segmentation method based on spectral information processing and Mamba spatial modeling according to claim 1, characterized in that, The loss function of the remote sensing image segmentation model is a weighted combination of pixel-level cross-entropy loss and Dice loss.

Images