Remote sensing image semantic segmentation method based on Mama and Transform architecture fusion

Abstract

The invention discloses a remote sensing image semantic segmentation method based on Mama and Transform architecture fusion. The method comprises the following steps: S1, data preprocessing; s2, pre-training the model by using the MAE; s3, preliminary feature extraction: performing preliminary feature extraction on the image subjected to MAE processing through linear projection, a convolutional layer and a nonlinear activation function to obtain a low-dimensional feature map; s4, CFF feature fusion: fusing the low-dimensional feature map and the image after MAE processing by using a CFF module; s5, feature enhancement of a Mamb-T module: inputting the feature map of the fused image into the Mamb-T module for operation to obtain further feature enhancement of the image; s6, multi-scale feature extraction; s7, a segmentation module, wherein the feature map is input to a Mask2Former segmentation module for final image segmentation; and S8, outputting and post-processing. According to the method, local and global features in the image can be captured, and the segmentation problem under a small water body and a complex background is effectively solved through combination of a self-adaptive attention mechanism and convolution operation.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image semantic segmentation, and in particular relates to a remote sensing image semantic segmentation method based on a Mamba structure encoder and a Transformer structure decoder. Background Art

[0002] With the development of remote sensing technology, high-resolution remote sensing images have been widely used in water resources management, environmental monitoring, and disaster warning. However, existing remote sensing image processing technologies still face many challenges in the accurate extraction of water bodies. In particular, when faced with small water bodies and complex backgrounds (such as vegetation and boundary areas), the accuracy and efficiency of existing methods are often insufficient.

[0003] At present, most mainstream water extraction technologies are based on convolutional neural networks (CNNs) in deep learning, which extract water features from images by performing multi-layer convolution processing on them. Although such methods have achieved good results in large-scale water extraction, they often perform poorly when dealing with small water bodies. The reason is that the locality of the convolution operation makes it difficult to capture contextual dependencies across long distances. In addition, traditional convolutional neural networks are prone to mis-segmentation when faced with complex backgrounds and noise, especially in areas with high-reflectivity objects or vegetation. The boundaries of water bodies are blurred, affecting the accuracy of segmentation results.

[0004] In addition, due to the large amount of high-resolution remote sensing image data and high computational complexity, existing technologies are inefficient in processing such data. Especially in scenarios that require real-time processing, existing methods are unable to meet the needs and take into account computational efficiency while ensuring segmentation accuracy. To overcome this problem, researchers have tried a variety of methods, including introducing mixed pooling modules and multi-scale convolution operations, but these methods still have a large trade-off between accuracy and computational complexity.

[0005] In recent years, the Transformer model has been introduced into the field of image processing due to its success in the field of natural language processing. The self-attention mechanism of the Transformer model can effectively capture global contextual information, making up for the shortcomings of convolutional neural networks in modeling long-distance dependencies. However, the Transformer model usually requires a lot of computing resources, and its high computing cost makes it challenging to process large-scale remote sensing images. Summary of the invention

[0006] In view of the above problems, the present invention introduces the combination of Mamba module and Transformer, and proposes a remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture. This method can not only capture local and global features in the image, but also effectively solve the segmentation problem of small water bodies and complex backgrounds through the combination of adaptive attention mechanism and convolution operation. At the same time, by introducing multi-scale feature extraction and hybrid pooling modules, the processing efficiency is greatly improved, which is particularly suitable for water body segmentation tasks of high-resolution remote sensing images.

[0007] The technical solution adopted by the present invention is: a remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture, comprising the following steps:

[0008] S1. Data preprocessing: preprocessing the input high-resolution remote sensing images;

[0009] S2.MAE pre-trained model, the pre-processed image is input into the MAE pre-trained model, the MAE pre-trained model masks a part of the input image, and then uses a self-supervised learning method to reconstruct the image;

[0010] S3. Preliminary feature extraction, the image after MAE processing is subjected to preliminary feature extraction through linear projection, convolution layer, and nonlinear activation function to obtain a low-dimensional feature map;

[0011] S4.CFF feature fusion, the low-dimensional feature map and the image processed by MAE are fused using the CFF module;

[0012] S5. Feature enhancement of Mamba-T module: the feature map of the fused image is input into the Mamba-T module for standardization, Mamba Block feature extraction, and CA attention module operation to obtain further feature enhancement of the image;

[0013] S6. Multi-scale feature extraction, using convolution kernels with various void ratios to process feature maps to obtain multi-scale information;

[0014] S7. Segmentation module. After feature extraction, the feature map is input into the segmentation module for final image segmentation. This module captures global information through the self-attention mechanism of Transformer and generates the final segmentation mask.

[0015] S8. Output and post-processing.

[0016] Based on the above solution, as a preferred embodiment, normalization, data enhancement and noise removal operations are performed in step S1 to improve the quality of the image, reduce the impact of noise on subsequent feature extraction, and enhance the generalization ability of the model.

[0017] (1) Normalization: Through normalization, all pixel values ​​are scaled to the range of [0, 1], thereby reducing the numerical differences between different images and improving the adaptability of the model to the input image.

[0018]

[0019] Where: X is the original input pixel value, X min and X max are the minimum and maximum pixel values ​​in the original image, respectively, norm is the normalized pixel value,

[0020] (2) Data enhancement: By rotating and flipping images, we can simulate the changes of remote sensing images at different angles and scales, and improve the robustness of the model to uncertain factors in actual scenes.

[0021] Rotation formula:

[0022]

[0023] Flip formula:

[0024] X filp (i, j) = X orig (i, W–j-1)

[0025] (3) Noise removal: Gaussian filtering is used to remove noise and reduce the interference in the extraction of water areas. The formula is as follows:

[0026]

[0027] Based on the above scheme, as a preferred embodiment, in step S2, some parts of the input image X are randomly selected for masking, and the mask ratio is 75%. The specific masking operation is to randomly select a part of the pixel area in the image and set these areas to zero, and define a mask matrix M, where M(i, j)=1 indicates that the pixel at the position (i, j) is not masked, and M(i, j)=0 indicates that the pixel at the position is masked. The masked image X masked It is expressed as:

[0028] X masked (i, j) = M (i, j) * X (i, j)

[0029] Where: X(i, j) is the pixel value of the input image at position (i, j), M(i, j) is the value in the mask matrix, which is 0 or 1,

[0030] Then perform MAE encoding and decoding:

[0031] Z encoded=f encoder (X masked )

[0032] X reconstructed =f decoder (Z encoded )

[0033] Among them, f encoder is the network function of the encoder, X masked is the input image after mask processing; f decoder is the network function of the decoder. The goal of the decoder is to reconstruct the image X reconstructed As close as possible to the original image X.

[0034] Based on the above solution, as a preferred method, in step S3, the Z output by MAE pre-training is then encoded Image features are used as input for feature extraction, including linear projection, convolution operation and nonlinear activation function. The projection operation calculation formula is:

[0035] Z=W*X+b

[0036] Where: X is the pixel value matrix of the input image, W is the weight matrix of the linear transformation, b is the bias term, and Z is the feature representation after projection. The high-dimensional pixel space of the original image is mapped to a lower-dimensional feature space through the weight matrix W and the bias term b, which helps to capture the local spatial structure in the image and provide more effective input features for the convolution operation.

[0037] Convolution operation is performed to capture the local features of different areas in the image, and then the nonlinear expression ability of the model is enhanced through nonlinear activation function. SiLU is used as the nonlinear activation function, and its formula is:

[0038] SiLU(x)=x*σ(x)

[0039] Where σ(x) is the Sigmoid function:

[0040]

[0041] Based on the above scheme, as a preferred embodiment, in step S4, the CFF module is used to fuse the feature maps from the MAE pre-training and the two different levels of projection, convolution, and activation functions to ensure that the network can obtain more context information from image information of different scales. The two feature maps are set as F 1 and F 2 , then the calculation formula of feature fusion is:

[0042] F fused =α*F 1 +β*F 2

[0043] Among them, α and β are learnable weight coefficients that control the weights of features at different layers. The weights α and β are not fixed, but are obtained through network adaptive learning. This adaptive allocation mechanism can dynamically adjust the weights of different feature layers, so that the network can self-optimize according to the characteristics of different images. The calculation formula is as follows:

[0044]

[0045]

[0046] Among them: [F 1 , F 2 ] is to put F 1 and F 2 After splicing, W α and W β is the learned weight matrix, b α and b β is the bias term, σ is the Sigmoid activation function, ensuring that the values ​​of α and β are in the range of [0, 1]. After being processed by the CFF module, the fused feature map F fused It will be used as input to the subsequent Mamba-T module to further enhance the effect of water extraction.

[0047] Based on the above scheme, as a preferred embodiment, in step S5, four Mamba-T Block network modules are used for encoding, and the input is processed at different resolutions from H / 4×W / 4×C to H / 32×W / 32×8C. As the depth of the network increases, the size of the feature map gradually decreases, while the number of channels gradually increases. Then, the output feature map is sent to the CA attention module.

[0048] Mamba-T Block first performs Layer Normalization (LN) layer normalization operation, which normalizes the input features to ensure that the numerical range of the feature map remains stable at each layer. The normalization calculation formula is:

[0049]

[0050] Where: X i is the i-th feature of the input feature, μ is the mean of the feature map, σ 2 is the variance of the feature map, γ and β are learnable scaling and offset parameters, ∈ is a small constant to prevent division by zero errors,

[0051] Secondly, the core Mamba Block module of Mamba-T Block is used for feature extraction, including linear projection, convolution operation, activation function (SiLU), SSM (Self-Similarity Module) and nonlinear processing. The specific steps are as follows: the layer-normalized image is divided into two paths for operation. The first path is linear projection-convolution operation-SiLU activation function-SSM, and the second path is linear projection-SiLU activation function. Then the two paths are output to the ARC module for fusion after nonlinear processing and linear projection. The SSM module is used to enhance the self-similarity of the feature map, which is achieved by comparing the similarities of different regions in the feature map.

[0052] SSM(F)=similarity_function(F, F)

[0053] The ARC module is used to further refine and combine the features from the multi-head self-attention mechanism. By dynamically adjusting the weights of the attention scores, the features from different channels are further integrated to improve the perception of the boundaries and internal areas of the water body. The adaptive weight calculation formula of ARC is as follows:

[0054]

[0055]

[0056] Among them: F attention is a feature from the Mamba layer, F LN is the feature from the layer normalization operation, W α and W β is the learned weight matrix, b α and b β is the bias term, σ is the Sigmoid activation function, ensuring that the values ​​of α and β are in the range of [0, 1]. Then, the attention features and convolution features are weighted and combined according to the weights:

[0057] F ARC =α*F attention +β*F LN

[0058] The ARC module achieves flexible fusion of different feature maps by adaptively adjusting the weights of different features. This module ensures that the model can pay attention to both the boundary details and the global structure of the water area.

[0059] After the ARC module, the feature map enters the MLP for further nonlinear mapping of the features. The MLP maps the high-dimensional features to the target output space through multiple fully connected layers. The MLP calculation formula is as follows:

[0060] F MLP =σ(W 2 *(σ(W 1 *F+b 1 ))+b 2 )

[0061] Where: F is the feature output by the ARC module, W1 and W2 are the weight matrices of the two-layer MLP, b1 and b2 are bias terms, σ is the activation function, SiLU is selected,

[0062] After MLP, it enters the ARC module again to fuse the features from MLP and previous layers, and outputs the refined final features to enter the CA channel attention module.

[0063] The CA (Channel Attention) mechanism is used to dynamically adjust the weight of each channel, enhance the model's attention to key water areas, and perform global average pooling:

[0064]

[0065] Where: G c is the global eigenvalue of channel c, X c (i, j) is the pixel value of channel c at pixel position (i, j), H and W are the height and width of the feature map respectively,

[0066] Attention weight calculation:

[0067] α c =σ(W c *G c +b c )

[0068] Where: α c is the attention weight of channel c, G c is the global eigenvalue of channel c, W c is the weight matrix of the fully connected layer, b c is the bias term, σ is the Sigmoid activation function, which is used to limit the attention weight to the range of [0, 1].

[0069] Channel weighting:

[0070] X′ c (i, j) = α c X c (i, j)

[0071] Where: X′ c (i, j) is the weighted pixel value of channel c at position (i, j), α c is the attention weight of the channel X c(i, j) is the pixel value of the original channel c.

[0072] Based on the above solution, preferably, in step S6, convolution kernels with various void ratios are used to process the feature map, so as to obtain multi-scale information, as follows:

[0073] 1×1 convolution: used to extract global information and maintain image detail features.

[0074] 3×3 dilated convolution (dilation rate = 6): used to capture medium-range water features.

[0075] 3×3 dilated convolution (dilation rate = 12): used to capture a larger range of water features.

[0076] 3×3 dilated convolution (dilation rate = 18): used to capture water features in a wider range.

[0077] Then various convolutions are calculated:

[0078]

[0079] Where: W r is a convolution kernel with different void rates r, X is the input feature map, Y is the output feature map after multi-scale convolution, and then enters the image pooling to further expand the receptive field of the network and help the model to globally perceive the context information of the entire image.

[0080] The MPM module sets different pooling window sizes s 1 ,s 2 ,…,s n The input feature map is pooled. Different window sizes can capture feature information of different scales, which is especially suitable for capturing water areas of different sizes. The specific calculation formula is as follows:

[0081]

[0082] Where: X is the input feature map, s i is the pooling window size of different scales, pooling(X, s i ) represents the pooling operation, and Z is the output feature map after multi-scale pooling.

[0083] Based on the above solution, as a preferred embodiment, in step S7, after completing the early multi-level feature extraction, the segmentation module performs image segmentation through the Mask2Transformer technology.

[0084] First, the feature map is input into a multi-layer Transformer encoder, which captures the long-distance dependencies in the image through a multi-head self-attention mechanism. The calculation steps of the self-attention mechanism are:

[0085] (1) Calculation of query, key, and value:

[0086] Q=XW Q , K = XW K , V = XW V

[0087] Where: X is the input feature map, W Q , W K , W V is the linear transformation matrix of query, key and value,

[0088] (2) Self-attention score calculation:

[0089]

[0090] Where: Q is the query matrix, K is the key matrix, V is the value matrix, d k is the dimension of the key matrix (used for scaling and normalization), softmax is used to normalize the attention weights,

[0091] (3) Multi-head attention mechanism:

[0092] MultiHead(Q,K,V)=Concat(head 1 , …, head h )W O

[0093] Where: h is the number of attention heads, each attention head processes different feature dimensions, W O is the final linear transformation weight matrix,

[0094] After completing the Transformer encoding, the generated feature map will be used to generate the final segmentation mask. The mask generation formula is:

[0095] M mask (i, j) = sigmoid(W m *F encoded (i, j)+b m )

[0096] Among them: F encoded (i, j) is the feature of the encoded feature map at position (i, j), W m and b m are the weights and biases of the mask generation layer, M mask(i, j) is the final generated water body segmentation mask, which indicates the probability that the pixel position belongs to the water body area. By binarizing each pixel, the probability map can be converted into a binary segmentation mask, where "1" represents the water body area and "0" represents the non-water body area.

[0097] Based on the above scheme, as a preferred embodiment, in step S8, after the segmentation mask is generated by Mask2Transformer, the last step is post-processing to improve the accuracy and consistency of the segmentation result. This step includes edge smoothing and morphological operations. First, smoothing is performed by Gaussian filtering:

[0098]

[0099] Then a dilation operation is performed to expand the boundaries of the water area and fill in small holes in the segmentation:

[0100]

[0101] Where: M mask is the original mask, B is the structural element, is the expansion operation,

[0102] Finally, the corrosion operation is performed to remove noise, shrink the segmentation boundary, and remove the mis-segmentation of non-water areas:

[0103]

[0104] Where: M mask is the original mask, B is the structural element, is a corrosion operation,

[0105] Through the combination of smoothing, dilation and erosion, the boundary of the segmentation result becomes smoother, ensuring that the final segmentation mask is more accurate.

[0106] Compared with the prior art, the beneficial effect of the present invention is that the present invention not only improves the accuracy in the water body extraction task, but also achieves a breakthrough in computing efficiency. It is suitable for large-scale remote sensing image processing scenarios and has a wide range of application potentials. The specific benefits are as follows:

[0107] (1) Significantly improve the accuracy of water extraction

[0108] Through the multi-level convolution and self-attention mechanism of the Mamba module, the present invention shows higher accuracy when processing complex backgrounds and small water bodies. The Mamba-T module can accurately capture the boundary features of water bodies, and use the ARC module to adaptively fuse features of different scales, making the boundaries of water bodies clearer, especially in the extraction of small water bodies. Compared with traditional convolutional neural networks, the present invention significantly reduces the phenomenon of mis-segmentation and missed segmentation in water body extraction tasks.

[0109] (2) Improve computing efficiency and be suitable for real-time processing

[0110] By combining the multi-scale hybrid pooling module (MPM) and Mask2Transformer, the present invention not only improves the accuracy of segmentation, but also greatly improves the processing efficiency. The multi-scale feature extraction module effectively reduces the computational complexity of the network, enabling the method to achieve fast processing on high-resolution and large-scale remote sensing images, meeting the scene requirements with high real-time requirements in practical applications.

[0111] (3) Adapt to water extraction at different scales

[0112] The existing technology often has difficulty in taking into account both large-scale water bodies and small water bodies. The present invention can simultaneously process water areas of different scales through the design of multi-scale convolution and mixed pooling modules. Whether it is a large lake or river or a small water area, the present invention can effectively extract it and ensure the integrity and consistency of the segmentation results.

[0113] (4) Enhanced robustness to complex background and noise

[0114] In remote sensing images, complex backgrounds and noise often interfere with the extraction of water bodies. By modeling the self-similarity of images through the SSM module, the present invention improves the ability to distinguish complex backgrounds, effectively filters the interference of background noise, and reduces the phenomenon of mis-segmentation. Even in areas with lush vegetation or dense buildings, the accuracy of water body extraction can still be guaranteed.

[0115] (5) Modular design facilitates expansion and optimization

[0116] The present invention adopts a modular design concept, and each functional module such as Mamba-T, ARC and Mask2Transformer can be flexibly combined and adjusted according to different application requirements. This modular design makes the method have good scalability and is easy to apply and optimize in other semantic segmentation or feature extraction tasks.

[0117] (6) Broad application prospects

[0118] Since the invention can effectively extract water information from high-resolution remote sensing images, especially the ability to process small water areas and complex backgrounds, it has a wide range of application prospects. It can be applied to water resources management, environmental monitoring, disaster warning, urban planning and other fields, especially providing strong technical support for tasks such as flood warning and ecological environment protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 is a flow chart of the present invention;

[0120] Figure 2 is a network structure diagram of the present invention;

[0121] Figure 3 This is a module diagram of the Mamba-T Block method of the present invention.

[0122] Figure 4 It is the module diagram of Mamba Block of the method of the present invention;

[0123] Figure 5 It is the evaluation index of the test set under various models;

[0124] Figure 6 is the loss function of the training set under various models;

[0125] Figure 7 is the accuracy of the training set under various models;

[0126] Figure 8 This is a comparison chart of water image segmentation results under various models. DETAILED DESCRIPTION

[0127] The present invention and the retrograde operation are further described below in conjunction with the accompanying drawings in the examples of the present invention.

[0128] like Figure 1-7 As shown, a remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture includes the following steps:

[0129] S1. Data preprocessing. First, the input high-resolution remote sensing image undergoes basic preprocessing operations, including image normalization, data enhancement (such as rotation, flipping, etc.), and noise removal. The purpose of normalization is to make the pixel values ​​of the image within the optimal range of network processing, data enhancement is used to increase the generalization ability of the model, and noise removal is to remove high-frequency noise in the image through methods such as median filtering or Gaussian filtering, so as to avoid interference with subsequent feature extraction.

[0130] S2.MAE pre-trained model. In the preliminary feature extraction process, the present invention uses the MAE (Masked Autoencoders) pre-trained model to enhance the model's feature learning ability. The MAE model masks part of the input image and then uses a self-supervised learning method to reconstruct the image. This method enables the model to learn the potential structural information in remote sensing images without labeled data, especially for the feature expression of water areas.

[0131] S3. Preliminary feature extraction. The preprocessed image is first subjected to preliminary feature extraction through linear projection, convolutional layer, nonlinear activation function, etc. to obtain a low-dimensional feature map.

[0132] S4.CFF feature fusion. In the feature extraction process, the CFF module is used to achieve efficient fusion between feature maps at different levels. The CFF module dynamically combines feature maps from different layers through a cross-level feature fusion mechanism to ensure that feature information at different scales can be effectively complementary and improve the accuracy of water segmentation.

[0133] S5. Feature enhancement of Mamba-T module

[0134] After preliminary feature extraction, the feature map of the image is input into the Mamba-T module for normalization, MambaBlock feature extraction, ARC channel attention and other operations to obtain further feature enhancement of the image.

[0135] S6. Multi-scale feature extraction. In order to process water areas at different scales, a multi-scale feature extraction module is introduced. This module combines convolution operations of different scales and mixed pooling technology to further enhance the network's perception of features of different scales, ensuring that the network can perceive both small water areas and large water areas at the same time.

[0136] S7. Mask2Transformer segmentation module. After feature extraction, the feature map is input into the Mask2Transformer module for final image segmentation. This module captures global information through the self-attention mechanism of the Transformer and generates the final segmentation mask.

[0137] S8. Output and post-processing. After the segmentation mask is generated, post-processing operations are performed to improve the accuracy and consistency of the segmentation results, including edge smoothing and morphological operations to ensure clear boundaries of water areas.

[0138] In the above step S1, normalization, data enhancement and noise removal operations are performed to improve the quality of the image, reduce the impact of noise on subsequent feature extraction, and enhance the generalization ability of the model.

[0139] (1) Normalization: Through normalization, all pixel values ​​are scaled to the range of [0, 1], thereby reducing the numerical differences between different images and improving the adaptability of the model to the input image.

[0140]

[0141] Where: X is the original input pixel value, X min and X max are the minimum and maximum pixel values ​​in the original image, respectively, norm is the normalized pixel value.

[0142] (2) Data enhancement: By rotating and flipping images, we can simulate the changes of remote sensing images at different angles and scales, thereby improving the robustness of the model to uncertain factors in actual scenes.

[0143] Rotation formula:

[0144]

[0145] Flip formula:

[0146] X filp (i, j) = X orig (i, W–j-1)

[0147] (3) Noise removal: Remove noise through Gaussian filtering to reduce the interference of water area extraction. The formula is as follows:

[0148]

[0149] In step S2, the MAE pre-training model is first introduced. MAE is a self-supervised learning model. One of the core ideas is to randomly mask part of the input image, thereby forcing the network to learn the global structure of the image. Some parts of the input image X are randomly selected for masking, and the mask ratio is 75%. The specific masking operation is to randomly select a part of the pixel area in the image and set these areas to zero. Define the mask matrix M, where M(i, j) = 1 means that the pixel at position (i, j) is not masked, and M(i, j) = 0 means that the pixel at this position is masked. The masked image X masked It is expressed as:

[0150] X masked (i, j) = M (i, j) * X (i, j)

[0151] Where: X(i, j) is the pixel value of the input image at position (i, j), and M(i, j) is the value in the mask matrix (0 or 1).

[0152] Then perform MAE encoding and decoding:

[0153] Z encoded =f encoder (X masked )

[0154] X reconstructed =f decoder (Z encoded )

[0155] Among them, f encoder is the network function of the encoder, X masked is the input image after mask processing; f decoder is the network function of the decoder. The goal of the decoder is to reconstruct the image X reconstructed As close as possible to the original image X.

[0156] In step S3, the MAE pre-trained output Z encoded Image features are used as input for feature extraction, which mainly involves linear projection, convolution operations and nonlinear activation functions. The projection operation calculation formula is:

[0157] Z=W*X+b

[0158] Where: X is the pixel value matrix of the input image, W is the weight matrix of the linear transformation, b is the bias term, and Z is the feature representation after projection. This operation maps the high-dimensional pixel space of the original image to a lower-dimensional feature space through the weight matrix W and the bias term b. This transformation helps capture the local spatial structure in the image and provides more effective input features for the convolution operation.

[0159] Furthermore, a convolution operation is performed to capture the local features of different regions in the image. The nonlinear expression ability of the model is then enhanced by a nonlinear activation function. In the present invention, SiLU is used as a nonlinear activation function, and its formula is:

[0160] SiLU(x)=x*σ(x)

[0161] Where σ(x) is the Sigmoid function:

[0162]

[0163] In step S4, the CFF module is used to fuse the feature maps from the MAE pre-training and the convolution, projection, and activation functions at two different levels to ensure that the network can obtain more contextual information from image information of different scales. 1 and F 2 , then the calculation formula of feature fusion is:

[0164] F fused =α*F 1+β*F 2

[0165] Among them, α and β are learnable weight coefficients that control the weights of features at different layers. The weights α and β are not fixed, but are learned through network adaptive learning. This adaptive allocation mechanism can dynamically adjust the weights of different feature layers, allowing the network to self-optimize according to the characteristics of different images. The calculation formula is as follows:

[0166]

[0167]

[0168] Among them: [F 1 , F 2 ] is to put F 1 and F 2 After splicing, W α and W β is the learned weight matrix, b α and b β is the bias term, σ is the Sigmoid activation function, ensuring that the values ​​of α and β are in the range of [0, 1]. After being processed by the CFF module, the fused feature map F fused It will be used as input to the subsequent Mamba-T module to further enhance the effect of water extraction.

[0169] In step S5, four Mamba-T Block network modules are mainly used for encoding, which process the input at different resolutions (H / 4×W / 4×C to H / 32×W / 32×8C). As the depth of the network increases, the size of the feature map gradually decreases, while the number of channels gradually increases. The output feature map is then sent to the CA attention module. Specifically, Mamba-T Block first performs a Layer Normalization (LN) layer normalization operation, which normalizes the input features to ensure that the numerical range of the feature map remains stable at each layer. Normalization calculation formula:

[0170]

[0171] Where: X i is the i-th feature of the input feature, μ is the mean of the feature map, σ 2 is the variance of the feature map, γ and β are learnable scaling and offset parameters, and ∈ is a small constant to prevent division by zero errors.

[0172] Secondly, the core Mamba Block module of Mamba-T Block is used for feature extraction, including linear projection, convolution operation, activation function (SiLU), SSM (Self-Similarity Module) and nonlinear processing. Specifically, the layer-normalized image is divided into two paths for operation. The first path is linear projection-convolution operation-SiLU activation function-SSM, and the second path is linear projection-SiLU activation function. Then the two paths are output to the ARC module for fusion after nonlinear processing and linear projection. The SSM module is used to enhance the self-similarity of the feature map, which is achieved by comparing the similarity of different regions in the feature map.

[0173] SSM(F)=similarity_function(F, F)

[0174] The ARC module is used to further refine and combine features from the multi-head self-attention mechanism. It further integrates features from different channels by dynamically adjusting the weights of the attention scores, thereby improving the perception of water body boundaries and internal areas. The adaptive weight calculation formula of ARC is as follows:

[0175]

[0176]

[0177] Among them: F attention is a feature from the Mamba layer, F LN is the feature from the layer normalization operation, W α and W β is the learned weight matrix, b α and b β is the bias term, σ is the Sigmoid activation function, ensuring that the values ​​of α and β are in the range of [0, 1]. Then, the attention features and convolution features are weighted and combined according to the weights:

[0178] F ARC =α*F attention +β*F LN

[0179] The ARC module achieves flexible fusion of different feature maps by adaptively adjusting the weights of different features. This module ensures that the model can pay attention to both the boundary details and the global structure of the water area.

[0180] After the ARC module, the feature map enters the MLP (Multi-layer Perceptron) for further nonlinear mapping of features. MLP maps high-dimensional features to the target output space through multiple fully connected layers. The calculation formula of MLP is as follows:

[0181] F MLP =σ(W 2 *(σ(W 1 *F+b 1 ))+b 2 )

[0182] Among them: F is the feature output by the ARC module, W1 and W2 are the weight matrices of the two-layer MLP, b1 and b2 are bias terms, and σ is the activation function, SiLU is selected here.

[0183] After MLP, it enters the ARC module again to fuse the features from MLP and previous layers, and outputs the refined final features to enter the CA channel attention module.

[0184] The CA (Channel Attention) mechanism is used to dynamically adjust the weight of each channel to enhance the model's attention to key water areas. Global average pooling:

[0185]

[0186] Where: G c is the global eigenvalue of channel c, X c (i, j) is the pixel value of channel c at pixel position (i, j), and H and W are the height and width of the feature map, respectively.

[0187] Attention weight calculation:

[0188] α c =σ(W c *G c +b c )

[0189] Where: α c is the attention weight of channel c, G c is the global eigenvalue of channel c, W c is the weight matrix of the fully connected layer, b c is the bias term, and σ is the Sigmoid activation function, which is used to limit the attention weight to the range of [0, 1].

[0190] Channel weighting:

[0191] X′ c (i, j) = α c X c (i, j)

[0192] Where: X′ c (i, j) is the weighted pixel value of channel c at position (i, j), α c is the attention weight of the channel Xc (i, j) is the pixel value of the original channel c.

[0193] In step S6, convolution kernels with various void ratios are used to process the feature map, thereby obtaining multi-scale information. Specifically, the following is the process:

[0194] 1×1 convolution: used to extract global information and maintain the detailed features of the image.

[0195] 3×3 dilated convolution (dilation rate = 6): used to capture medium-range water features.

[0196] 3×3 dilated convolution (dilation rate = 12): used to capture water features in a larger range.

[0197] 3×3 dilated convolution (dilation rate = 18): used to capture water features in a wider range.

[0198] Then various convolutions are calculated:

[0199]

[0200] Where: W r is a convolution kernel with different dilation rates r, X is the input feature map, and Y is the output feature map after multi-scale convolution. Then, image pooling is performed to further expand the receptive field of the network and help the model to globally perceive the context information of the entire image.

[0201] The MPM module sets different pooling window sizes s 1 ,s 2 ,…,s n Perform pooling on the input feature map. Different window sizes can capture feature information of different scales, especially suitable for capturing water areas of different sizes. The specific calculation formula is as follows:

[0202]

[0203] Where: X is the input feature map, s i is the pooling window size of different scales, pooling(X, s i ) represents the pooling operation, and Z is the output feature map after multi-scale pooling.

[0204] In step S7, after completing the previous multi-level feature extraction, the segmentation module performs image segmentation through the Mask2Transformer technology. The Mask2Transformer module mainly captures the global information in the remote sensing image through the self-attention mechanism of the Transformer, and generates the final water body segmentation mask. This module can effectively integrate features from different levels and scales to achieve accurate water body segmentation. First, the feature map is input into the multi-layer Transformer encoder, which captures the long-distance dependencies in the image through the multi-head self-attention mechanism. The calculation steps of the self-attention mechanism:

[0205] (1) Calculation of query, key, and value:

[0206] Q=XW Q , K = XW K , V = XW V

[0207] Where: X is the input feature map, W Q , W K , W V is the linear transformation matrix of queries, keys, and values.

[0208] (2) Self-attention score calculation:

[0209]

[0210] Where: Q is the query matrix, K is the key matrix, V is the value matrix, d k is the dimension of the key matrix (used for scaling and normalization), and softmax is used to normalize the attention weights.

[0211] (3) Multi-head attention mechanism:

[0212] MultiHead(Q,K,V)=Concat(head 1 , …, head h )W O

[0213] Where: h is the number of attention heads, each attention head processes different feature dimensions, W O is the final linear transformation weight matrix.

[0214] After completing Transformer encoding, the generated feature map will be used to generate the final segmentation mask. Mask generation formula:

[0215] M mask (i, j) = sigmoid(W m *Fencoded (i, j)+b m )

[0216] Among them: F encoded (i, j) is the feature of the encoded feature map at position (i, j), W m and b m are the weights and biases of the mask generation layer, M mask (i, j) is the final generated water body segmentation mask, which indicates the probability that the pixel position belongs to the water body area. By binarizing each pixel (e.g., setting a threshold), the probability map can be converted into a binary segmentation mask, where "1" represents the water body area and "0" represents the non-water body area.

[0217] In step S8, after the segmentation mask is generated by Mask2Transformer, the last step is post-processing to improve the accuracy and consistency of the segmentation result. This step mainly includes edge smoothing and morphological operations. First, smoothing is performed by Gaussian filtering:

[0218]

[0219] Then a dilation operation is performed to expand the boundaries of the water area and fill in small holes in the segmentation:

[0220]

[0221] Where: M mask is the original mask, B is the structural element, It is an expansion operation.

[0222] Finally, the corrosion operation is performed to remove noise, shrink the segmentation boundary, and remove the mis-segmentation of non-water areas:

[0223]

[0224] Where: M mask is the original mask, B is the structural element, It is a corrosion operation.

[0225] Through the combination of smoothing, dilation and erosion, the boundary of the segmentation result becomes smoother, ensuring that the final segmentation mask is more accurate.

[0226]

[0227] Table 1 shows the processing results of various algorithm models

[0228] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture, characterized in that: The following steps are involved: S1. Data preprocessing: preprocessing the input high-resolution remote sensing images; S2.MAE pre-trained model, the pre-processed image is input into the MAE pre-trained model, the MAE pre-trained model masks a part of the input image, and then uses a self-supervised learning method to reconstruct the image; S3. Preliminary feature extraction, the image after MAE processing is subjected to preliminary feature extraction through linear projection, convolution layer, and nonlinear activation function to obtain a low-dimensional feature map; S4.CFF feature fusion, the low-dimensional feature map and the image processed by MAE are fused using the CFF module; S5. Feature enhancement of Mamba-T module: the feature map of the fused image is input into the Mamba-T module for standardization, Mamba Block feature extraction, and CA attention module operation to obtain further feature enhancement of the image; S6. Multi-scale feature extraction, using convolution kernels with various void ratios to process feature maps to obtain multi-scale information; S7. Segmentation module. After feature extraction, the feature map is input into the segmentation module for final image segmentation. This module captures global information through the self-attention mechanism of Transformer and generates the final segmentation mask. S8. Output and post-processing.

2. The remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture as claimed in claim 1, characterized in that: In step S1, normalization, data enhancement and noise removal are performed to improve the quality of the image, reduce the impact of noise on subsequent feature extraction, and enhance the generalization ability of the model. (1) Normalization: Through normalization, all pixel values ​​are scaled to the range of [0, 1], thereby reducing the numerical differences between different images and improving the adaptability of the model to the input image. Where: X is the original input pixel value, X min and X max are the minimum and maximum pixel values ​​in the original image, respectively, norm is the normalized pixel value, (2) Data enhancement: By rotating and flipping images, we can simulate the changes of remote sensing images at different angles and scales, and improve the robustness of the model to uncertain factors in actual scenes. Rotation formula: Flip formula: X filp (i,j)=X orig (i,W–j-1) (3) Noise removal: Gaussian filtering is used to remove noise and reduce the interference in the extraction of water areas. The formula is as follows:

3. The remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture as claimed in claim 1, characterized in that In step S2, some parts of the input image X are randomly selected for masking, and the mask ratio is 75%. The specific masking operation is to randomly select a part of the pixel area in the image and set these areas to zero, and define the mask matrix M, where M(i,j)=1 means that the pixel at position (i,j) is not masked, and M(i,j)=0 means that the pixel at this position is masked. The masked image X masked It is expressed as: X masked (i,j)=M(i,j)*X(i,j) Where: X(i,j) is the pixel value of the input image at position (i,j), M(i,j) is the value in the mask matrix, which is 0 or 1, Then perform MAE encoding and decoding: Z encoded =f encoder (X masked ) X reconstructed =f decoder (Z encoded ) Among them, f encoder is the network function of the encoder, X masked is the input image after mask processing; f decoder is the network function of the decoder. The goal of the decoder is to reconstruct the image X reconstructed As close as possible to the original image X.

4. The remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture as claimed in claim 1, characterized in that: In step S3, the MAE pre-trained output Z encoded Image features are used as input for feature extraction, including linear projection, convolution operation and nonlinear activation function. The projection operation calculation formula is: Z=W*X+b Where: X is the pixel value matrix of the input image, W is the weight matrix of the linear transformation, b is the bias term, and Z is the feature representation after projection. The high-dimensional pixel space of the original image is mapped to a lower-dimensional feature space through the weight matrix W and the bias term b, which helps to capture the local spatial structure in the image and provide more effective input features for the convolution operation. Convolution operation is performed to capture the local features of different areas in the image, and then the nonlinear expression ability of the model is enhanced through nonlinear activation function. SiLU is used as the nonlinear activation function, and its formula is: SiLU(x)=x*σ(x) Where σ(x) is the Sigmoid function:

5. The remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture as claimed in claim 1, characterized in that: In step S4, the CFF module is used to fuse the feature maps from MAE pre-training and two different levels of projection, convolution, and activation functions to ensure that the network can obtain more contextual information from image information of different scales. The two feature maps are set as F1 and F2, and the calculation formula for feature fusion is: F fused =α*F1+β*F2 Among them, α and β are learnable weight coefficients that control the weights of features at different layers. The weights α and β are not fixed, but are obtained through network adaptive learning. This adaptive allocation mechanism can dynamically adjust the weights of different feature layers, so that the network can self-optimize according to the characteristics of different images. The calculation formula is as follows: Among them: [F1, F2] is the feature after concatenating F1 and F2, W α and W β is the learned weight matrix, b α and b β is the bias term, σ is the Sigmoid activation function, ensuring that the values ​​of α and β are in the range of [0,1]. After processing by the CFF module, the fused feature map F fused It will be used as input to the subsequent Mamba-T module to further enhance the effect of water extraction.

6. The remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture according to claim 1, characterized in that: In step S5, four Mamba-T Block network modules are used for encoding, and the input is processed at different resolutions from H / 4×W / 4×C to H / 32×W / 32×8C. As the depth of the network increases, the size of the feature map gradually decreases, while the number of channels gradually increases, and then the output feature map is sent to the CA attention module. Mamba-T Block first performs Layer Normalization (LN) layer normalization operation, which normalizes the input features to ensure that the numerical range of the feature map remains stable at each layer. The normalization calculation formula is: Where: X i is the i-th feature of the input feature, μ is the mean of the feature map, σ 2 is the variance of the feature map, γ and β are learnable scaling and offset parameters, ∈ is a small constant to prevent division by zero errors, Secondly, the core Mamba Block module of Mamba-T Block is used for feature extraction, including linear projection, convolution operation, activation function (SiLU), SSM (Self-Similarity Module) and nonlinear processing. The specific steps are as follows: the layer-normalized image is divided into two paths for operation. The first path is linear projection-convolution operation-SiLU activation function-SSM, and the second path is linear projection-SiLU activation function. Then the two paths are output to the ARC module for fusion after nonlinear processing and linear projection. The SSM module is used to enhance the self-similarity of the feature map, which is achieved by comparing the similarities of different regions in the feature map. SSM(F)=similarity_function(F,F) The ARC module is used to further refine and combine the features from the multi-head self-attention mechanism. By dynamically adjusting the weights of the attention scores, the features from different channels are further integrated to improve the perception of the boundaries and internal areas of the water body. The adaptive weight calculation formula of ARC is as follows: Among them: F attention is a feature from the Mamba layer, F LN is the feature from the layer normalization operation, W α and W β is the learned weight matrix, b α and b β is the bias term, σ is the Sigmoid activation function, ensuring that the values ​​of α and β are in the range of [0,1]. Then, the attention features and convolution features are weighted and combined according to the weights: F ARC =α*F attention +β*F LN The ARC module achieves flexible fusion of different feature maps by adaptively adjusting the weights of different features. This module ensures that the model can pay attention to both the boundary details and the global structure of the water area. After the ARC module, the feature map enters the MLP for further nonlinear mapping of the features. The MLP maps the high-dimensional features to the target output space through multiple fully connected layers. The MLP calculation formula is as follows: F MLP =σ(W2*(σ(W1*F+b1))+b2) Where: F is the feature output by the ARC module, W1 and W2 are the weight matrices of the two-layer MLP, b1 and b2 are bias terms, σ is the activation function, SiLU is selected, After MLP, it enters the ARC module again to fuse the features from MLP and previous layers, and outputs the refined final features to enter the CA channel attention module. The CA (Channel Attention) mechanism is used to dynamically adjust the weight of each channel, enhance the model's attention to key water areas, and perform global average pooling: Where: G c is the global eigenvalue of channel c, X c (i, j) is the pixel value of channel c at pixel position (i, j), H and W are the height and width of the feature map respectively, Attention weight calculation: a c =σ(W c *G c +b c ) Where: α c is the attention weight of channel c, G c is the global eigenvalue of channel c, W c is the weight matrix of the fully connected layer, b c is the bias term, σ is the Sigmoid activation function, which is used to limit the attention weight to the range of [0,1]. Channel weighting: X′ c (i,j)=α c X c (i,j) Where: X′ c (i, j) is the weighted pixel value of channel c at position (i, j), α c is the attention weight of the channel X c (i,j) is the pixel value of the original channel c.

7. The remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture according to claim 1, characterized in that: In step S6, convolution kernels with various void ratios are used to process the feature map, thereby obtaining multi-scale information, as follows: 1×1 convolution: used to extract global information and maintain image detail features. 3×3 dilated convolution (dilation rate = 6): used to capture medium-range water features. 3×3 dilated convolution (dilation rate = 12): used to capture a larger range of water features. 3×3 dilated convolution (dilation rate = 18): used to capture water features in a wider range. Then various convolutions are calculated: Where: W r is a convolution kernel with different void rates r, X is the input feature map, Y is the output feature map after multi-scale convolution, and then enters the image pooling to further expand the receptive field of the network and help the model to globally perceive the context information of the entire image. The MPM module sets different pooling window sizes s1, s2, …, s n The input feature map is pooled. Different window sizes can capture feature information of different scales, which is especially suitable for capturing water areas of different sizes. The specific calculation formula is as follows: Where: X is the input feature map, s i is the pooling window size of different scales, pooling(X,s i ) represents the pooling operation, and Z is the output feature map after multi-scale pooling.

8. The remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture as claimed in claim 1, characterized in that: In step S7, after completing the previous multi-level feature extraction, the segmentation module performs image segmentation using the Mask2Transformer technology. First, the feature map is input into a multi-layer Transformer encoder, which captures the long-distance dependencies in the image through a multi-head self-attention mechanism. The calculation steps of the self-attention mechanism are: (1) Calculation of query, key, and value: Q=XW Q ,K=XW K ,V=XW V Where: X is the input feature map, W Q , W K , W V is the linear transformation matrix of query, key and value, (2) Self-attention score calculation: Where: Q is the query matrix, K is the key matrix, V is the value matrix, d k is the dimension of the key matrix (used for scaling and normalization), softmax is used to normalize the attention weights, (3) Multi-head attention mechanism: MultiHead(Q,K,V)=Concat(head1,…,head h )W O Where: h is the number of attention heads, each attention head processes different feature dimensions, W O is the final linear transformation weight matrix, After completing the Transformer encoding, the generated feature map will be used to generate the final segmentation mask. The mask generation formula is: M mask (i,j)=sigmoid(W m *F encoded (i,j)+b m ) Among them: F encoded (i, j) is the feature of the encoded feature map at position (i, j), W m and b m are the weights and biases of the mask generation layer, M mask (i, j) is the final generated water body segmentation mask, which indicates the probability that the pixel position belongs to the water body area. By binarizing each pixel, the probability map can be converted into a binary segmentation mask, where "1" represents the water body area and "0" represents the non-water body area.

9. The remote sensing image semantic segmentation method based on the fusion of Mamba and Transformer architecture as claimed in claim 1, characterized in that: In step S8, after the segmentation mask is generated by Mask2Transformer, the last step is post-processing to improve the accuracy and consistency of the segmentation result. This step includes edge smoothing and morphological operations. First, smoothing is performed by Gaussian filtering: Then a dilation operation is performed to expand the boundaries of the water area and fill in small holes in the segmentation: Where: M mask is the original mask, B is the structural element, is the expansion operation, Finally, the corrosion operation is performed to remove noise, shrink the segmentation boundary, and remove the mis-segmentation of non-water areas: Where: M mask is the original mask, B is the structural element, is a corrosion operation, Through the combination of smoothing, dilation and erosion, the boundary of the segmentation result becomes smoother, ensuring that the final segmentation mask is more accurate.

Images