A Semantic Segmentation Method for Land Use Remote Sensing Images Based on an Improved RMSIN Model

Abstract

This invention discloses a semantic segmentation method for land use remote sensing images based on an improved RMSIN model. This segmentation method improves the RMSIN model by removing the language modality correlation module, adding a CSMamba module between the Various Receptive and Vision Gate, adjusting the decoder, and stitching feature maps of multiple different scales together by channel to achieve multi-scale feature fusion. It also combines skip connections for upsampling, effectively restoring image details and improving segmentation accuracy, thus significantly improving the semantic segmentation accuracy and efficiency of land use remote sensing images.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image segmentation technology, specifically to a semantic segmentation method for land use remote sensing images based on an improved RMSIN model. Background Technology

[0002] Remote sensing technology acquires information about the Earth's surface features. This data primarily comes from sensors carried by satellites or aircraft, which collect remote sensing land imagery data. Remote sensing land imagery data can be used for various applications, including environmental monitoring, agricultural management, and urban planning. Because it provides continuous observation of vast areas, it is particularly important for managing the dynamic changes in land information. Remote sensing technology can obtain valuable information about land change and resource allocation, distinguishing different land use types, such as urban areas, farmland, forests, and water bodies.

[0003] In existing technologies, CN119152205A discloses a semantic segmentation method for multi-source remote sensing images based on Transformer, Mamba, and diffusion models. It uses a row-column pooling Transformer module instead of the traditional Transformer module as the backbone. Both RGB and depth information undergo multiple processing steps using the row-column pooling Transformer module, simultaneously extracting detailed texture information and large-scale structural information in the vertical and horizontal directions of the feature map. In this process, the temporal step position encoding is improved by combining Gaussian position encoding and trigonometric function position encoding to embed the time step into the model. After processing by the row-column pooling Transformer module, the RGB and depth features enter a cross-Mamba attention fusion module, which replaces the traditional Mamba module. This module constructs relevant parameters based on HiPPO continuous-time memory theory and exchanges specific matrices through state-space equation operations. Subsequently, a feature-weighted mask attention fusion module is used to weight the RGB and depth features, highlighting important feature regions before cross-attention fusion. First, a weight mask is generated using depthwise separable convolution and a Softmax activation function. Then, it undergoes reshaping, matrix multiplication, and other operations before finally being fused using a cross-attention mechanism. The decoder module consists of convolutional layers, linear layers, upsampling using bilinear interpolation, and a sigmoid activation function. Output features are obtained through depthwise separable convolution, linear transformation, upsampling, and activation function operations. Finally, the model training and optimization process is performed to train the parameters. However, the segmentation method disclosed in CN119152205A has a complex model structure and is difficult to tune. The model has numerous hyperparameters, and the modules influence each other; finding the optimal combination requires extensive experimentation and time. During training, noise samples are randomly drawn from a Gaussian distribution and added to the data, assuming that noise in real-world applications conforms to a Gaussian distribution. However, in real remote sensing scenarios, noise sources are complex and diverse, and may not perfectly conform to a Gaussian distribution. This specific assumption about noise may lead to insufficient robustness of the model when facing complex real-world noise, resulting in decreased segmentation performance.

[0004] In the prior art, CN114419449A discloses a remote sensing image semantic segmentation method based on self-attention multi-scale feature fusion. This method pre-trains a Swin-T network on a dataset as the encoder and uses a pyramidal network structure based on self-attention multi-scale feature fusion as the decoder. The feature encoder contains four sequentially connected Swin-T modules, which sequentially acquire features from the input remote sensing image, generating four feature maps at different scales, which are then passed to the decoder. The decoder has three self-attention multi-scale feature fusion modules, which input the feature maps generated by the first, second, and third Swin-T modules. Each module performs upsampling and channel number adjustment operations on the feature maps generated by different modules according to the current scale and number of channels to achieve feature fusion. The attention calculation module in each feature fusion module performs global average pooling on the adjusted three feature maps, concatenates the pooled feature maps, calculates self-attention, splits the result to obtain a relevance score, multiplies it with the original feature map, concatenates the results along the channel dimension, and adjusts the number of channels. The feature map generated by the fourth module of Swin-T is sequentially superimposed with the feature maps output by each feature fusion module. All superimposed feature maps and the feature map generated by the fourth module of Swin-T are upsampled to the original image size. Classification is predicted for each pixel, and the prediction results from the four scales are fused to obtain the final segmentation result. Furthermore, a multi-scale strategy is adopted for remote sensing semantic segmentation prediction, transforming the input image at different scales for segmentation prediction, and finally superimposing the segmentation results from all scales. However, the segmentation method disclosed in CN114419449A may overemphasize the correlation of distant pixels in some cases, especially when there is a lack of obvious contextual dependencies in the image. This over-modeling may lead to a waste of computational resources, particularly when processing local features, resulting in information redundancy. In addition, the differences in spatial size and semantic information between feature maps of different scales complicate multi-scale fusion, especially in pixel-level detail alignment and fusion. Improper handling may affect the final segmentation result. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a fusion model, a land use remote sensing image semantic segmentation model based on the fusion model and an improved RMSIN model, and a land use remote sensing image semantic segmentation method, which significantly improves the segmentation accuracy and efficiency of the model.

[0006] Firstly, a fusion model for multi-scale feature map fusion is provided, including:

[0007] The multi-scale feature map acquisition module is used to acquire multiple feature maps at different scales. Let the number of feature maps be N, and they are sorted in ascending order of scale as the first scale feature map, the second scale feature map, ..., the Nth scale feature map.

[0008] The first stitching module is used to stitch together multiple feature maps of different scales obtained by the multi-scale feature map acquisition module by channel and output the first stitched feature map.

[0009] The first fusion module is connected to the first scale feature map by skip connection. It is used to perform a first upsampling on the first concatenated feature map, then concatenate and fuse the first scale feature map obtained by skip connection with the feature map output after the first upsampling, and then perform a convolution operation on the concatenated and fused feature map and output it.

[0010] The second fusion module, which is connected to the second scale feature map by skip connection, is used to perform a second upsampling on the feature map output by the first fusion module, and then concatenate and fuse the second scale feature map obtained by skip connection with the feature map output after the second upsampling. Finally, the concatenated and fused feature map is convolved and output.

[0011] The nth fusion module, with skip connections to the nth scale feature map (n = 1, 2, ..., N), is used to upsample the feature map output by the (n-1)th fusion module to the nth scale. Then, the nth scale feature map obtained by the skip connections is concatenated and fused with the nth upsampled feature map. Finally, the concatenated and fused feature map is convolved and output. The output of the Nth fusion module is used as the fusion result.

[0012] In some embodiments, the multi-scale feature map acquisition module includes:

[0013] Multiple feature maps of different scales are obtained by using the outputs of layers of different depths in the same serial processing channel of the same deep neural network module.

[0014] or,

[0015] Based on the input image, multiple parallel processing channels are used to generate encoding branches or feature extraction branches, resulting in feature maps of different scales output by the parallel branches.

[0016] Secondly, a fusion method for multi-scale feature map fusion is provided, including the following steps:

[0017] Obtain multiple feature maps at different scales. Let the number of feature maps be N, and sort them in ascending order of scale as the first scale feature map, the second scale feature map, ..., the Nth scale feature map;

[0018] The multi-scale feature map acquisition module concatenates multiple feature maps of different scales according to channels and outputs the first concatenated feature map.

[0019] The first concatenated feature map is upsampled, and then the first scale feature map is obtained by using skip connections. The first scale feature map is concatenated and fused with the feature map output after the first upsampling. The concatenated and fused feature map is then convolved and output, which is called the first fused output.

[0020] The first fusion output is upsampled a second time, and then a second-scale feature map is obtained by using skip connections. The second-scale feature map is then concatenated and fused with the feature map output after the second upsampling. Finally, the concatenated and fused feature map is convolved and output, which is denoted as the second fusion output.

[0021] Similarly, the (n-1)th fused output is upsampled to the nth level. Then, the nth scale feature map obtained by skip connections is concatenated and fused with the feature map output after the nth upsampling. Finally, the concatenated and fused feature map is convolved and output, denoted as the nth fused output, where n = 1, 2, ..., N.

[0022] The fusion result is based on the Nth fusion output as a multi-scale feature map.

[0023] In some implementations, obtaining multiple feature maps at different scales includes: obtaining multiple feature maps at different scales based on the outputs of layers of different depths in the same serial processing channel of the same deep neural network module; or, obtaining feature maps at different scales output by multiple parallel branches based on the encoding branches or feature extraction branches of multiple different parallel processing channels applied to the input image.

[0024] Thirdly, a semantic segmentation model for land use remote sensing images based on an improved RMSIN model is provided, including:

[0025] The first coding module is an improvement on the multi-scale coding module of the RMSIN model. It discards the natural speech data input and the units on the natural speech data processing path in each coding layer of the multi-scale coding module of the RMSIN model, and retains the image input and the units on the image processing path in each coding layer to form the first coding module.

[0026] The second encoding module is used to further process the feature maps of multiple scales output by the first encoding module. The second encoding module adopts the cross-scale interaction module of the RMSIN model.

[0027] The decoding module processes the feature maps at multiple scales output by the second encoding module using the fusion model as described in the first aspect or the fusion method as described in the second aspect above, to obtain the semantic segmentation result of the remote sensing image.

[0028] In some implementations, the first encoding module includes multiple encoding layers, each encoding layer outputting a feature map at one scale, and each encoding layer includes:

[0029] The image input unit is used to acquire land use remote sensing images or feature maps output from the previous coding layer;

[0030] Converter layer

[0031] Multiple sensory field units,

[0032] CSMamba unit,

[0033] Visual gate unit,

[0034] The image input to the image input unit is sequentially processed through a transformer layer, a multi-receptive field unit, a CSMamba unit, and a visual gate unit for image feature encoding.

[0035] Fourthly, a training method for a land use remote sensing image semantic segmentation model based on an improved RMSIN model is provided, including the following steps:

[0036] Step B1: Collect remote sensing image data and divide the dataset: Divide the dataset into training set, validation set, and test set in a 6:2:2 ratio;

[0037] Step B2: Data Augmentation and Labeling: Apply rotations of 0°, 90°, 180°, and 270°, and scaling of 0.75x, 1x, and 1.25x to the images in the training, validation, and test sets; use LabelMe labeling software to label the augmented images to obtain a dataset containing different labeled data, which represents land use types, and output a PNG format mask image, where each pixel is labeled with a category number;

[0038] Step B3: Construct a land use remote sensing image semantic segmentation model based on the improved RMSIN model as described in the third aspect:

[0039] Step B4: Train the improved RMSIN model based on the dataset to obtain the trained land use remote sensing image semantic segmentation model.

[0040] Fifthly, a semantic segmentation method for land use remote sensing images based on an improved RMSIN model is provided, including the following steps:

[0041] The remote sensing image is input into the first encoding module for first encoding to obtain feature maps at multiple scales. The first encoding module is obtained by improving the multi-scale encoding module based on the RMSIN model. In the improved multi-scale encoding module of the RMSIN model, the input of natural speech data and the units on the natural speech data processing path are discarded in each encoding layer, while the units on the image input and image processing path are retained in each encoding layer.

[0042] The cross-scale interaction module of the RMSIN model is used to process feature maps of multiple scales sequentially to obtain further processed feature maps of multiple scales.

[0043] The feature maps at multiple scales are processed using the fusion method described in the second aspect to obtain the fusion result as the semantic segmentation result of the remote sensing image.

[0044] In some implementations, the encoding method of the first encoding module includes:

[0045] Acquire the input land use remote sensing image;

[0046] The input image is encoded sequentially through multiple coding layers, with each coding layer outputting a feature map at one scale. Within each coding layer, the image features are encoded sequentially through a transformer layer, a multi-receptive field unit, a CSMamba unit, and a visual gate unit.

[0047] In a sixth aspect, a computer-readable storage medium is provided, on which computer instructions are stored, which, when executed by a processor, implement the steps of the above-described fusion method, or the steps of the above-described land use remote sensing image semantic segmentation method, or the steps of the above-described land use remote sensing image semantic segmentation model training method.

[0048] The present invention provides a semantic segmentation method for land use remote sensing images based on an improved RMSIN model, which has the following characteristics:

[0049] Beneficial effects:

[0050] 1. The fusion model of this invention concatenates feature maps of multiple different scales by channel to achieve multi-scale feature fusion, and combines skip connections for upsampling, which can effectively restore image details and improve segmentation accuracy. Semantic information is extracted through convolution operations, enhancing the model's ability to understand complex images.

[0051] 2. This invention presents a land use remote sensing image semantic segmentation model based on a fusion model and an improved RMSIN model. Through multi-scale feature fusion and a self-attention mechanism, it can more effectively handle the diversity and complexity in remote sensing images, improving segmentation accuracy, especially in the accurate prediction of detailed regions, such as the boundary between water and land. Simultaneously, this method can also accurately identify the segmentation of different types of land features, from vast bodies of water to scattered farmland, enhancing its ability to handle diverse land feature characteristics. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the fusion model in the embodiments of this application;

[0053] Figure 2 This is a schematic diagram of the structure of the land use remote sensing image semantic segmentation model based on the improved RMSIN model in the embodiments of this application;

[0054] Figure 3 This is a schematic diagram of the structure within each coding layer of the first coding module in the embodiments of this application;

[0055] Figure 4 This is a schematic diagram of the structure of the CSMamba module in an embodiment of this application;

[0056] Figure 5 This is a flowchart of the method for training a semantic segmentation model of land use remote sensing images in the embodiments of this application. Detailed Implementation

[0057] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0058] The following describes exemplary embodiments of this application.

[0059] Example 1

[0060] This application provides a fusion model for multi-scale feature map fusion, such as... Figure 1 As shown, it includes:

[0061] The multi-scale feature map acquisition module 101 is used to acquire multiple feature maps of different scales. Let the number of feature maps be N, and they are sorted in ascending order of scale as the first scale feature map, the second scale feature map, ..., the Nth scale feature map.

[0062] The first stitching module 102 is used to stitch together multiple feature maps of different scales obtained by the multi-scale feature map acquisition module by channel and output the first stitched feature map.

[0063] The first fusion module 103 is connected to the first scale feature map by skip connection. It is used to perform a first upsampling on the first spliced ​​feature map, then splice and fuse the first scale feature map obtained by skip connection with the feature map output after the first upsampling, and then perform a convolution operation on the spliced ​​and fused feature map and output it.

[0064] The second fusion module 104 is connected to the second scale feature map by skip connection. It is used to perform a second upsampling on the feature map output by the first fusion module, and then concatenate and fuse the second scale feature map obtained by skip connection with the feature map output after second upsampling. Finally, the concatenated and fused feature map is convolved and output.

[0065] The nth fusion module, with skip connections to the nth scale feature map (n = 1, 2, ..., N), is used to upsample the feature map output by the (n-1)th fusion module, then concatenate and fuse the nth scale feature map obtained by the skip connections with the nth upsampled feature map, and finally perform convolution operation on the concatenated and fused feature map and output it.

[0066] The output of the Nth fusion module is used as the fusion result.

[0067] In this embodiment, the fusion model concatenates feature maps of multiple different scales by channel to achieve multi-scale feature fusion. Combined with skip connections for upsampling, it effectively restores image details and extracts semantic information through convolution operations, enhancing the model's ability to understand complex images. The fusion model achieves more detailed regions and multi-scale features during image processing. It innovatively concatenates, upsamples, and fuses features of different scales, effectively overcoming the limitations of single-scale feature utilization and enabling comprehensive mining and utilization of multi-scale information. This enhances the model's generalization ability, allowing it to accurately capture key features when faced with images of various scales. It can be understood that the fusion module provided in this embodiment can be used in target recognition tasks. In target recognition tasks, after preliminary feature extraction to obtain feature maps of multiple scales, the fusion module of this embodiment is used for multi-scale feature map processing, and target recognition and classification are performed through a classification network. The fusion module provided in this embodiment can also be used in image semantic segmentation tasks, and further, in semantic segmentation tasks of land use remote sensing images. After preliminary feature extraction to obtain feature maps of multiple scales, the fusion module of this embodiment is used for image semantic segmentation.

[0068] Example 2

[0069] This application provides a fusion model for multi-scale feature map fusion, including a multi-scale feature map acquisition module 101, a first stitching module 102, a first fusion module 103, a second fusion module 104, ..., the nth fusion module, ..., the Nth fusion module, with the output of the Nth fusion module as the fusion result.

[0070] In this embodiment of the application, the multi-scale feature map acquisition module 101 is further described. Optionally, the multi-scale feature map acquisition module 101 includes: obtaining multiple feature maps of different scales based on the outputs of layers of different depths in the same serial processing channel of the same deep neural network module; or, obtaining feature maps of different scales output by multiple parallel branches based on the encoding branches or feature extraction branches of multiple parallel processing channels that are different from each other for the input image.

[0071] In this embodiment, the multi-scale feature map acquisition module can apply multiple different encoding branches or feature extraction branches to the input image to obtain feature maps of different scales output by multiple branches. For example, in the YOLOv5 network, the NECK network outputs three feature maps of different scales. Another example is the RMSIN model, where input image data and speech data are processed by an intra-scale interaction module (IIM) and a cross-scale interaction module (CIM) to obtain four feature maps of different scales. In another implementation, the multi-scale feature map acquisition module can obtain multiple feature maps of different scales based on layers of different depths within the same deep neural network. Examples include the bottom-up process of the FPN feature pyramid, the ResNet network, or the Faster R-CNN backbone network.

[0072] The other related technical solutions in this application can be found in the relevant description in Embodiment 1, and can achieve the same beneficial effects. To avoid repetition, they will not be described again here.

[0073] Example 3

[0074] This application provides a fusion method for multi-scale feature map fusion, including the following steps:

[0075] Step A1: Obtain multiple feature maps at different scales. Let the number of feature maps be N, and sort them in ascending order of scale as the first scale feature map, the second scale feature map, ..., the Nth scale feature map.

[0076] Step A2: The multi-scale feature map acquisition module concatenates multiple feature maps of different scales according to channels and outputs the first concatenated feature map;

[0077] Step A3: Perform a first upsampling on the first concatenated feature map, then use skip connections to obtain a first-scale feature map, and concatenate and fuse the first-scale feature map with the feature map output after the first upsampling. Then perform a convolution operation on the concatenated and fused feature map and output it, which is called the first fused output.

[0078] Step A4: Perform a second upsampling on the first fusion output, then use skip connections to obtain a second-scale feature map, and concatenate and fuse the second-scale feature map with the feature map output after the second upsampling. Then perform a convolution operation on the concatenated and fused feature map and output it, which is called the second fusion output.

[0079] Step A5, and so on, the (n-1)th fused output is upsampled to the nth level, and then the nth scale feature map obtained by skip connections is concatenated and fused with the feature map output after the nth upsampling. Then the concatenated and fused feature map is convolved and output, denoted as the nth fused output, n = 1, 2, ..., N;

[0080] Step A6: Use the Nth fusion output as the fusion result of the multi-scale feature map.

[0081] The fusion method for multi-scale feature map fusion provided in this application embodiment belongs to the same concept as the fusion model embodiment for multi-scale feature map fusion provided in Embodiment 1 above. The specific implementation process is detailed in Embodiment 1 above, and will not be repeated here.

[0082] Example 4

[0083] This application provides a method for fusing multi-scale feature maps, including steps A1 to A6 as described in Embodiment 3. In this embodiment, step A1 is further explained. Step A1 involves obtaining multiple feature maps of different scales, including: obtaining multiple feature maps of different scales based on the outputs of layers of different depths in the same serial processing channel of the same deep neural network module; or, obtaining feature maps of different scales output by multiple parallel branches based on the encoding branches or feature extraction branches of multiple different parallel processing channels applied to the input image.

[0084] The fusion method for multi-scale feature map fusion provided in this application embodiment belongs to the same concept as the fusion model embodiment for multi-scale feature map fusion provided in Embodiment 2 above. The specific implementation process is detailed in Embodiment 2 above, and will not be repeated here.

[0085] Example 5

[0086] See Figure 2 This application provides a semantic segmentation model for land use remote sensing images based on an improved RMSIN model, comprising:

[0087] The first coding module is an improvement on the multi-scale coding module of the RMSIN model (that is, the part of the RMSIN model that contains 4 Encoder Layers, each Encoder Layer outputs a scale feature map). The first coding module is formed by discarding the natural speech data input and the units on the natural speech data processing path in each coding layer of the multi-scale coding module of the RMSIN model, and retaining the image input and the units on the image processing path in each coding layer.

[0088] The second encoding module, which is the cross-scale interaction module of the RMSIN model, namely the CIM module, is used to further process the feature maps of multiple scales output by the first encoding module.

[0089] The decoding module processes the feature maps of multiple scales output by the second encoding module using the fusion model described in Embodiment 1 or the fusion method described in Embodiment 3 to obtain the semantic segmentation result of the remote sensing image.

[0090] In this embodiment, the first encoding module borrows the Intra-scale Interaction Module (IIM) of RMSIN, but improves the IIM module of RMSIN. In the IIM module, the original multi-receptive field feature extraction within a single size is retained; the language modality-related parts are removed, that is, the cross-modal alignment branch and language gate are deleted, simplifying it to single-modal visual feature processing.

[0091] The main task of the modified IIM module is to extract rich feature information from various scales of the image, enhancing the model's spatial perception capabilities. By effectively enhancing key information in the image, the IIM module helps the model better capture detailed information about objects and improves the prediction accuracy for different categories, especially in complex scenes.

[0092] The adjusted RMSIN decoder achieves multi-scale feature fusion by concatenating the outputs of the four CSIE stages channel-by-channel and incorporates skip connections for upsampling, effectively restoring image details and improving segmentation accuracy. Semantic information is extracted through convolutional operations, enhancing the model's ability to understand complex remote sensing images. This change makes the decoder more adaptable to semantic segmentation tasks, especially in capturing detailed regions and multi-scale features, significantly improving the model's segmentation accuracy and efficiency.

[0093] Optionally, the first encoding module includes multiple encoding layers, each encoding layer outputting a feature map at one scale, such as... Figure 3 As shown, each coding layer includes:

[0094] The image input unit is used to acquire land use remote sensing images or feature maps output from the previous coding layer;

[0095] Transformer Layer

[0096] Multiple receptive fields

[0097] CSMamba (Channel and Spatial Mamba Block) unit,

[0098] Vision gate unit

[0099] The image input to the image input unit is sequentially processed through a transformer layer, a multi-receptive field unit, a CSMamba unit, and a visual gate unit for image feature encoding.

[0100] In this embodiment, the IIM module is further improved in the first encoding module by adding a CSMamba unit to the retained image input and image processing paths. Specifically, a CSMamba unit is added between the Transformer Layer and the multi-receptive field unit. Therefore, the first encoding module in this embodiment sequentially includes: an image input unit, a Transformer Layer, a Various Receptive, a CSMamba unit, and a vision gate unit. In this embodiment, the first encoding module adds CSMamba between the Various Receptive and the Vision Gate in the improved RMSIN. The CSMamba module combines techniques such as depthwise convolution, channel and spatial attention, and 2D selective scanning, enabling it to effectively capture long-range contextual information with linear time complexity when processing remote sensing images.

[0101] Specifically, the first encoding module includes multiple encoder layers (e.g., four encoder layers) connected sequentially, each outputting a feature map of a single scale. Each encoder layer uses φ i This indicates that the output features of each coding layer are represented by... Indicate, then Language feature F was deleted l It only retains the visual features of the previous coding layer. As input.

[0102] Multiple sensory fields:

[0103] (1) For the visual features of the input By using multiple branches, each with different convolution kernel settings (such as size and stride), features with different receptive fields can be obtained.

[0104] (2) Calculate the weight of each pixel using the formula:

[0105]

[0106] σ: Sigmoid function, used to normalize weights; The convolution kernel of the j-th convolutional branch; Input features.

[0107] (3) Weight w i Used to enhance input features: in, This indicates element-wise multiplication.

[0108] CSMamba Unit: A high-efficiency feature fusion module that combines channel and spatial attention mechanisms with the linear time complexity of the Mamba framework to effectively capture long-range dependencies in remote sensing images, while improving the accuracy of multi-scale information fusion and semantic segmentation.

[0109] Visual gate unit: Further adjusts the enhanced features.

[0110]

[0111] By adjusting features using activation functions (Tanh and ReLU) and linear projection, local detail information can be supplemented.

[0112] Output: Visual features enhanced by multi-receptor field branching and visual gating. The output formula is adjusted to

[0113]

[0114] See Figure 4 In the CSMamba module, the input feature F is processed through two parallel branches. The first branch primarily extracts richer feature information by expanding the number of feature channels and performing depthwise convolution. After expanding the number of channels through linear layers, features are extracted using depthwise convolution and the SiLU activation function, then processed by a 2D selective scan module (2D-SSM), and finally normalized using LayerNorm. The purpose of this process is to capture long-range dependencies and effectively extract multi-scale feature information, laying the foundation for subsequent feature fusion and global context modeling. The second branch weights the input features through channel and spatial attention (CS) mechanisms, using an attention module to focus on important regions, and then applies the SiLU activation function to enhance the feature representation, thereby effectively integrating global and local information of the image.

[0115] Input features: F∈R H×W×C This represents a feature map of an image with height H, width W, and number of channels C.

[0116] First branch:

[0117] (1) Channel expansion: The number of channels of the input feature is expanded to λC through a linear layer (where λ is a predefined expansion factor).

[0118] F' = Linear(F)

[0119] Where: F∈R H×W×C This is the input feature map, where H is the height, W is the width, and C is the number of channels. F'∈R H×W×λC It is the expanded feature map, with the number of channels extended to λC.

[0120] (2) Depth-Wise Convolution (DWConv): Performs depth-wise convolution operations. Depth-wise convolution performs convolution independently on each channel, reducing computational complexity and the number of parameters. Compared to standard convolution, the computational cost of depth-wise convolution is O(C·K). 2 The computational cost of standard convolution is O(H·W), while that of standard convolution is O(C). 2 ·K 2 ·H·W), where K is the convolution kernel corresponding to the input channel F.

[0121] (3) Activation function: Use SiLU (Sigmoid Linear Unit) activation function to enhance feature representation.

[0122] SiLU(F i ) = F i ·σ(F i )

[0123] Wherein: F i This is the feature map after depthwise convolution (i.e., the value input to the SiLU activation function). σ(F) i ) is the Sigmoid activation function, that is:

[0124] (4) 2D Selective Scan Module (2D-SSM): Used to handle long-range dependencies in the image. It flattens the image features into a 1D sequence and scans in four directions: top left to bottom right, bottom right to top left, top right to bottom left, and bottom left to top right. Each scan utilizes a Selective State-Space Model to model long-range dependencies. In this way, it can capture long-range dependencies in each direction. The scan results in the four directions (different features of long-range dependencies) are merged through an aggregation operation to recover the 2D structure of the image.

[0125] Second branch:

[0126] Channel attention and spatial attention modules (CS) are introduced to improve the representational power of feature maps. The channel attention module strengthens attention to important channels through global feature aggregation, while the spatial attention module improves the response to key spatial regions by focusing on spatial dimensions. The combination of the two significantly improves the semantic segmentation effect of remote sensing images, especially in the processing of complex scenes and multi-scale information. The SiLU activation function is also used for feature transformation.

[0127] Merge the first and second branches:

[0128] The output features of the two branches are fused using a Hadamard product (element-wise multiplication). This approach effectively combines two different types of information: one is the global context obtained through 2D selective scanning, and the other is the features focused through channel and spatial attention.

[0129] Finally, the fused features are mapped back to the original number of channels C through a linear layer to obtain the final output features F. out .

[0130] In summary, the overall process formula for CSMamba can be expressed as:

[0131] F1=LN(2D-SSM(SiLU(DWConv(Linear(F)))))

[0132] F2=SiLU(CS(F))

[0133] F out =Linear(F1⊙F2).

[0134] Adding CSMamba between the Various Receptive and Vision Gate in the improved RMSIN effectively enhances long-range dependency modeling and information fusion capabilities, improving the model's ability to handle complex scenes and multi-scale features in remote sensing images. Through the channel and spatial attention mechanisms provided by CSMamba, global contextual information can be accurately captured while maintaining computational efficiency, and features from different scales can be fused, thereby improving the accuracy of semantic segmentation, especially when handling detailed regions and complex boundaries.

[0135] Example 6

[0136] See Figure 5 This application provides a training method for a land use remote sensing image semantic segmentation model based on an improved RMSIN model, comprising the following steps:

[0137] Step B1: Collect remote sensing image data and divide the dataset: Divide the dataset into training set, validation set, and test set in a 6:2:2 ratio;

[0138] Step B2: Data Augmentation and Labeling: Rotations of 0°, 90°, 180°, and 270°, and scaling by 0.75x, 1x, and 1.25x were applied to the images in the training, validation, and test sets, increasing the dataset by twelve times. The augmented images were labeled using Labelme software to obtain a dataset containing different labeled data. The labeled data represents land use types, including four categories: water, residential, farmland, and forest. A PNG mask image was output, with each pixel labeled as a category number (e.g., 0 for water, 1 for residential, 2 for farmland, and 3 for forest).

[0139] Step B3: Construct a land use remote sensing image semantic segmentation model based on the improved RMSIN model as described in Example 5:

[0140] Step B4: Train the improved RMSIN model based on the dataset to obtain the trained land use remote sensing image semantic segmentation model. Step B4 includes: during the training process, using cross-validation, evaluating the model performance with the validation set after each training cycle, and adjusting the model parameters according to the evaluation results to continuously optimize the model performance until the best results are achieved.

[0141] Example 7

[0142] This application provides a method for semantic segmentation of land use remote sensing images based on an improved RMSIN model, including the following steps:

[0143] Step C1: Input the remote sensing image into the first encoding module to perform the first encoding to obtain feature maps at multiple scales. The first encoding module is obtained by improving the multi-scale encoding module of the RMSIN model. In the improved multi-scale encoding module of the RMSIN model, the input of natural speech data and the units on the natural speech data processing path are discarded in each encoding layer, while the units on the image input and image processing path in each encoding layer are retained.

[0144] Step C2: Use the cross-scale interaction module of the RMSIN model to process the feature maps of multiple scales in turn to obtain optimized feature maps of multiple scales.

[0145] Step C3: Process the feature maps at multiple scales using the fusion model described in Example 1 or the fusion method described in Example 3 to obtain the fusion result as the semantic segmentation result of the remote sensing image.

[0146] Furthermore, in step C1, the encoding method of the first encoding module includes:

[0147] Step C11: Acquire the input land use remote sensing image;

[0148] Step C12: The input image is encoded sequentially through multiple coding layers (for example, four coding layers can be used), and each coding layer outputs a feature map of one scale; within each coding layer, the image features are encoded sequentially through a transformer layer, a multi-receptive field unit, a CSMamba unit, and a visual gate unit.

[0149] The land use remote sensing image semantic segmentation method based on the improved RMSIN model in this application embodiment is conceptually the same as the segmentation model described in Embodiment 5 above, and will not be repeated here.

[0150] According to embodiments of this application, this application also provides a computer-readable storage medium storing computer instructions thereon. When executed by a processor, the instructions implement the steps of the fusion method for multi-scale feature map fusion as described in Embodiment 1 or Embodiment 2; or, when executed by a processor, the instructions implement the steps of the land use remote sensing image semantic segmentation method based on the improved RMSIN model as described in Embodiment 7; or implement the steps of the training method for the land use remote sensing image semantic segmentation model based on the improved RMSIN model as described in Embodiment 6. The term "computer-readable medium" refers to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor.

[0151] This invention is not limited to the specific embodiments described above. Any modifications made by those skilled in the art based on the above concept without creative effort are within the scope of protection of this invention.

Claims

1. A semantic segmentation model for land use remote sensing images based on an improved RMSIN model, characterized in that, include: The first coding module is an improvement on the multi-scale coding module of the RMSIN model. It discards the natural speech data input and the units on the natural speech data processing path in each coding layer of the multi-scale coding module of the RMSIN model, and retains the image input and the units on the image processing path in each coding layer to form the first coding module. The second encoding module is used to further process the feature maps of multiple scales output by the first encoding module. The second encoding module adopts the cross-scale interaction module of the RMSIN model. The decoding module uses a fusion model to process the feature maps at multiple scales output by the second encoding module to obtain the semantic segmentation results of the remote sensing image. The fusion model includes: The multi-scale feature map acquisition module is used to acquire multiple feature maps at different scales. Let the number of feature maps be N, and they are sorted in ascending order of scale as the first scale feature map, the second scale feature map, ..., the Nth scale feature map. The first stitching module is used to stitch together multiple feature maps of different scales obtained by the multi-scale feature map acquisition module by channel and output the first stitched feature map. The first fusion module is connected to the first scale feature map by skip connection. It is used to perform a first upsampling on the first concatenated feature map, then concatenate and fuse the first scale feature map obtained by skip connection with the feature map output after the first upsampling, and then perform a convolution operation on the concatenated and fused feature map and output it. The second fusion module, which is connected to the second scale feature map by skip connection, is used to perform a second upsampling on the feature map output by the first fusion module, and then concatenate and fuse the second scale feature map obtained by skip connection with the feature map output after the second upsampling. Finally, the concatenated and fused feature map is convolved and output. The nth fusion module, with skip connections to the nth scale feature map (n=1, 2, ..., N), is used to upsample the feature map output by the (n-1)th fusion module to the nth scale. Then, the nth scale feature map obtained by the skip connections is concatenated and fused with the nth upsampled feature map. Finally, the concatenated and fused feature map is convolved and output. The output of the Nth fusion module is used as the fusion result.

2. The land use remote sensing image semantic segmentation model based on the improved RMSIN model according to claim 1, characterized in that, The first encoding module includes multiple encoding layers, each encoding layer outputting a feature map at one scale, and each encoding layer includes: The image input unit is used to acquire land use remote sensing images or feature maps output from the previous coding layer; Converter layer Multiple sensory field units, CSMamba unit, Visual gate unit, The image input to the image input unit is sequentially processed through a transformer layer, a multi-receptive field unit, a CSMamba unit, and a visual gate unit for image feature encoding.

3. A training method for a land use remote sensing image semantic segmentation model based on an improved RMSIN model, characterized in that, include: Step B1: Collect remote sensing image data and divide the dataset: Divide the dataset into training set, validation set, and test set in a 6:2:2 ratio; Step B2: Data Augmentation and Labeling: Apply rotations of 0°, 90°, 180°, and 270°, and scaling of 0.75x, 1x, and 1.25x to the images in the training, validation, and test sets; use Labelme labeling software to label the augmented images to obtain a dataset containing different labeled data, which represents land use types, and output a PNG format mask map, where each pixel's label is a category number; Step B3: Construct a land use remote sensing image semantic segmentation model based on the improved RMSIN model as described in any one of claims 1-2: Step B4: Train the improved RMSIN model based on the dataset to obtain the trained land use remote sensing image semantic segmentation model.

4. A semantic segmentation method for land use remote sensing images based on an improved RMSIN model, characterized in that, include: The remote sensing image is input into the first encoding module for first encoding to obtain feature maps at multiple scales. The first encoding module is obtained by improving the multi-scale encoding module based on the RMSIN model. In the improved multi-scale encoding module of the RMSIN model, the input of natural speech data and the units on the natural speech data processing path are discarded in each encoding layer, while the units on the image input and image processing path are retained in each encoding layer. The cross-scale interaction module of the RMSIN model is used to process feature maps of multiple scales sequentially to obtain further processed feature maps of multiple scales. The feature maps at multiple scales are processed using a fusion method to obtain the fusion result as the semantic segmentation result of the remote sensing image; The fusion method includes: acquiring multiple feature maps at different scales, assuming the number of feature maps is N, and sorting them in ascending order of scale as first-scale feature map, second-scale feature map, ..., Nth-scale feature map; concatenating the multiple feature maps at different scales acquired by the multi-scale feature map acquisition module by channel and outputting a first concatenated feature map; performing a first upsampling on the first concatenated feature map, then using skip connections to acquire a first-scale feature map, and concatenating and fusing the first-scale feature map with the feature map output after the first upsampling; then performing a convolution operation on the concatenated and fused feature map and outputting it as the first fused output; and finally, performing a convolution operation on the first fused feature map. The output is upsampled a second time, and then a second-scale feature map is obtained using skip connections. This second-scale feature map is then concatenated and fused with the feature map output after the second upsampling. The concatenated and fused feature map is then convolved and output, denoted as the second fused output. This process is repeated, with the (n-1)th fused output being upsampled a second time. The nth-scale feature map obtained using skip connections is then concatenated and fused with the feature map output after the nth upsampling. The concatenated and fused feature map is then convolved and output, denoted as the nth fused output, where n = 1, 2, ..., N. The Nth fused output is used as the fusion result of the multi-scale feature maps.

5. The land use remote sensing image semantic segmentation method based on the improved RMSIN model according to claim 4, characterized in that, The encoding method of the first encoding module includes: Acquire the input land use remote sensing image; The input image is encoded sequentially through multiple coding layers, with each coding layer outputting a feature map at one scale. Within each coding layer, the image features are encoded sequentially through a transformer layer, a multi-receptive field unit, a CSMamba unit, and a visual gate unit.

6. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instruction is executed by the processor, it implements the steps of the method as described in claim 3 or the steps of the method as described in any one of claims 4-5.

Images