Mamba-based image semantic segmentation method and system

Abstract

The invention relates to a Mama-based image semantic segmentation method and system, and the method comprises the steps: inputting an original image into a feature extraction module to extract features, and adding a plurality of Mama sub-modules to the feature extraction module Mception on the basis of an Xception network; a result obtained after first down-sampling of the extracted features is input into an improved RASPP module to enhance spatial context information fusion, then global features are summarized through an R-Mama module, and the long-distance dependency relationship of the image is enhanced; inputting a result of the extracted features after the second downsampling into the improved RASPP; performing up-sampling on the result of the second down-sampling post-processing to perform size recovery; and splicing and fusing the results, adjusting the number of channels, performing up-sampling to restore the image size, and outputting a predicted image. Compared with the prior art, the method has the advantages of enhancing the precision and speed of semantic segmentation, reducing the complexity and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of image semantic segmentation, and in particular to a Mamba-based image semantic segmentation method and system. Background Art

[0002] With the development of science and technology, semantic segmentation technology can be seen everywhere in life. While promoting the development of science and technology, semantic segmentation has also brought a lot of convenience to our lives. For example, applying semantic segmentation to autonomous driving systems can help people with assisted driving to a certain extent and protect people's lives. In addition, semantic segmentation can also be applied to agricultural monitoring, which can identify crops and monitor the growth of crops and the environmental conditions of farmland. In medical image processing, semantic segmentation can accurately locate lesions, assist doctors in diagnosis, and to a certain extent help doctors better understand the structure of the surgical site and reduce surgical risks. In remote sensing, semantic segmentation technology can be used to classify land in remote sensing images and identify buildings and vegetation. After a natural disaster occurs, semantic segmentation of remote sensing images can quickly identify disaster-stricken areas, damaged buildings and infrastructure, etc., providing timely and accurate information for emergency rescue and loss assessment of disasters. There are many typical segmentation models in image segmentation. FCN, U-Net, SegNet, DeepLab, etc. have relatively good performance in image segmentation. In the DeepLab series, the latest model is DeepLabv3+, which adds a simple but effective decoder module to the infrastructure of DeepLabv3 to better restore spatial information and refine segmentation boundaries; the ASPP module also allows the network to process receptive fields of different sizes at the same time, improving the accuracy of semantic segmentation. However, the DeepLabv3+ model also has shortcomings. It is weak in processing detailed information. For example, it may not be able to process all the detailed information for very small objects and their edges. In addition, DeepLabv3+ has high computational complexity and is difficult to train, resulting in slow inference speed.

[0003] The development of the Mamba model is based on the state space model (SSM). The key to SSM is the state equation: h′(t)=Ah(t)+Bx(t) and the output equation: y(t)=Ch(t)+Dx(t). Among them, x(t) is the input, h′(t) represents the current state, h(t) represents the previous state of the current state, and ABCD are weight matrices that can be obtained through learning. Subsequently, Albert Gu et al. proposed the structured state space S4 model based on SSM. The core idea of ​​Mamba is to transform the sequence modeling problem into a state space model. By introducing a selection mechanism, the model can dynamically adjust the evolution of the state according to the input sequence. This selection mechanism can effectively filter out irrelevant information while retaining necessary information, thereby achieving efficient modeling of long sequences. It can achieve performance matching that of Transformer in many language tasks, and has linear complexity and higher reasoning throughput.

[0004] How to achieve high-precision and fast speech segmentation in images has become a technical problem that needs to be solved. Summary of the invention

[0005] The purpose of the present invention is to provide a Mamba-based image semantic segmentation method and system in order to overcome the defects of the above-mentioned prior art.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to one aspect of the present invention, a method for image semantic segmentation based on Mamba is provided, the method comprising the following steps:

[0008] Step S1, inputting the original image into a feature extraction module to extract features, wherein the feature extraction module uses a Mception network, and the Mception network adds multiple Mamba submodules on the basis of the Xception network;

[0009] Step S2, inputting the first downsampling result of the features extracted in step S1 into the improved RASPP module to enhance the fusion of spatial context information, and then summarizing the global features through the R-Mamba module to enhance the long-distance dependency of the image;

[0010] Step S3, inputting the result of the second downsampling of the features extracted in step S1 into the improved RASPP;

[0011] Step S4, upsampling the result in step S3 and restoring the size to the same as the result in step S2;

[0012] Step S5, concatenating and fusing the result of step S2 and the result of step S4;

[0013] Step S6, adjusting the number of channels of the result of step S5 and then upsampling to restore the image size, and outputting the predicted image.

[0014] Preferably, the Mception network includes three parts connected in sequence, in the first part, the Mamba submodule is passed at least once before each downsampling, in the second part, the Mamba submodule is repeated multiple times, and in the third part, each convolution is immediately input into the Mamba submodule.

[0015] Preferably, the specific calculation of the Mamba submodule is:

[0016] input=LN(input)

[0017] input1=SiLu(Linear(input))

[0018] input2=SSM(SiLu(f DWConv (Linear(input))))

[0019] output=input+Linear(input1×input2)

[0020] Where input represents the input image features, and output represents the output image features; represents 1×1 convolution, f DWConv Denotes depthwise separable convolution, LN denotes normalization, SiLu denotes activation function, SSM denotes state space model, and Linear denotes linear projection.

[0021] Preferably, the R-Mamba module adds residual connection and layer normalization based on Mamba.

[0022] Preferably, the improved RASPP module replaces the original convolution with a separate convolution and uses a ReLu operation after each convolution.

[0023] More preferably, the improved RASPP module inputs the feature map into multiple different channels for processing: in the first channel, only one convolution is used to adjust the number of channels; in the second channel, the feature map is passed through a separate convolution; in the subsequent channels, the feature map is combined with the result of the previous channel, and then multiple separate convolution calculations are performed in sequence; finally, the outputs in all channels are dimensionalized and then spliced ​​and fused.

[0024] More preferably, the calculation of the improved RASPP module is specifically as follows:

[0025]

[0026] map out =map 1 / 4 +MLP(LN(f concat (x1,x2,x3,x4)))

[0027] Among them, map 1 / 4 It represents the feature map of the original image after 4 times downsampling. They represent 3×1 convolution, 1×3 convolution and ReLu operations with expansion rates of 6, 12 and 18 respectively. MLP represents multi-layer perceptron. f concat represents concatenation operation, and LN represents normalization processing.

[0028] According to another aspect of the present invention, a Mamba-based image semantic segmentation system is provided, which includes an image acquisition preprocessing module and a segmentation module, wherein the segmentation module includes an encoder and a decoder, and the image acquisition preprocessing module is used to preprocess the collected original image, and then input it into the encoder. It is characterized in that the encoder includes a feature extraction module based on the Mception network and an improved RASPP module; the decoder includes an R-Mamba module.

[0029] Preferably, the feature extraction module adds multiple Mamba submodules based on the Mception network; and the R-Mamba module adds residual connection and layer normalization based on Mamba.

[0030] Preferably, the improved RASPP module replaces the original convolution with a separation convolution, and uses a ReLu operation after each convolution;

[0031] The improved RASPP module passes the feature map into multiple different channels respectively: in the first channel, only one convolution is used to adjust the number of channels; in the second channel, the feature map passes a separation convolution; in the subsequent channels, the feature map is combined with the result of the previous channel, and then multiple separation convolution calculations are performed in sequence; finally, the outputs in all channels are dimensionalized and then spliced ​​and fused.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1) The present invention newly adds the Mamba structure to the Mception network to effectively obtain longer-distance context information and reduce the computational complexity; features of different scales are processed by the improved RASPP module, and the improved RASPP module fuses feature maps of different scales to improve the accuracy of small object and edge segmentation, thereby enhancing the accuracy of semantic segmentation.

[0034] 2) The Mception structure of the present invention uses Mamba to update the traditional convolution, which can effectively capture the long-term dependencies in long sequences. When the length of the processed data sequence increases, it does not grow exponentially, but linearly, which reduces the complexity and speeds up the semantic segmentation.

[0035] 3) The improved RASPP module in the present invention replaces the original convolution with a separation convolution and uses a ReLu operation after each convolution to enhance its ability to learn complex patterns from input data, which can effectively reduce the number of parameters and improve feature extraction capabilities and training speed.

[0036] 4) The present invention introduces R-Mamba in the encoder. R-Mamba adds residual connection and layer normalization on the basis of Mamba to enhance the spatial modeling ability of the original Mamba, which is used to summarize the global features in the RASPP module and capture the long-distance dependencies of the image, thereby further enhancing the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the structure of the Mamba submodule in the present invention;

[0038] Figure 2 It is a schematic diagram of the structure of the R-Mamba module in the present invention;

[0039] Figure 3 It is a schematic diagram of the structure of the Mception module in the present invention;

[0040] Figure 4 It is a structural schematic diagram of the RASPP module in the present invention;

[0041] Figure 5 It is a structural schematic diagram of the overall model in the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0043] Example 1

[0044] This embodiment relates to a Mamba-based image semantic segmentation method, which aims to obtain long-range features to better segment small objects and segmentation edges with higher accuracy while reducing computational complexity.

[0045] like Figure 5 , the specific steps of this method are as follows:

[0046] Step 1: Input the original image into the Mception feature extraction module for feature extraction.

[0047] Step 2: The feature extracted in step 1 is downsampled by 4 times and input into the improved RASPP, and then passes through the R-Mamba module;

[0048] Step 3: Downsample the features extracted in step 1 by 16 times and input them into the improved RASPP.

[0049] Step 4: Upsample the result in step 3 by 4 times and restore the size to the same size as the result in step 2.

[0050] Step 5: concatenate and fuse the results of step 2 and step 4.

[0051] Step 6: Adjust the number of channels of the result of step 5 and perform 4-fold upsampling to restore the image size and output the predicted image.

[0052] Step 1 includes: inputting the image into the feature extraction module, which uses the Mception network. Mception is based on Xception and adds a Mamba submodule. A total of 35 Mamba submodules have been added, which are mainly used to extract long-distance information.

[0053] The picture Input feature extraction network Mception. Figure 3The Mception network consists of three parts connected in sequence. In the first part, the Mamba submodule is passed twice before each downsampling. In the second part, the Mamba submodule is repeated 24 times. In the third part, each convolution is immediately input into Mamba, and finally downsampled 16 times before output. The calculation formula of the Mamba submodule is as follows:

[0054] input=LN(input)

[0055] input1=SiLu(Linear(input))

[0056] input2=SSM(SiLu(f DWConv (Linear(input))))

[0057] output=input+Linear(input1×input2)

[0058] Where input represents the input signal and output represents the output signal. represents 1×1 convolution, f DWConv Denotes depthwise separable convolution, LN denotes normalization, SiLu denotes activation function, SSM denotes state space model, and Linear denotes linear projection.

[0059] like Figure 1 In the Mamba submodule, the data is first normalized by layers to make the data distribution more stable, and then it is subjected to two linear projections so that the model can process higher-dimensional feature spaces and capture more detailed and complex features. The depthwise separable convolution after the linear projection focuses on extracting spatial features within each channel, while the SSM is responsible for processing these features and capturing long-term dependencies in sequence data. The features obtained by the depthwise separable convolution and SSM complement each other. Finally, they are added together through residual connections to enhance feature representation.

[0060] The result of step 1 is downsampled and input into the improved RASPP module of the present invention, and the improved RASPP module can effectively enhance the fusion of spatial context information.

[0061] Step 2: The features extracted in step 1 are downsampled by 4 times and then passed through the improved RASPP module of the present invention. The improved RASPP module can effectively enhance the fusion of spatial context information. Figure 4 , the original 3×3 convolution is replaced by a separated convolution, which becomes two convolutions of 3×1 and 1×3, which can effectively reduce the number of parameters and improve the feature extraction capability. ReLu is used after each convolution to improve the training and reasoning speed. The calculation formula is as follows:

[0062]

[0063] map out =map 1 / 4 +MLP(LN(f concat (x1,x2,x3,x4)))

[0064] Among them, map 1 / 4 It represents the feature map of the original image after 4 times downsampling. They represent 3×1 convolution, 1×3 convolution and ReLu operations with expansion rates of 6, 12 and 18 respectively. MLP represents multi-layer perceptron. f concat represents concatenation operation, and LN represents normalization processing.

[0065] The feature map after Mception processing contains rich information of the image, but it needs further processing. The feature map enters 4 different channels respectively: In the first channel, only a 1×1 convolution is used to adjust the number of channels for subsequent fusion operations of different features. In the second channel, the feature map is passed through a 3×1 and 1×3 dilated convolution with a dilation rate of 6 to extract features under a larger receptive field. In the third channel, the feature map is combined with the result of the second channel, and then 3×1, 1×3 dilated convolutions with a dilation rate of 6 and 3×1, 1×3 dilated convolutions with a dilation rate of 12 are calculated in sequence. In the fourth channel, the feature map is combined with the result of the third channel, and then 3×1, 1×3 dilated convolutions with dilation rates of 6, 12 and 18 are calculated in sequence. The calculations in the third and fourth channels are all to obtain a larger receptive field and obtain richer contextual information. The output of each convolution block will be activated by ReLU to enhance its ability to learn complex patterns from the input data. Finally, the outputs from the first, second, third, and fourth channels are concatenated after dimensionality adjustment to form a richer feature representation. The combined layer normalization module is used to make the training process converge faster, while the MLP is used to adjust the dimension.

[0066] like Figure 2 ,R-Mamba adds residual connections and layer normalization on the basis of Mamba to enhance the spatial modeling ability of the original Mamba. It is used to summarize the global features in the RASPP module and capture the long-range dependencies of images, especially marine images.

[0067] Step 5: Concatenate the results of step 4 with the results of step 2. This part of the result obtains both the spatial semantic information when downsampled by 4 times and the spatial semantic information when downsampled by 16 times. By combining the two, the semantic information is enriched. The detailed information of low-level features can be used to assist the extraction of high-level features, and the semantic information of high-level features can be used to correct errors that may be caused by low-level features.

[0068] Step 6: Perform a 3×3 convolution on the result of step 5 and adjust the number of channels to the number of categories. Finally, perform a 4x upsampling to restore the spatial resolution in order to improve the accuracy and reliability of segmentation.

[0069] Example 2

[0070] This embodiment also relates to a Mamba-based image semantic segmentation system, including an image acquisition preprocessing module and a segmentation module, wherein the segmentation module includes an encoder and a decoder. The image acquisition preprocessing module is used to preprocess the collected original image and then input it into the encoder.

[0071] The encoder is used to extract features from the input image, including a feature extraction module and two improved RASPP modules. The feature extraction module uses the Mception network, in which the Mception network adds multiple Mamba submodules based on the Xception network.

[0072] The improved RASPP module replaces the original convolution with a separate convolution, such as Figure 4 , the original 3×3 convolution is replaced by a separated convolution, which becomes two convolutions of 3×1 and 1×3, which can effectively reduce the number of parameters and improve the feature extraction capability. ReLu is used after each convolution to improve the training and inference speed.

[0073] The improved RASPP module allows feature maps to enter multiple different channels separately: in the first channel, only one convolution is used to adjust the number of channels; in the second channel, the feature map passes through a separate convolution; in subsequent channels, the feature map is combined with the result of the previous channel, and then multiple separate convolution calculations are performed in sequence; finally, the outputs in all channels are dimensionalized and concatenated.

[0074] The calculation of the improved RASPP module is as follows:

[0075]

[0076] map out =map 1 / 4 +MLP(LN(f concat (x1,x2,x3,x4)))

[0077] Among them, map 1 / 4 It represents the feature map of the original image after 4 times downsampling. They represent 3×1 convolution, 1×3 convolution and ReLu operations with expansion rates of 6, 12 and 18 respectively. MLP represents multi-layer perceptron. f concat represents concatenation operation, and LN represents normalization processing.

[0078] The decoder is used to fuse and splice the feature maps of different scales output by the encoder and output the image segmentation result. The decoder includes the R-Mamba module and upsampling. The R-Mamba module adds residual connection and layer normalization on the basis of Mamba.

[0079] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A Mamba-based image semantic segmentation method, characterized in that: The method comprises the following steps: Step S1, inputting the original image into a feature extraction module to extract features, wherein the feature extraction module uses a Mception network, and the Mception network adds multiple Mamba submodules on the basis of the Xception network; Step S2, inputting the first downsampling result of the features extracted in step S1 into the improved RASPP module to enhance the fusion of spatial context information, and then summarizing the global features through the R-Mamba module to enhance the long-distance dependency of the image; Step S3, inputting the result of the second downsampling of the features extracted in step S1 into the improved RASPP; Step S4, upsampling the result in step S3 and restoring the size to the same as the result in step S2; Step S5, concatenating and fusing the result of step S2 and the result of step S4; Step S6, adjusting the number of channels of the result of step S5 and then upsampling to restore the image size, and outputting the predicted image.

2. The image semantic segmentation method based on Mamba according to claim 1, characterized in that: The Mception network includes three parts connected in sequence. In the first part, the Mamba submodule is passed at least once before each downsampling. In the second part, the Mamba submodule is repeated multiple times. In the third part, each convolution is immediately input into the Mamba submodule.

3. The image semantic segmentation method based on Mamba according to claim 1, characterized in that: The specific calculation of the Mamba submodule is: input=LN(input) input1=SiLu(Linear(input)) input2=SSM(SiLu(f DWConv (Linear(input)))) output=input+Linear(input1×input2) Where input represents the input image features, and output represents the output image features; represents 1×1 convolution, f DWConv Denotes depthwise separable convolution, LN denotes normalization, SiLu denotes activation function, SSM denotes state space model, and Linear denotes linear projection.

4. The image semantic segmentation method based on Mamba according to claim 1, characterized in that: The R-Mamba module adds residual connections and layer normalization based on Mamba.

5. The Mamba-based image semantic segmentation method according to claim 1, characterized in that: The improved RASPP module replaces the original convolution with a separate convolution and uses a ReLu operation after each convolution.

6. The Mamba-based image semantic segmentation method according to claim 5, characterized in that: The improved RASPP module inputs the feature map into multiple different channels for processing: in the first channel, only one convolution is used to adjust the number of channels; in the second channel, the feature map is passed through a separate convolution; in the subsequent channels, the feature map is combined with the result of the previous channel, and then multiple separate convolution calculations are performed in sequence; finally, the outputs in all channels are dimensionalized and spliced ​​and fused.

7. The Mamba-based image semantic segmentation method according to claim 6, characterized in that: The calculation of the improved RASPP module is specifically as follows: map out =map 1 / 4 +MLP(LN(f concat (x1,x2,x3,x4))) Among them, map 1 / 4 It represents the feature map of the original image after 4 times downsampling. They represent 3×1 convolution, 1×3 convolution and ReLu operations with expansion rates of 6, 12 and 18 respectively. MLP represents multi-layer perceptron. f concat represents the concatenation operation, and LN represents the normalization process.

8. A system using the Mamba-based image semantic segmentation method according to any one of claims 1 to 7, the system comprising an image acquisition preprocessing module and a segmentation module, wherein the segmentation module comprises an encoder and a decoder, and the image acquisition preprocessing module is used to preprocess the collected original image and then input it into the encoder, characterized in that: The encoder includes a feature extraction module based on the Mception network and an improved RASPP module; the decoder includes an R-Mamba module.

9. The system according to claim 8, characterized in that The feature extraction module adds multiple Mamba submodules based on the Mception network; the R-Mamba module adds residual connection and layer normalization based on Mamba.

10. The system according to claim 8, characterized in that The improved RASPP module replaces the original convolution with a separate convolution and uses a ReLu operation after each convolution; The improved RASPP module passes the feature map into multiple different channels respectively: in the first channel, only one convolution is used to adjust the number of channels; in the second channel, the feature map passes a separation convolution; in the subsequent channels, the feature map is combined with the result of the previous channel, and then multiple separation convolution calculations are performed in sequence; finally, the outputs in all channels are dimensionalized and then spliced ​​and fused.

Images