Visual Mama and multi-scale feature fused image segmentation network construction method

Abstract

The invention discloses an image segmentation network construction method fusing visual Mama and multi-scale features, and the method comprises the steps: designing an MFVM-Unet based on a U-Net network structure, and enabling the MFVM-Unet to serve as an image segmentation network; wherein the MFVM-Unet comprises an encoder, a decoder and a multi-scale feature information guiding module; the encoder is used for gradually extracting multi-level features of an input image; the decoder is used for recovering the spatial resolution of the feature map; and the multi-scale feature information guiding module is used for integrating multi-level features. According to the method, the extraction capability of local and global features can be enhanced, and the segmentation precision of a complex-form region is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for constructing an image segmentation network integrating visual Mamba and multi-scale features. Background Art

[0002] With the rapid development of computer vision technology, automated segmentation methods have shown high efficiency and consistency in processing large-scale image data. Such methods can not only effectively reduce the uncertainty caused by human operations, but also significantly improve the accuracy and efficiency of segmentation, which has important practical value and significance.

[0003] At present, state space models (SSMs) represented by Mamba have shown excellent performance in capturing long-range dependencies. The Mamba network not only has a strong global information modeling capability, but also significantly reduces the computational complexity through parallel computing, achieving a balance between performance and efficiency. Among them, VMamba further improves its performance in two-dimensional image processing tasks by introducing a two-dimensional selective scanning (2D Selective Scan, SS2D) module in the visual state-space (Visual State-Space, VSS) module. However, the fixed scanning strategy of the VMamba network focuses more on the integration of local features and directional information when processing images. Especially when dealing with complex forms or large-scale target areas, the flexibility of its global feature modeling will be limited to a certain extent, and it cannot fully cope with the diversity challenges in the field of image segmentation.

[0004] Therefore, in order to solve the above problems, a method for constructing an image segmentation network that integrates visual Mamba and multi-scale features is needed, which can enhance the ability to extract local and global features and significantly improve the segmentation accuracy of complex morphological areas. Summary of the invention

[0005] In view of this, the purpose of the present invention is to overcome the defects in the prior art and provide an image segmentation network construction method that integrates visual Mamba and multi-scale features, which can enhance the ability to extract local and global features and significantly improve the segmentation accuracy of complex morphological areas.

[0006] The image segmentation network construction method of the present invention integrating visual Mamba and multi-scale features includes:

[0007] Designing MFVM-Unet based on the U-Net network structure, and using the MFVM-Unet as an image segmentation network;

[0008] The MFVM-Unet includes an encoder, a decoder and a multi-scale feature information guidance module;

[0009] The encoder is used to gradually extract multi-level features of the input image;

[0010] The decoder is used to restore the spatial resolution of the feature map;

[0011] The multi-scale feature information guiding module is used to integrate multi-level features.

[0012] Furthermore, the multi-level features of the input image are gradually extracted, including:

[0013] First, the input image is processed by the multi-channel visual state space (M-VSS) module to generate a feature map of size H×W×64;

[0014] Subsequently, downsampling and feature extraction operations are performed layer by layer to generate feature maps of sizes H / 2×W / 2×128, H / 4×W / 4×256, and H / 8×W / 8×512, respectively;

[0015] In the last layer of the encoder, a deep feature map of size H / 16×W / 16×1024 is generated through downsampling and residual modules; the residual module in the network consists of two layers of 3×3 convolutional layers with batch normalization, and the input and output are added using residual connections, and nonlinear transformation is performed through the ReLU activation function.

[0016] Further, the multi-scale feature information guidance (MFG) module includes a semantic and detail injection (SDI) module and a feature fusion attention (FFA) module;

[0017] The SDI module is used to perform multi-scale fusion of the outputs of different levels of the encoder, and the results are then passed to the FFA module. The FFA module integrates the features output by the SDI module, the initial features output by the encoder, and the segmentation mask generated by the decoder, and passes them to the decoder.

[0018] Furthermore, the spatial resolution of the feature map is restored, including:

[0019] After each upsampling, the feature map is added element-by-element with the output of the FFA module and further processed by the residual module; at the same time, the segmentation mask generated by the decoder is input into the shallower FFA module during the upsampling process. The SDI and FFA modules establish a close connection between the encoder and decoder.

[0020] Furthermore, the M-VSS module:

[0021] The input image first generates a preliminary feature map through a local residual block, and then the feature map is layer-normalized and divided into two branches according to the number of channels. In the first branch, the preliminary feature map is processed by a linear layer and a SiLU activation function. In the second branch, the preliminary feature map is processed by a linear layer, a depth-wise separable convolution and a SiLU activation function, and then input into a multi-channel two-dimensional selective scanning (M-SS2D) module for further feature extraction. The extracted feature map is layer-normalized and then fused by element-wise multiplication with the feature map processed by the first branch. Finally, the fused feature map is summed through a linear layer and a residual connection to generate the final output.

[0022] Furthermore, the M-SS2D module:

[0023] First, the preliminary feature map is divided into four independent parts according to the channel, and each part is input into its own two-dimensional selective scanning (SS2D) module; the SS2D module includes three components: 4-way cross scanning, S6 module and 4-way cross merging;

[0024] Subsequently, the feature maps output by each SS2D module are further subjected to feature extraction through dilated convolutions with different dilation rates;

[0025] Finally, the feature maps generated by convolutions with different dilation rates are concatenated and fused to form the final feature map.

[0026] Furthermore, the FFA module:

[0027] Map the features output by the SDI module The initial feature f output by the i-th layer encoder i 0 And the segmentation mask map output by the i+1 layer decoder As input;

[0028] The segmentation mask image is upsampled using linear interpolation to match the size;

[0029] Segmentation mask map after matching size Apply the decomposition method to generate two different attention feature maps: reverse attention map and edge attention map Among them, the reverse attention map is used to re-evaluate and refine the areas that were not noticed in the prediction; the edge attention map uses the Laplacian pyramid method to strengthen the edge information in the feature map;

[0030] The reverse attention map and edge attention map Respectively with the feature map f i sMultiply pixel by pixel to get two attention feature maps, and then add the two attention feature maps and feature map f i s Combined features obtained by splicing and fusion

[0031] The attention mask is introduced to guide the network to focus on the key area and suppress background noise and redundant information, where the attention mask Α i and the attention feature map f i a for:

[0032] Among them, Conv(.) represents a 3×3 convolution operation, and the symbol σ represents the Sigmoid function;

[0033] The initial feature f generated by the encoder i 0 and the attention feature map f i a The concatenation is performed, and then the number of channels is adjusted through the convolution operation to adapt it to the input of the i-th layer decoder to obtain the final fine features.

[0034] The beneficial effects of the present invention are as follows: the present invention discloses a method for constructing an image segmentation network integrating visual Mamba and multi-scale features, which integrates a multi-channel visual state space (M-VSS) module and a multi-scale feature information guidance (MFG) module in the classic U-Net architecture, the former performs well in modeling local and global information, and the latter can enhance the ability to distinguish between the target area and the background by integrating feature information of different depths. The M-VSS module combines the advantages of convolutional neural networks in local feature extraction with the ability of multi-channel two-dimensional selective scanning M-SS2D modules in global information capture, and enhances the ability to extract local and global information on the basis of optimizing computational complexity. The MFG module effectively integrates feature information of different depths, further improving the robustness of segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0036] Figure 1 This is the overall structure diagram of the MFVM-UNet network of the present invention;

[0037] Figure 2 (a) is a schematic diagram of the M-VSS module of the present invention;

[0038] Figure 2 (b) is a schematic diagram of the M-SS2D module of the present invention;

[0039] Figure 3 It is a schematic diagram of the SS2D module of the present invention;

[0040] Figure 4 (a) is a schematic diagram of an SDI module of the present invention;

[0041] Figure 4 (b) is a schematic diagram of the FFA module of the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with the accompanying drawings, as shown in the drawings:

[0043] This embodiment discloses a method for constructing an image segmentation network integrating visual Mamba and multi-scale features, including:

[0044] Designing MFVM-Unet based on the U-Net network structure, and using the MFVM-Unet as an image segmentation network;

[0045] The MFVM-Unet includes an encoder, a decoder and a multi-scale feature information guidance module;

[0046] The encoder is used to gradually extract multi-level features of the input image;

[0047] The decoder is used to restore the spatial resolution of the feature map;

[0048] The multi-scale feature information guiding module is used to integrate multi-level features.

[0049] The present invention constructs a U-type network (MFVM-UNet) that integrates visual Mamba and multi-scale feature information. The multi-channel visual state space (M-VSS) module in the network effectively combines the convolutional neural network and the multi-channel two-dimensional selective scanning (M-SS2D) module, significantly improving the ability to extract local and global information, and excels in perceiving complex forms and scattered tiny areas. In addition, the multi-scale feature information guidance (MFG) module enhances the network's segmentation ability under low contrast and high noise conditions by integrating multi-level features, further improving the segmentation accuracy of the target area.

[0050] In this embodiment, the overall architecture of MFVM-UNet proposed by the present invention is as follows Figure 1 As shown in Figure 2, MFVM-UNet is based on the structural design of U-Net and consists of three main parts: encoder, decoder and multi-scale feature information guidance module.

[0051] The encoder gradually extracts multi-level features of the input image through the M-VSS module and step-by-step downsampling operations. First, the input image is processed by the initial M-VSS module to generate a feature map of size H×W×64. Subsequently, downsampling and feature extraction operations are performed layer by layer to generate feature maps of size H / 2×W / 2×128, H / 4×W / 4×256, and H / 8×W / 8×512. In the last layer of the encoder, a deep feature map of size H / 16×W / 16×1024 is generated through downsampling and the residual block. The residual block in the network consists of two layers of 3×3 convolutional layers with batch normalization, using residual connections to add the input and output, and performing nonlinear transformations through the ReLU activation function.

[0052] In order to further improve the feature fusion capability, the present invention replaces the traditional skip connection with a multi-scale feature information guidance (Multi-scale Feature Guidance, MFG) module. The module includes a semantics and detail injection (Semantics and Detail Infusion, SDI) module and a feature fusion attention (Feature Fusion Attention, FFA) module. The SDI module is responsible for multi-scale fusion of the outputs of different levels of the encoder, and the results are then passed to the FFA module. The FFA module further improves the feature resolution capability by integrating the features output by the SDI module, the initial features output by the encoder, and the segmentation mask generated by the decoder, and passes it to the decoder.

[0053] The decoder restores the spatial resolution of the feature map by gradually upsampling. After each upsampling, the feature map is element-wise summed with the output of the FFA module and further processed by the Residual Block. At the same time, the segmentation mask generated by the decoder is input into the shallower FFA module during the upsampling process. The SDI and FFA modules establish a close connection between the encoder and decoder, optimize feature transfer and fusion, and effectively improve the accuracy and robustness of segmentation.

[0054] In this embodiment, the core SS2D module of VMamba has insufficient flexibility in global feature modeling and limitations in dealing with complex morphology and large-scale capture. The present invention proposes an M-SS2D module to enhance the global feature integration capability and the detail capture effect of the target area. On this basis, the M-VSS module is proposed, and its structure is as follows: Figure 2 (a) shown.

[0055] The input image is first processed through a local residual block (ResBlock) to generate a preliminary feature map. The feature map is then layer normalized and divided into two branches based on the number of channels. In the first branch, the preliminary feature map is processed by a linear layer and a SiLU activation function. In the second branch, the preliminary feature map is processed by a linear layer, a depthwise separable convolution, and a SiLU activation function, and then input into the M-SS2D module for further feature extraction. The extracted feature map is layer normalized and then fused by element-wise multiplication with the feature map processed by the first branch. Finally, the fused feature map is summed through a linear layer and a residual connection to generate the final output.

[0056] In this embodiment, the overall process of the M-SS2D module is as follows: Figure 2 Specifically, the preliminary feature map is first divided into four independent parts by channel, and each part is input into its own SS2D module. SS2D consists of three components: 4-way cross scanning, S6 module and 4-way cross merging, as shown in Figure 3 shown.

[0057] Through the multi-channel division method, the network can capture diverse feature maps in parallel on different channels and improve the feature expression ability. Subsequently, the feature map output by each SS2D module is further extracted through dilated convolution (Dilated Conv2d) with different dilation rates (d=1, 2, 5, 7). Dilated convolution can expand the receptive field without increasing the computational complexity, effectively enhancing the ability to capture contextual information. Finally, the feature maps generated by convolutions with different dilation rates are spliced ​​and fused to form the final feature map.

[0058] The M-VSS module and M-SS2D module enhance the local and global feature modeling capabilities through residual connection, multi-channel feature division, cross scanning and dilated convolution mechanism, and can capture the detailed information of the lesions more comprehensively.

[0059] In this embodiment, the multi-scale feature information guided MFG module includes an SDI module and an FFA module. The SDI module is mainly responsible for integrating the encoder's different scale features to enhance the network's perception of multi-scale information. The FFA module further enhances and filters these multi-scale features to extract more representative feature information, thereby improving the robustness of the network.

[0060] For the Semantics and Detail Injection (SDI) module:

[0061] The overall architecture of the SDI module is as follows Figure 4 (a) shows the module obtains each layer of input feature map from the encoder Where j represents the jth layer, H, W and C represent the height, width and number of channels of the input image respectively. Then, the channel and spatial attention mechanisms are applied to each layer of feature maps respectively, so as to effectively enhance the feature representation capability. Then, the 1×1 convolution operation is used to convert the output of each layer into Align the number of channels and get the corresponding target mapping The present invention sets C to 64 by default.

[0062] Next, the feature map sizes of the outputs of different layers are matched to the size of the i-th layer of the decoder (1≤i≤4).

[0063] Specifically, the size of the j-th layer feature map is adjusted to correspond to f i 2 The matching formula is as follows:

[0064]

[0065] Among them, down , Iden and up Represent downsampling, identity mapping and upsampling operations respectively, the purpose is to The resolution is adjusted to H i ×W i .

[0066] Finally, a 3×3 convolution is used on the resized 4 feature maps. Smoothing is performed. Then the four feature maps are multiplied pixel by pixel, as shown in formula (5). This operation can enhance the feature interaction ability and help improve the nonlinear expression ability and information fusion effect of the network.

[0067]

[0068] Among them, f i s Refers to the feature fusion result of the i-th layer, and then the fusion result is sent to the FFA module of the corresponding layer for the next step.

[0069] For the Feature Fusion Attention (FFA) module:

[0070] The present invention improves the EGA module in MEGANet and proposes the FFA module, whose overall architecture is as follows: Figure 4 (b) The module input includes the feature map output from SDI The initial feature f output by the i-th layer encoder i 0 And the segmentation mask map output by the deeper (i+1 layer) decoder

[0071] Since the deeper segmentation mask The size and feature map f i s If the image does not match, the segmentation mask image is first upsampled using linear interpolation to match the size. Apply the decomposition method to generate two different attention feature maps: reverse attention map and edge attention map The formulas are shown in (6) and (7):

[0072]

[0073] Among them, g is a convolution operator with Gaussian filtering, d represents a downsampling operation, and u represents an upsampling operation. The reverse attention map can re-evaluate and refine the unnoticed areas in the prediction, thereby reducing the prediction error and improving the prediction accuracy. The edge attention map uses the Laplacian pyramid method to strengthen the edge information in the feature map, thereby further improving the network's performance in edge detection and detail preservation.

[0074] Then, and With the feature map f i s Multiply pixel by pixel. Then the two attention feature maps after the operation and f i s Combined features obtained by splicing and fusion The formula is as follows:

[0075]

[0076] Among them, [.] represents concatenation, and Conv(.) means fusing the concatenated results into a combined feature with C channels through a 1×1 convolution operation.

[0077] Since edge information may contain noise and redundant details that are useless for segmentation, an attention mask is introduced here to guide the network to focus on key areas and suppress background noise and redundant information. The formulas for the attention mask and attention feature map are defined as follows:

[0078]

[0079] Among them, Conv(.) represents a 3×3 convolution operation, and the symbol σ represents the Sigmoid function.

[0080] Since the initial image features contain a lot of fine-grained information, which may be lost in deep convolution operations, the present invention chooses to convert the f generated by the encoder into i 0 and the attention feature map fi a Then, the number of channels is adjusted through convolution operation to adapt it to the input of the i-th layer decoder to obtain the final fine features. The formula is defined as follows:

[0081] f i d =Conv([f i 0 ,f i a ]) (10)

[0082] Among them, Conv(.) means that the concatenated results are fused into a channel number C through a 1×1 convolution operation i combination characteristics.

[0083] In addition, to improve the accuracy of pixel classification and the integrity of segmented shapes, the present invention combines binary cross entropy (BCE) loss and Dice loss. BCE loss is used to process unbalanced data at the pixel level, while Dice loss helps to ensure that the overall shape of the overall segmented area is more accurate. Therefore, the weighted loss function L BceDice It is given by the following formula:

[0084] L BceDice =λ1L Bce +λ2L Dice (11)

[0085]

[0086] Among them, y i is the true label of the ith pixel, is the predicted label of the ith pixel, N is the total number of pixels, X and Y represent the predicted and true segmented regions, respectively. λ1 and λ2 are the weights of the loss function, and the default weights can be set to 1.

[0087] Experimental results show that the DSC, JI, SE, ACC and MCC indicators of the MFVM-Unet network of the present invention on a certain dataset reached 89.30%, 80.66%, 88.74%, 99.69% and 89.14% respectively, which is significantly better than the existing image segmentation network. In addition, the excellent performance of the network on other public datasets further proves its good generalization ability and practical application potential.

[0088] The image segmentation network constructed by the present invention can effectively improve the segmentation ability of complex and scattered areas in the image by combining the excellent long-range dependency modeling ability of the Mamba architecture with the local feature expression advantage of multi-scale feature fusion. In addition, while optimizing the computational complexity, the network enhances the processing ability of low contrast and high noise between the target area and the normal area in the image, thereby improving the overall segmentation accuracy.

[0089] The present invention proposes an optimized multi-channel visual state space (M-VSS) module and a multi-scale feature information guidance (MFG) module. The M-VSS module combines the advantages of convolutional neural networks in local feature extraction with the ability of multi-channel two-dimensional selective scanning (M-SS2D) modules in global information capture, and enhances the ability to extract local and global information on the basis of optimizing computational complexity. The MFG module further enhances the ability to distinguish between the target area and the background, and improves the robustness of segmentation.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for constructing an image segmentation network integrating visual Mamba and multi-scale features, characterized in that: include: Designing MFVM-Unet based on the U-Net network structure, and using the MFVM-Unet as an image segmentation network; The MFVM-Unet includes an encoder, a decoder and a multi-scale feature information guidance module; The encoder is used to gradually extract multi-level features of the input image; The decoder is used to restore the spatial resolution of the feature map; The multi-scale feature information guiding module is used to integrate multi-level features.

2. The method for constructing an image segmentation network integrating visual Mamba and multi-scale features according to claim 1, characterized in that: Gradually extract the multi-level features of the input image, including: First, the input image is processed by the multi-channel visual state space (M-VSS) module to generate a feature map of size H×W×64; Subsequently, downsampling and feature extraction operations are performed layer by layer to generate feature maps of sizes H / 2×W / 2×128, H / 4×W / 4×256, and H / 8×W / 8×512, respectively; In the last layer of the encoder, a deep feature map of size H / 16×W / 16×1024 is generated through downsampling and residual modules; the residual module in the network consists of two layers of 3×3 convolutional layers with batch normalization, and the input and output are added using residual connections, and nonlinear transformation is performed through the ReLU activation function.

3. The method for constructing an image segmentation network integrating visual Mamba and multi-scale features according to claim 1, characterized in that: The multi-scale feature information guidance (MFG) module includes a semantic and detail injection (SDI) module and a feature fusion attention (FFA) module; The SDI module is used to perform multi-scale fusion of the outputs of different levels of the encoder, and the results are then passed to the FFA module. The FFA module integrates the features output by the SDI module, the initial features output by the encoder, and the segmentation mask generated by the decoder, and passes them to the decoder.

4. The method for constructing an image segmentation network integrating visual Mamba and multi-scale features according to claim 3, characterized in that: Restore the spatial resolution of the feature map, including: After each upsampling, the feature map is added element-by-element with the output of the FFA module and further processed by the residual module; at the same time, the segmentation mask generated by the decoder is input into the shallower FFA module during the upsampling process. The SDI and FFA modules establish a close connection between the encoder and decoder.

5. The method for constructing an image segmentation network integrating visual Mamba and multi-scale features according to claim 2, characterized in that: The M-VSS module: The input image first generates a preliminary feature map through a local residual block, and then the feature map is layer-normalized and divided into two branches according to the number of channels. In the first branch, the preliminary feature map is processed by a linear layer and a SiLU activation function. In the second branch, the preliminary feature map is processed by a linear layer, a depth-wise separable convolution and a SiLU activation function, and then input into a multi-channel two-dimensional selective scanning (M-SS2D) module for further feature extraction. The extracted feature map is layer-normalized and then fused by element-wise multiplication with the feature map processed by the first branch. Finally, the fused feature map is summed through a linear layer and a residual connection to generate the final output.

6. The method for constructing an image segmentation network integrating visual Mamba and multi-scale features according to claim 5, characterized in that: The M-SS2D module: First, the preliminary feature map is divided into four independent parts according to the channel, and each part is input into its own two-dimensional selective scanning (SS2D) module; the SS2D module includes three components: 4-way cross scanning, S6 module and 4-way cross merging; Subsequently, the feature maps output by each SS2D module are further subjected to feature extraction through dilated convolutions with different dilation rates; Finally, the feature maps generated by convolutions with different dilation rates are concatenated and fused to form the final feature map.

7. The method for constructing an image segmentation network integrating visual Mamba and multi-scale features according to claim 3, characterized in that: The FFA module: Map the features output by the SDI module The initial feature f output by the i-th layer encoder i 0 And the segmentation mask map output by the i+1 layer decoder As input; The segmentation mask image is upsampled using linear interpolation to match the size; Segmentation mask map after matching size Apply the decomposition method to generate two different attention feature maps: reverse attention map and edge attention map Among them, the reverse attention map is used to re-evaluate and refine the areas that were not noticed in the prediction; the edge attention map uses the Laplacian pyramid method to strengthen the edge information in the feature map; The reverse attention map and edge attention map Respectively with the feature map f i s Multiply pixel by pixel to get two attention feature maps, and then add the two attention feature maps and feature map f i s Combined features obtained by splicing and fusion The attention mask is introduced to guide the network to focus on the key area and suppress background noise and redundant information, where the attention mask Α i and the attention feature map f i a for: Among them, Conv(.) represents a 3×3 convolution operation, and the symbol σ represents the Sigmoid function; The initial feature f generated by the encoder i 0 and the attention feature map f i a The concatenation is performed, and then the number of channels is adjusted through the convolution operation to adapt it to the input of the i-th layer decoder to obtain the final fine features.

Images