CNN and subtraction vision Mama mixed neural network medical image segmentation method fusing golden section proportion
By integrating a hybrid neural network of CNN and subtractive visual Mamba, the computational efficiency and accuracy issues of traditional CNN and Vision Transformer in medical image segmentation are solved, achieving efficient and accurate medical image segmentation and improving the model's computational efficiency and segmentation accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TAIYUAN UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2026-01-20
- Publication Date
- 2026-05-05
AI Technical Summary
Existing convolutional neural networks are limited by the receptive field of the convolutional kernel when processing medical images, making it difficult to effectively process long sequence information. Vision Transformer, due to the high computational complexity of its attention mechanism, has excessively high computational cost in lightweight tasks, resulting in limited efficiency and accuracy in medical image segmentation.
A novel medical image segmentation method is designed by employing a hybrid neural network that combines the golden ratio CNN with subtractive visual Mamba. Through the ESVM-Net neural network, a multi-level encoder and decoder, a cue word model, multi-kernel convolutional blocks, and an attention mechanism are combined. The subtractive visual Mamba model is used to learn background features, and multi-kernel convolutional blocks and decoders are combined to restore image resolution. The cue word model is used to guide network training and reduce redundant information.
It achieves efficient and accurate medical image segmentation with low computational cost, improving segmentation accuracy and computational efficiency, and even surpassing the performance of heavyweight models, with good generalization performance.
Smart Images
Figure CN121982308A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the intersection of medical image segmentation technology and computer-aided diagnosis. It uses deep learning computation to accurately and efficiently segment lesion areas in medical images to facilitate subsequent clinical medical diagnosis. Specifically, it is a medical image segmentation method that integrates a CNN based on the golden ratio and a subtractive visual Mamba hybrid neural network. Background Technology
[0002] Medical image segmentation technology plays a crucial role in clinical medical decision-making. Over the past decade, Convolutional Neural Networks (CNNs) and Vision Transformers have become two mainstream methods for solving such problems. For CNNs, due to the inherent nature of convolution, even with large kernels, it is difficult to clearly handle long-range dependencies. For Transformer methods, although attention mechanisms can establish long-range dependencies between pixels, their quadratic complexity leads to enormous computational and memory consumption. To address these issues, many researchers have designed state-of-the-art (SOTA) models that combine the advantages of Transformers and CNNs, such as UNET, TransUNet, and Swin UNETR. While these models exhibit good numerical performance, the attention mechanism within the Transformer still requires a sequence × sequence matrix to handle long-range dependencies in the image. Although convolution can reduce the size of feature maps, the complexity of the model increases rapidly with the size of large medical images, and repeated convolutions can lead to the network forgetting global dependencies. To overcome these difficulties, the State Space Model (SSM) was introduced as a module for constructing and analyzing long sequence neural networks, and Mamba (SSM) emerged. Considering its linear computational complexity and parallel training structure, Zhu et al. first designed the Vision Mamba model involving Mamba for image recognition. Without an attention mechanism, this model can complete inference on 1248×1248 images and save 86.8% of GPU memory, showing great potential in many fields. However, subsequent research found that when the 2D-selective-scan (SS2D) module of SSM operates on 2D images, the same attention to the global receptive field may introduce more redundant information unrelated to the target features. In addition, by introducing a higher-order 2D selective-scan (SS2D) to the method of Rao et al. to reduce the introduction of redundant information by the receptive field, better results were achieved. However, as the network depth increases, the low-scale feature maps of this model are prone to losing details of the original image. Summary of the Invention
[0003] This invention addresses the technical problem in the field of medical image segmentation that traditional CNNs are limited in their ability to process long sequences of information due to the inherent limitation of the receptive field of the convolution kernel, while Vision Transformer (Vit) has excessively high computational costs for lightweight downstream tasks due to the secondary computational complexity of the attention mechanism. The invention provides a medical image segmentation method that integrates a CNN based on the golden ratio and a subtractive visual Mamba hybrid neural network.
[0004] This invention is achieved using the following technical solution: a medical image segmentation method that integrates the golden ratio CNN and subtractive visual Mamba hybrid neural network, comprising the following steps:
[0005] S1. Preprocess the sample medical images: correct the different resolutions of the sample medical images to the same set resolution;
[0006] S2. Construct the ESVM-Net neural network, including the prompt word model, encoder network, decoder network, bottleneck layer, and skip connection processing module;
[0007] The encoder network consists of multi-level encoders, each of which uses a subtractive visual Mamba model. The visual Mamba model learns the background features of the image through a dual-path design, and then the extracted target features are processed by multi-kernel convolutional blocks.
[0008] The decoder network includes a multi-level decoder. Each level of the decoder uses a local feature extraction module constructed with multi-kernel convolutional blocks and skip connections with SA and CA attention mechanisms. Then, a channel attention mechanism is used to fuse the information of the encoder and the decoder.
[0009] S3. The processed image is fed into the prompt word model and encoder network. Except for the last level encoder, each level encoder extracts features from the preprocessed medical image and sends the extracted feature maps to the next level encoder and the skip connection processing module respectively. The last level encoder inputs the extracted feature maps to the first level decoder. During the encoder processing, the prompt word model is connected to the encoder network.
[0010] S4. Except for the first-level decoder, the other levels of decoders receive the input from the skip connection processing module and the input from the previous level decoder and perform reconstruction processing to obtain a feature map of the same size as the previous level decoder. Then, deconvolution is used to expand the size of the feature map. Finally, after processing by multiple levels of decoders, the feature map is restored to the size of the original image.
[0011] S5. Use a combination of BCEDiceLOSS and MCEloss loss functions to train the overall network;
[0012] S6. After training through steps S1-S5, the real-time image can be input into the model to obtain the final medical segmentation image.
[0013] Furthermore, in the ESVM-Net neural network, the prompt word model uses a pre-trained DenseNet, with all its parameters frozen during training. The output of the prompt word model is simultaneously connected to a five-level encoder. For each encoder level, the output of the prompt word model is divided into two paths. One path's output is fed into the corresponding encoder after being element-wise added to the other path's output through a bottleneck layer. The five-level encoder is sequentially connected via dual paths. The first-level encoder is used to input the pre-processed medical image. The first four encoder levels are connected to the decoder via a skip connection processing module. The fifth-level encoder is connected to the first-level decoder. The five decoders are connected sequentially. The previous level decoder is also connected to the next level decoder via a skip connection processing module. Each level decoder upsamples the received image through bilinear interpolation and then processes it through a local feature extraction module, SA, and CA to obtain a feature map.
[0014] Furthermore, in steps S3-S4, the feature map extraction specifically includes:
[0015] S31. The preprocessed medical image is processed through the prompt word model and the first-level encoder. The medical image generates feature image data C1_target and C2_background after passing through the first-level encoder. The target feature is input into the skip connection processing module. The target and background are input into the next level encoder and the output of the prompt word model is added. The final result is D1_target and D1_background.
[0016] S32. In the second encoder, the feature image data D1_target and D1_background are processed again by step S31 to obtain feature image data D2_target and D2_background;
[0017] S33. In the third encoder, the feature image data D2_target and D2_background are processed again by step S31 to obtain feature image data D3_target and D3_background;
[0018] S34. In the fourth encoder, the feature image data D3_target and D3_background are processed again by step S31 to obtain feature image data D4_target and D4_background;
[0019] S34. In the fifth encoder, the feature image data D4_target and D4_background are processed again by step S31 to obtain feature image data D5_target and D5_background;
[0020] S36. Input the feature image data D1_target, D2_target, D3_target, D4_target into the skip connection processing module for processing to obtain feature maps A1, A2, A3, A4 respectively;
[0021] S41. Input the feature image data D5_target into the first-level decoder for processing to obtain feature map B1;
[0022] S42. Input feature map B1 into the second-level decoder for processing, and then add the feature map extracted by the skip connection processing module to obtain feature map C1. The formula is C1=Decoder(B1)+A4.
[0023] S43. Repeat the above S42 operation in the remaining decoder stages until the image is restored to the resolution of the original image. This feature map is denoted as C4.
[0024] S44. Process feature map C4 using the Sigmoid function to obtain the final feature map.
[0025] Furthermore, the segmentation network in S5 is trained using the following loss function:
[0026]
[0027] in, To divide the network loss, , To set coefficients;
[0028]
[0029] in: n The total number of samples, y The label represents the real label, and the output of the segmentation network after processing the labeled image.
[0030] Furthermore, the encoder employs a subtractive Visual Mamba model (SVMB), which includes a Visual State Space Block (VSSB), a Point Convolutional Module (PWC), and a Local Feature Extraction Module (LFEM). VSSB and PWC serve as two parallel inputs to receive image feature data from the previous level. The output of VSSB is divided into two paths: one path is subtracted element-wise from the output of PWC and then input to LFEM; the other path is added element-wise to one output of LFEM and then output to the next level of SVMB. Simultaneously, the other output of LFEM is added element-wise to the output of the prompt word model and then output to the next level of SVMB.
[0031] The VSSB includes an LN and a Linear array that receive the output from the previous stage and are connected sequentially. The Linear output is divided into two paths. One path has SiLu (activation function), SS2D (2D selective scan module), and LN (layer normalization) sequentially according to the data flow. The other path is a Scale output. The two outputs are multiplied by Hadamard and then input into another Linear array. At the same time, the output from the previous stage is passed through a multi-kernel convolution block and then added element-wise with the output of another Linear array before being input into an LFEM and output by the LFEM.
[0032] The LFEM sequentially includes PWC, BN / ReLU (batch normalization / linear correction unit), multi-kernel residual convolution block MRCB, and BN and PWC. The first PWC accepts external input. The MRCB includes three sets of parallel depthwise separable convolution DWC and BN / ReLU connected to each depthwise separable convolution. The three sets of parallel structures perform element-wise addition of their own output results with the output results of BN / ReLU. The results of the above element-wise addition are then added element-wise again and input to BN and the second PWC. Finally, the output result of the second PWC is added element-wise with the output result of the previous stage accepted by the LFEM to obtain the final output of the LFEM.
[0033] The skip connection processing module adopts the efficient feature fusion attention block EFFA, which includes two parallel depthwise separable convolutions DWC and BN / ReLU. The output results of the two BN / ReLU are added element by element and the result is input to the efficient channel attention ECA. At the same time, the external input result is also output after performing a Hadamard product with the output result of ECA.
[0034] The efficient channel attention (ECA) includes a max pooling layer, an average pooling layer, PWC / BN, and a sigmoid function. The external input results are concatenated after passing through the max pooling layer and the average pooling layer, and then sequentially passed through the PWC / BN and sigmoid functions for output.
[0035] This invention discloses a novel prompt-guided medical image segmentation model called ESVM-Unet (Efficient Subtractive Vision Mamba Network), which enhances the model's ability to remember small targets by separating the background of the image. Furthermore, this invention designs a prompt-guided mechanism that continuously uses the original image as a cue word to guide the network during training, aiming to prevent deep modules from forgetting local details of the original image. Finally, to further improve the network's computational efficiency, this invention uses only convolutions to manipulate features during the decoding process. In summary, this invention designs various efficient convolutional modules to improve performance. By integrating these innovations, the model of this invention achieves a good balance between computational efficiency and segmentation accuracy, thus providing a lightweight model that even surpasses the performance of heavyweight models, and exhibits good generalization performance.
[0036] Beneficial effects of this invention:
[0037] 1. A novel cue-guided CNN-Mamba hybrid model architecture: Similar to LoRa fine-tuning, this invention freezes the weights of the pre-trained DenseNet model and uses efficient multi-kernel convolutional blocks to input the local details extracted by the pre-trained model into the main network. In the decoder stage, the model uses multi-scale residual convolutional blocks and various complex attention mechanisms to restore image resolution, effectively improving segmentation accuracy and computational efficiency. The main network of this invention adopts a novel top-down aggregation structure to allow the model to understand feature information from shallow to deep, thereby achieving accurate and efficient segmentation.
[0038] 2. Subtractive Visual Mamba module and multi-scale residual convolutional block: In the encoder stage, the VisionMamba model is used to learn the low-order structure of the image background to highlight key information of the image in reverse. This has a significant improvement in performance compared to the traditional visualmamba method of directly extracting image features.
[0039] 3. A novel learning rate scheduler: By adjusting the iteration step size using the golden ratio, the model converges faster. The update strategy proposed in this invention improves the DICE score of the same model by 1% to 2%.
[0040] 4. ESVM-net demonstrates state-of-the-art performance on three public datasets and one real-world clinical dataset, surpassing not only many advanced lightweight models but also, on some datasets, large vision models based on SAM. Attached Figure Description
[0041] Figure 1The overall architecture of ESVM-Net is as follows: (a) ESVM-Net: The cue word model uses a pre-trained DenseNet with all its parameters frozen during training. This cue word model provides local features to the main network through two convolutional layers. (b) Vision State Space Block (VSSB). (c) Subtractive Vision Mamba Block (SVMB). (d) Local Feature Extraction Module (LFEM) and Multiple-kernel Residual Convolutional Blocks (MRCB). (e) Efficient Channel Attention (ECA). (f) Efficient Feature Fusion Attention Block (EFFA), which fuses Channel Attention (CA) and Spatial Attention (SA). Pointwise Convolution (PWC) and Depthwise Convolution (DWC).
[0042] Figure 2 This is a visual comparison of various medical image segmentation methods on representative lesion images from four datasets in the example. Areas with obvious segmentation errors are marked with red boxes.
[0043] Figure 3 This is a flowchart of the method described in this invention. Detailed Implementation
[0044] The proposed ESVM-Net is implemented in PyTorch, and all experiments were run on a single NVIDIA RTX 3090 GPU with 24GB of VRAM. Horizontal and vertical flipping, as well as random rotation, were used to augment the dataset. To reduce training and inference time, all image sizes were uniformly set to 256×256. The multi-scale convolutional kernels in LFEM were [1, 3, 5]. The prompt model used a DenseNet model pre-trained on image1k. The number of channels in the model was [16, 32, 64, 96, 160], and the model was normalized before training. During training, AdamW was used as the optimizer, and a learning rate scheduler based on the golden ratio was used, with a learning rate and weight decay of 0.001 and 0.01, respectively. Training was performed for 250 epochs on each of the four datasets. The batch size was 8 for training on Clindb and 16 for the other datasets. Model weights were saved based on the highest Dice score during training. The segmentation map output by the model was calculated using a combined loss function of BCE(1) and DICE(1), and the original image reconstructed during the encoding stage was calculated using mseloss, with combination coefficients of 1, 1, and 0.01, respectively. Finally, to ensure fairness, publicly available model code was used and run under the same training framework, with parameters following the corresponding authors' papers.
[0045] To evaluate the contributions of each component in ESVM-Net, this invention conducted comprehensive ablation experiments. Specifically, the impact of each component was evaluated by removing each module from the model one by one. The specific data are shown below:
[0046] Table 1 Ablation Experiment Results
[0047] .
[0048] Ablation experiments were conducted on each component as shown in Table 1. SVMB and LFEM, as core components, showed the greatest improvement in segmentation accuracy (around 3%), which also demonstrates that the visual Mamba model has a stronger advantage in background modeling.
[0049] To evaluate the performance of the proposed ESVN-Net, this invention was quantitatively and qualitatively compared with several state-of-the-art (SOTA) medical image segmentation methods, including UNet, AttUNet, TransUNet, SwinUNet, MSLAU-Net, and UltraLight-VM-UNet, which are advanced models in the medical field. The experimental results are shown below.
[0050] Table 2. Quantitative comparison results across the four datasets.
[0051] .
[0052] Table 2 shows that the model of the present invention has shown strong competitiveness with extremely low parameters, reaching SOTA level on multiple datasets, and its segmentation accuracy is significantly better than models with hundreds of times more parameters on most datasets.
Claims
1. A medical image segmentation method that integrates the golden ratio CNN and subtractive visual Mamba hybrid neural network, characterized in that: Includes the following steps: S1. Preprocess the sample medical images: correct the different resolutions of the sample medical images to the same set resolution; S2. Construct the ESVM-Net neural network, including the prompt word model, encoder network, decoder network, bottleneck layer, and skip connection processing module; The encoder network consists of multi-level encoders, each of which uses a subtractive visual Mamba model. The visual Mamba model learns the background features of the image through a dual-path design, and then the extracted target features are processed by multi-kernel convolutional blocks. The decoder network includes multi-level decoders. Each level of decoder uses a local feature extraction module constructed with multi-kernel convolutional blocks and SA and CA attention mechanisms for skip connections. Then, a channel attention mechanism is used to fuse the information of the encoder and decoder. S3. The processed image is fed into the prompt word model and encoder network. Except for the last level encoder, each level encoder extracts features from the preprocessed medical image and sends the extracted feature maps to the next level encoder and the skip connection processing module respectively. The last level encoder inputs the extracted feature maps to the first level decoder. During the encoder processing, the prompt word model is connected to the encoder network. S4. Except for the first-level decoder, the other levels of decoders receive the input from the skip connection processing module and the input from the previous level decoder and perform reconstruction processing to obtain a feature map of the same size as the previous level decoder. Then, deconvolution is used to expand the size of the feature map. Finally, after processing by multiple levels of decoders, the feature map is restored to the size of the original image. S5. Use a combination of BCEDiceLOSS and MCEloss loss functions to train the overall network; S6. After training through steps S1-S5, the real-time image can be input into the model to obtain the final medical segmentation image.
2. The medical image segmentation method as described in claim 1, which integrates the golden ratio CNN and subtractive visual Mamba hybrid neural network, is characterized in that... In the ESVM-Net neural network, the prompt word model uses a pre-trained DenseNet, with all its parameters frozen during training. The output of the prompt word model is simultaneously connected to a five-level encoder. For each encoder level, the output of the prompt word model is divided into two paths. One path's output is fed into the corresponding encoder after being element-wise added to the other path's output through a bottleneck layer. The five-level encoder is sequentially connected via dual paths. The first-level encoder is used to input the pre-processed medical image. The first four encoder levels are connected to the decoder via a skip connection processing module. The fifth-level encoder is connected to the first-level decoder. The five decoders are connected sequentially. The previous level decoder is also connected to the next level decoder via a skip connection processing module. Each level decoder upsamples the received image through bilinear interpolation and then processes it through a local feature extraction module, SA, and CA to obtain a feature map.
3. The medical image segmentation method as described in claim 2, which integrates the golden ratio CNN and subtractive visual Mamba hybrid neural network, is characterized in that... In steps S3-S4, feature map extraction specifically includes: S31. The preprocessed medical image is processed through the prompt word model and the first-level encoder. The medical image generates feature image data C1_target and C2_background after passing through the first-level encoder. The target feature is input into the skip connection processing module. The target and background are input into the next level encoder and the output of the prompt word model is added. The final result is D1_target and D1_background. S32. In the second encoder, the feature image data D1_target and D1_background are processed again by step S31 to obtain feature image data D2_target and D2_background; S33. In the third encoder, the feature image data D2_target and D2_background are processed again by step S31 to obtain feature image data D3_target and D3_background; S34. In the fourth encoder, the feature image data D3_target and D3_background are processed again by step S31 to obtain feature image data D4_target and D4_background; S34. In the fifth encoder, the feature image data D4_target and D4_background are processed again by step S31 to obtain feature image data D5_target and D5_background; S36. Input the feature image data D1_target, D2_target, D3_target, D4_target into the skip connection processing module for processing to obtain feature maps A1, A2, A3, A4 respectively; S41. Input the feature image data D5_target into the first-level decoder for processing to obtain feature map B1; S42. Input feature map B1 into the second-level decoder for processing, and then add the feature map extracted by the skip connection processing module to obtain feature map C1. The formula is C1=Decoder(B1)+A4. S43. Repeat the above S42 operation in the remaining decoder stages until the image is restored to the resolution of the original image. This feature map is denoted as C4. S44. Process feature map C4 using the Sigmoid function to obtain the final feature map.
4. A medical image segmentation method that integrates the golden ratio CNN and subtractive visual Mamba hybrid neural network as described in any one of claims 1-3, characterized in that, The segmentation network in S5 is trained using the following loss function: ； in, To divide the network loss, , To set coefficients; ； Where: n is the total number of samples, y represents the true label, and represents the output of the segmentation network after processing the labeled image.
5. A medical image segmentation method that integrates the golden ratio CNN and subtractive visual Mamba hybrid neural network as described in any one of claims 1-3, characterized in that, The encoder uses a subtractive Visual Mamba model (SVMB), which includes a Visual State Space Block (VSSB), Point Convolutional Module (PWC), and a Local Feature Extraction Module (LFEM). VSSB and PWC are two parallel inputs that accept image feature data from the previous level. The output of VSSB is divided into two paths: one path is subtracted element-wise from the output of PWC and then input to LFEM; the other path is added element-wise to one output of LFEM and then output to the next level of SVMB. At the same time, the other output of LFEM is added element-wise to the output of the prompt word model and then output to the next level of SVMB. The VSSB includes LN and Linear, which receive the output of the previous stage and are connected sequentially. The Linear output is divided into two paths. One path has SiLu, SS2D and LN sequentially according to the data flow, and the other path is output through Scale. The two outputs are multiplied by Hadamard and then input into another Linear. At the same time, the output of the previous stage is passed through a multi-kernel convolution block and then added element-wise with the output of another Linear before being input into an LFEM and output by the LFEM. The LFEM sequentially includes PWC, BN / ReLU, multi-kernel residual convolution block MRCB, and BN and PWC. The first PWC accepts external input. The MRCB includes three sets of parallel depthwise separable convolutions (DWC) and BN / ReLU connected to each depthwise separable convolution. The three sets of parallel structures perform element-wise addition of their own output results with the output results of BN / ReLU. The results of the above element-wise addition are then added element-wise again and input to BN and the second PWC. Finally, the output result of the second PWC is added element-wise with the output result of the previous stage accepted by the LFEM to obtain the final output of the LFEM. The skip connection processing module adopts the efficient feature fusion attention block EFFA, which includes two parallel depthwise separable convolutions DWC and BN / ReLU. The output results of the two BN / ReLU are added element by element and the result is input to the efficient channel attention ECA. At the same time, the external input result is also output after performing a Hadamard product with the output result of ECA. The efficient channel attention (ECA) includes a max pooling layer, an average pooling layer, PWC / BN, and a sigmoid function. The external input results are correlated after passing through the max pooling layer and the average pooling layer, and then sequentially passed through the PWC / BN and sigmoid functions for output.