A multi-modal medical image segmentation method based on double-branch independent coding and reliability-aware fusion

Abstract

This invention discloses a multimodal medical image segmentation method based on bi-branch independent coding and reliability-aware fusion, belonging to the field of medical image processing. The invention employs a bi-branch independent coding architecture, utilizing two structurally consistent and parameter-independent VMamba encoders to extract features specific to different modalities. Simultaneously, a reliability-aware fusion mechanism is designed, consisting of symbolic channel reliability gating, a dynamic cross-modal interaction module, and an adaptive fusion module, to evaluate feature reliability from both the channel dimension and the modal level. In the encoding stage, a selective state-space model is used to achieve long-distance context modeling with low computational overhead. In the decoding stage, multi-scale skip connections are combined to fuse deep and shallow features, fully preserving details. This invention improves lesion segmentation accuracy, boundary delineation effect, and model robustness, and can be widely applied to automatic segmentation of lesions such as tumors, possessing good prospects for clinical and industrial applications.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and in particular to a multimodal medical image segmentation method based on dual-branch independent coding and reliability-aware fusion. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, neural network methods have become an important research direction in the field of image segmentation and have been widely applied in medical image segmentation tasks. Neural network-based segmentation methods can automatically extract lesion regions in an end-to-end, data-driven manner, effectively eliminating the reliance on manual intervention in traditional methods. Convolutional Neural Networks (CNNs), Fully Convolutional Neural Networks (FCNs), UNet, and their various variants are widely used in medical image segmentation, with UNet variants showing outstanding advantages in medical image segmentation tasks due to their superior feature extraction capabilities. In recent years, thanks to its self-attention mechanism and ability to effectively capture contextual information, Transformer models have gradually been applied to the field of medical image segmentation and have demonstrated performance superior to traditional FCN-based models.

[0003] However, despite the significant progress made by the aforementioned models in medical image segmentation, certain limitations remain. On one hand, traditional deep learning models based on CNNs and FCNs primarily rely on local convolutional operations for feature extraction, lacking explicit global attention modeling mechanisms. Therefore, they inherently fall short in capturing long-range dependencies and global contextual information. On the other hand, while Transformers can effectively model global information through self-attention mechanisms, their computational complexity typically increases quadratically with the length of the input sequence, leading to high computational and memory overhead in high-resolution medical image segmentation tasks. To alleviate this problem, Mamba introduces a selective state-space model, reducing computational complexity to a linear scale while maintaining long-range contextual modeling capabilities, thus demonstrating strong performance potential in visual modeling and medical image segmentation tasks. Nevertheless, in multimodal medical image segmentation tasks, existing methods still face the challenge of insufficient intermodal feature fusion. Many methods typically employ a shared encoder to extract features uniformly across different modalities. While this can reduce model complexity and promote cross-modal information interaction, it can also easily overlook the differences between different modalities in imaging mechanisms, structural information, and semantic representation. This weakens modality-specific features and may even cause mutual interference between features of different modalities, ultimately affecting segmentation accuracy and model robustness.

[0004] In view of the above background, this invention proposes a segmentation method for multimodal medical images such as PET / CT, which is particularly suitable for automatic segmentation of tumor regions. Summary of the Invention

[0005] In view of the above-mentioned defects in the existing technology, the technical problem to be solved by the present invention is to improve the performance of automatic segmentation of tumor regions.

[0006] To achieve the above objectives, this invention provides a multimodal medical image segmentation method based on dual-branch independent coding and reliability-aware fusion, belonging to the field of medical image processing. This invention employs a dual-branch independent coding architecture, utilizing two structurally consistent and parameter-independent VMamba encoders to extract features specific to different modalities. Simultaneously, a reliability-aware fusion mechanism is designed, consisting of symbol channel reliability gating, a dynamic cross-modal interaction module, and an adaptive fusion module, to evaluate feature reliability from both the channel dimension and the modal level. In the coding stage, a selective state-space model is used to achieve long-distance context modeling with low computational overhead. In the decoding stage, multi-scale skip connections are combined to fuse deep and shallow features, fully preserving details.

[0007] Furthermore, the method includes the following steps: Step S1: Obtain the first modality medical image and the second modality medical image corresponding to the spatial location of the same case; Step S2: Using a dual-branch independent coding structure, feature extraction is performed on the first modality medical image and the second modality medical image respectively to obtain the first modality features and the second modality features; Step S3: Perform reliability-aware cross-modal fusion processing on the first modal feature and the second modal feature to obtain multimodal fusion features; Step S4: Input the multimodal fusion features into the decoding module, combine them with multi-scale skip connections to complete feature recovery, and output the final medical image segmentation result.

[0008] Furthermore, the dual-branch independent coding structure in step S2 includes two encoders with identical structures and independent parameters; the two encoders are initialized with the same pre-trained weights, and their parameters are independently updated based on the corresponding modal data during model training.

[0009] Furthermore, the encoder uses the visual state space model VMamba as the basic backbone network. The encoder is divided into four feature extraction stages, from Stage 1 to Stage 4. Each stage outputs hierarchical features with different resolutions and channel dimensions. Each hierarchical feature is transmitted to the skip connection branches of the subsequent fusion module and decoding module.

[0010] Furthermore, the reliability-aware cross-modal fusion processing in step S3 includes three sub-steps: symbol channel reliability gating processing, dynamic cross-modal interaction processing, and adaptive fusion processing.

[0011] Furthermore, the symbolic channel reliability gating process specifically involves: concatenating the first modal feature and the second modal feature along the channel dimension to obtain a hybrid feature; performing layer normalization, transpose, and flattening operations on the hybrid feature; modeling cross-modal channel dependencies using a one-dimensional selective state space model; generating signed channel reliability weights using the Tanh activation function; and multiplying the channel reliability weights by the two modal features channel by channel to complete feature correction.

[0012] Furthermore, the dynamic cross-modal interaction processing specifically involves: inputting the two modal features, after symbol channel reliability gating correction, into convolutional blocks for dimensionality reduction; extracting the region tokens of the first modal feature; dividing the second modal feature into several region blocks and further splitting them into local tokens; modeling the location information of the first modal region through region Mamba blocks; modeling the local feature dependencies of the second modality through local Mamba blocks; and fusing the region tokens and local tokens within the same region to obtain the cross-modal interaction feature.

[0013] Furthermore, the adaptive fusion process specifically involves: concatenating the corrected first modal features and the second modal features along the channel dimension, generating modal-level adaptive weights after global average pooling, convolutional dimensionality reduction, and ReLU activation; weighting the two modal features using the adaptive weights, and combining them with dynamic cross-modal interaction features to obtain the final multimodal fusion features.

[0014] Furthermore, in step S4, the decoding module adopts a multi-level upsampling structure. Each decoding layer performs element-wise addition and fusion of the current decoded features with the features of the corresponding level of the encoder. The fused features are processed by the Mamba upsampling layer containing the visual state space model block to model long-distance contextual dependencies.

[0015] Furthermore, the first modal medical image is a PET image, and the second modal medical image is a CT image; the PET image is used to provide metabolic information, and the CT image is used to provide human anatomical structure information.

[0016] This invention fully utilizes the metabolic information provided by PET images and the anatomical structure information provided by CT images to achieve reliable-guided adaptive fusion while maintaining the specific expression of each modality. Compared with simple stitching, direct addition, or fixed-weight fusion methods, this invention can reduce the interference of low-quality modal features, noise features, and invalid information on the segmentation results, and improve the accuracy of lesion region segmentation, boundary delineation ability, and model robustness.

[0017] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0018] Figure 1 This is a flowchart of a preferred embodiment of the present invention; Figure 2 This is a system architecture diagram of a preferred embodiment of the present invention. Detailed Implementation

[0019] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0020] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0021] like Figure 1-2 As shown, this embodiment uses a PET / CT medical image segmentation task as an example to illustrate the specific implementation of the present invention. The first modality is a PET image, and the second modality is a CT image. PET images mainly provide high metabolic information, while CT images mainly provide anatomical structural information.

[0022] S1: Acquire first modality medical images and second modality medical images

[0023] A first modal medical image and a second modal medical image are acquired at corresponding locations within the same disease space. In this embodiment, the first modal medical image is a PET image, and the second modal medical image is a CT image. Both are used as model inputs for subsequent multimodal medical image segmentation.

[0024] S2: Dual-branch independent encoding extracts features of each modality.

[0025] The first and second modal medical images are converted into tensors of dimensions [B, 128, 128, 96] after convolution, transpose, and layer normalization operations, where B represents the batch size.

[0026] Subsequently, the first modality medical image is input into the first modality encoder, and the second modality medical image is input into the second modality encoder. The first and second modality encoders have the same structure but independent parameters, and are used to extract feature representations of different modalities respectively.

[0027] In this embodiment, the basic backbone network of the dual-branch independent feature encoder adopts the visual state space model VMamba. The parameters of the first encoder and the second encoder are loaded with pre-trained weights trained on ImageNet during the initialization phase, and are updated independently during the training phase.

[0028] The feature extraction processes of both the first and second encoders are divided into four stages, namely Stage 1 to Stage 4. Different levels are used to extract semantic features at different resolutions and transmit them to the next level and the cross-modal interaction module. The output dimensions of the four stages are as follows: Stage 1: [B, 96, 128, 128] Stage 2: [B,192,64,64] Stage 3: [B,384,32,32] Stage 4: [B,768,16,16] The independent modal features extracted at each stage are processed by the cross-modal feature interaction module to obtain multimodal fusion features. These fusion features are then connected to the corresponding upsampling stages in the decoding module to integrate the spatial information retained in the encoding stage and the semantic information recovered in the decoding stage.

[0029] S3: Reliability-Aware Fusion

[0030] The features at each level output from the first and second modal encoders are input into the cross-modal feature interaction module. This module includes the symbol channel reliability gated (SCRG), the dynamic cross-modal interaction module (DCIM), and the adaptive fusion module (ASF).

[0031] B1: Symbol Channel Reliability Gated SCRG

[0032] Given two different modalities of input features, they are first concatenated along the channel dimension and layer normalization is applied to obtain a mixed feature. Then, the mixed feature is transposed and flattened, and the channel dependency is modeled using a one-dimensional selective state-space model to obtain the weight coefficients corresponding to the two modalities.

[0033] Next, the Tanh activation function is used to generate reliability weight coefficients for the two modalities. These reliability weights are used to enhance correct feature information and suppress interfering information. Finally, the reliability weights are multiplied channel-by-channel by the features of the two modalities to obtain the corrected modal features.

[0034] The purpose of this step is to evaluate the reliability of different modal features in the channel dimension and dynamically adjust the modal feature response based on the reliability, so as to avoid the negative impact of noise or interference modes on the final segmentation result.

[0035] B2: Dynamic Cross-Modal Interaction Module DCIM

[0036] Given two modal features after SCRG correction, the two features are first fed into a convolutional stem composed of convolutional blocks for dimensionality reduction to obtain the corresponding modal features.

[0037] For the first modality feature, region tokens are extracted, with each region token corresponding to a first modality feature within a region. For the second modality feature, due to its clear anatomical structure and ability to provide fine segmentation information, the second modality feature is first segmented into several region blocks, and then each region block is further subdivided into several local blocks, so that a region block consists of a series of local tokens.

[0038] Subsequently, the location information in the region token of the first modality feature and the structural information in the local token of the second modality feature are dynamically interacted. Specifically, the inherent region location information in the first modality feature is processed using region Mamba blocks, and the complex dependencies of the local second modality features within each region block are modeled using local Mamba blocks. Finally, the region token of the first modality feature is integrated with the local token of the second modality feature within the same region to obtain the cross-modal interaction feature.

[0039] The purpose of this step is to fully integrate the metabolic location information in PET with the local anatomical structure information in CT through regional and local information interaction, thereby improving the ability to identify lesion areas.

[0040] B3: Adaptive Fusion Module (ASF)

[0041] The first and second modal features are concatenated along the channel dimension, and global average pooling is applied to the fused features. Then, convolutional blocks are used for dimensionality reduction to decrease computation. Finally, the ReLU activation function is applied to activate the features, yielding the corresponding weights for the two modal features.

[0042] Finally, the weights are multiplied by the corresponding features and fed into the decoder along with the fused features obtained from DCIM to obtain the multimodal fused features.

[0043] The purpose of this step is to automatically adjust the modality fusion weights based on the contribution of different modal features to the current segmentation task, thereby improving the effectiveness of multimodal feature fusion.

[0044] S4: The decoding module generates the target segmentation result.

[0045] The multimodal fusion features are input into the decoding module for layer-by-layer upsampling and feature recovery, and the target segmentation result is output.

[0046] In each layer of decoding, the output features of the previous layer are first fused with the output features of the encoder at the corresponding stage through element-wise addition to introduce high-frequency spatial information retained in the encoding stage. Subsequently, the fused features are processed using a Mamba upsampling layer, which contains multiple visual state space model blocks to model long-distance dependencies in the decoding process and enhance the semantic expressive power of the features.

[0047] After multiple levels of cyclic upsampling, the decoding module finally uses a final upsampling layer to restore the feature map to the original image resolution, and then uses a 1×1 convolutional layer to map the channels to the number of target categories, thereby generating the final medical image segmentation result.

[0048] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multimodal medical image segmentation method based on dual-branch independent coding and reliability-aware fusion, belonging to the field of medical image processing. This invention employs a dual-branch independent coding architecture, utilizing two structurally consistent and parameter-independent VMamba encoders to extract features specific to different modalities. Simultaneously, a reliability-aware fusion mechanism is designed, consisting of symbol channel reliability gating, a dynamic cross-modal interaction module, and an adaptive fusion module, to evaluate feature reliability from both the channel dimension and the modal level. In the coding stage, a selective state-space model is used to achieve long-distance context modeling with low computational overhead. In the decoding stage, multi-scale skip connections are combined to fuse deep and shallow features, fully preserving details.

2. The multimodal medical image segmentation method based on dual-branch independent coding and reliability-aware fusion as described in claim 1, the method comprising the following steps: Step S1: Obtain the first modality medical image and the second modality medical image corresponding to the spatial location of the same case; Step S2: Using a dual-branch independent coding structure, feature extraction is performed on the first modality medical image and the second modality medical image respectively to obtain the first modality features and the second modality features; Step S3: Perform reliability-aware cross-modal fusion processing on the first modal feature and the second modal feature to obtain multimodal fusion features; Step S4: Input the multimodal fusion features into the decoding module, combine them with multi-scale skip connections to complete feature recovery, and output the final medical image segmentation result.

3. The multimodal medical image segmentation method based on bi-branch independent coding and reliability-aware fusion as described in claim 2, wherein the bi-branch independent coding structure in step S2 includes two encoders with identical structures and independent parameters; the two encoders are initialized with the same pre-trained weights, and their parameters are independently updated according to the corresponding modal data during model training.

4. The multimodal medical image segmentation method based on dual-branch independent coding and reliability-aware fusion as described in claim 3, wherein the encoder uses the visual state space model VMamba as the basic backbone network, the encoder is divided into four feature extraction stages from Stage 1 to Stage 4, each stage outputs hierarchical features with different resolutions and different channel dimensions, and each hierarchical feature is transmitted to the skip connection branches of the subsequent fusion module and decoding module respectively.

5. The multimodal medical image segmentation method based on dual-branch independent coding and reliability-aware fusion as described in claim 4, wherein the reliability-aware cross-modal fusion processing in step S3 sequentially includes three sub-steps: symbol channel reliability gating processing, dynamic cross-modal interaction processing, and adaptive fusion processing.

6. The multimodal medical image segmentation method based on bi-branch independent coding and reliability-aware fusion as described in claim 5, wherein the symbolic channel reliability gating process specifically comprises: concatenating the first modality feature and the second modality feature along the channel dimension to obtain a hybrid feature; performing layer normalization, transpose, and flattening operations on the hybrid feature; modeling cross-modal channel dependencies using a one-dimensional selective state space model; generating signed channel reliability weights by combining the Tanh activation function; and multiplying the channel reliability weights by the two modality features channel by channel to complete feature correction.

7. The multimodal medical image segmentation method based on dual-branch independent coding and reliability-aware fusion as described in claim 6, wherein the dynamic cross-modal interaction processing specifically comprises: inputting the two modal features after symbolic channel reliability gating correction into convolutional blocks for dimensionality reduction; extracting the region tokens of the first modality feature; dividing the second modality feature into several region blocks and further splitting them into local tokens; modeling the location information of the first modality region through region Mamba blocks; modeling the local feature dependency relationship of the second modality through local Mamba blocks; and fusing the region tokens and local tokens within the same region to obtain the cross-modal interaction feature.

8. The multimodal medical image segmentation method based on dual-branch independent coding and reliability-aware fusion as described in claim 7, wherein the adaptive fusion processing specifically comprises: concatenating the corrected first modality feature and the second modality feature along the channel dimension, generating modality-level adaptive weights after global average pooling, convolutional dimensionality reduction, and ReLU activation; weighting the two modality features using the adaptive weights, and combining them with dynamic cross-modal interaction features to obtain the final multimodal fusion feature.

9. The multimodal medical image segmentation method based on dual-branch independent coding and reliability-aware fusion as described in claim 8, wherein the decoding module in step S4 adopts a multi-level upsampling structure, and each decoding layer performs element-level addition fusion of the current decoded features with the features of the corresponding level of the encoder; the fused features are processed by the Mamba upsampling layer containing the visual state space model block to model long-distance contextual dependencies.

10. The multimodal medical image segmentation method based on dual-branch independent coding and reliability-aware fusion as described in claim 9, wherein the first modal medical image is a PET image and the second modal medical image is a CT image; the PET image is used to provide metabolic information and the CT image is used to provide human anatomical structure information.

Images