Tumor medical image segmentation method based on multi-modal information interaction

Abstract

The invention relates to a tumor medical image segmentation method based on multi-modal information interaction, and the method comprises the steps: firstly carrying out the preprocessing of medical images of different modals, including spatial alignment and contrast normalization; then establishing a multi-modal segmentation network Mmi-Unet, using a ResNet encoder to independently encode the features of each modal, and realizing feature interaction through an enhancement-plain-scan cross-modal attention module and a plain-scan-enhancement cross-modal attention module; in the process, calculation of attention scores is optimized through information entropy, so that the distribution consistency of the feature region is enhanced; then, adaptive weighted integration of different modal features is realized through a dynamic modal fusion module, so that the expression ability of the multi-modal features is improved; and finally, in combination with the improved Mmi-Unet architecture, realizing high-precision segmentation through multi-scale feature fusion and joint optimization. The invention provides an efficient and accurate medical image segmentation solution, and can effectively deal with a complex multi-modal image collaborative optimization problem.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and more specifically to a tumor medical image segmentation method based on multimodal information interaction. Background Technology

[0002] Medical image segmentation is a crucial task in medical image analysis, widely used in the diagnosis and treatment of various diseases. With the continuous advancement of medical imaging technology, multimodal imaging such as CT and MRI has achieved significant applications in medical image segmentation. Nevertheless, medical image segmentation still faces a series of challenges.

[0003] First, the shapes of targets such as tumors and organs in images are diverse, the boundaries are blurred, and the tissue density is similar, which makes accurate segmentation more complicated. Traditional medical image segmentation methods mainly rely on manual annotation or automatic segmentation techniques based on single-modality images. However, manual annotation is not only time-consuming, but also easily affected by operator bias. With the application of deep learning technology, automated segmentation methods have gradually become mainstream. Among them, U-Net, as a classic segmentation network, effectively extracts features through encoder-decoder structure and skip connections, and has shown good results in medical image segmentation. However, it can be seen from the scheme in reference [1] that U-Net has limited ability to capture details of complex lesions and performs poorly when dealing with blurred boundaries and complex shapes. At the same time, reference [2] points out that single-modality images often cannot fully present the different features of the target and it is difficult to make full use of the complementary information of each modality, thus limiting the performance of the segmentation model.

[0004] To address the challenges in medical image segmentation, multimodal image fusion methods have gradually become the key to improving segmentation accuracy. Common fusion methods include input-level fusion and intermediate-level fusion, as seen in reference [3]. Input-level fusion in reference [4] stacks image data from different modalities as network input, which is simple and efficient, but it fails to fully explore the complex relationships between modalities and is prone to redundant information and interference. Intermediate-level fusion, on the other hand, performs dynamic interaction of modal features within the network, which can better capture complementary information between modalities and improve segmentation performance. However, despite some progress, there are still problems in practical applications: input-level fusion fails to capture deep relationships between modalities and is prone to introducing redundant information; although intermediate-level fusion effectively improves synergy, its performance is limited by the fusion strategy and encoder design, as seen in reference [5].

[0005] Therefore, this invention designs a multimodal medical image segmentation and modeling method to solve the above problems.

[0006] References [1] Mao Mengting. Research on multimodal brain tumor segmentation technology based on deep learning [D]. Central China Normal University, 2024. DOI:10.27159 / d.cnki.ghzsu.2024.000594. [2] Ding Hao. Research on multimodal MRI brain tumor segmentation method based on deep learning [D]. Shanghai Normal University, 2025. [3] Zhang Dingke, Yang Wenxia, ​​Zhang Yuanzhou. Multimodal brain tumor segmentation algorithm based on improved 3D U-Net [J]. Modern Information Technology, 2023, 7(13): 80-83+87. DOI:10.19850 / j.cnki.2096-4706.2023.13.016. [4] Huang Jingteng, Li Qiang, Guan Xin. An improved U-shaped network for brain tumor segmentation [J]. Journal of Engineering Science, 2023, 45(06):1003-1012. [5] Ma Qipeng. Research on MRI brain tumor segmentation algorithm based on deep learning [D]. Jiangnan University, 2021. DOI:10.27169 / d.cnki.gwqgu.2021.001183. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a tumor medical image segmentation method based on multimodal information interaction. Using the segmentation network model of this invention for medical image segmentation can effectively handle complex application environments involving multimodal image collaborative optimization.

[0008] The tumor medical image segmentation method based on multimodal information interaction includes the following steps: Step S1: Preprocess the medical images of different modalities, including using interpolation methods to adjust the resolution to unify the size of the images of different modalities and normalizing the interpolated images; Step S2: Based on the Unet model, a multimodal segmentation network called Mmi-Unet network is established. Features of each modality are independently encoded, and feature interaction is achieved through the enhancement-flat scan cross-modal attention module and the flat scan-enhancement cross-modal attention module. The Mmi-Unet network uses dual-branch residual blocks as the encoder of the U-net model, providing an independent encoding path for each modality, so that the features of each modality are learned in an independent branch, avoiding information mixing. Step S3: Weighted fusion of multimodal features is performed through the dynamic modal fusion module to achieve cross-modal information integration; Step S4: Combine the Mmi-Unet network from step S2 to decode the fused multimodal features and generate high-precision segmentation results; Step S5: Train and optimize the Mmi-Unet network using a combined loss function based on Dice coefficients and cross-entropy to obtain a segmentation network model for high-precision segmentation of target lesions in the medical images preprocessed in Step S1.

[0009] Specifically, step S2 includes the following sub-steps: Step S2.1: Use a dual-branch residual block as the Unet encoder to extract features of mode one and mode two respectively, while maintaining the independence of modal features; Step S2.2: Interaction between modal features is achieved through an enhanced-flat-scan cross-modal attention module and a flat-scan-enhanced cross-modal attention module, guiding the features of modality one and modality two to achieve cross-modal feature alignment; the method is as follows: Step 2.2.1: Let the input features of Mode 1 and Mode 2 be respectively... ,in Where C is the spatial dimension of the feature map, and C is the number of channels; Step 2.2.2: Divide the input features into blocks, assuming each block is P×P in size, forming a total of L patches. Let represent the feature vector corresponding to the i-th patch block, where ; All block features of each of the two modes Flattened into a one-dimensional vector, we obtain: and ; then linearly mapped to dimension D; Step 2.2.3: Generate mode 1 After linear mapping, it becomes the query vector. And the generation of mode two Mapped to key vector AND value vector And use a multi-head attention mechanism to calculate the attention score matrix. Let the number of attention heads be . Then the dimension of each head is ; Step 2.2.4: Generate mode 2 After linear mapping, it becomes the query vector. And the mode one generated Mapped to key vector AND value vector Similarly, the attention score matrix is ​​calculated using a multi-head attention mechanism. : ; Step S2.3: Introduce patch information entropy weights into the attention mechanism to enhance attention to high-information regions. The method is as follows: Step 2.3.1: First, process each patch block. The probability distribution is obtained using the Softmax function. in This represents the normalized weight of the j-th feature component in the i-th patch block; Step 2.3.2: Calculate information entropy Step 2.3.3: Then normalize all information entropy values ​​H to and Let these represent the maximum and minimum values ​​of the information entropy, respectively. The above operations are performed on the features of mode one and mode two respectively, yielding the normalized entropy vector for mode one. And the normalized entropy vector in mode 2 ; Step 2.3.4: The normalized entropy vector is used to weight the attention scores; Combined value vector Connecting to the residuals yields the final output features of the enhanced-flat-scan cross-modal attention module. ; Combined value vector Connecting the residuals yields the final output features of the flat-scan-enhanced cross-modal attention module. .

[0010] Specifically, step S3 includes the following sub-steps: Step S3.1: In the dynamic modality fusion module, dynamically weight the modal features and set two trainable model parameters. and Then, dynamic weights are generated using the Softmax function. and This ensures that the importance of multimodal features is adaptively adjusted. Step S3.2: Modal features and The final multimodal fusion feature vector is obtained by performing element-wise multiplication with the dynamically trained weights and then adding them together. .

[0011] Specifically, step S4 utilizes the skip connections of U-Net to pass the fused features back to the ResNet decoders of their respective modalities, including: Step S4.1: The decoder adopts a symmetric U-Net structure, combining multi-scale features and global contextual information; Step S4.2: Retain key modal features through residual connections to refine the boundaries of the segmentation results.

[0012] Specifically, the combined loss function described in step S5 is: It is the Dice coefficient. It is the cross-entropy loss function. and These are the weighting coefficients for different loss terms.

[0013] The advantages of this invention are as follows: The multimodal segmentation network Mmi-Unet combines the global interaction capabilities of the ResNet encoder and Transformer, enabling efficient extraction of local features and capture of global dependencies between modalities. First, a bidirectional cross-attention mechanism is employed, using features from different modalities as queries, keys, and values ​​to achieve information interaction and collaboration between modalities, thereby enhancing feature fusion performance. Furthermore, the model introduces the concept of information entropy, prioritizing regions with higher information content to reduce model bias towards a single modality. To further improve performance, the model employs an improved skip connection strategy and a dynamic coefficient fusion module to achieve multi-scale feature fusion and adaptively selectively enhance key modal features, avoiding over-reliance on a single modality. Through these innovative designs, this invention provides an efficient and accurate medical image segmentation solution that effectively addresses complex multimodal image collaborative optimization problems. Attached Figure Description

[0014] Figure 1 The overall structure of the multimodal segmentation network Mmi-Unet proposed in this invention is shown.

[0015] Figure 2 This is a schematic diagram of the various modal interaction modules proposed in this invention, including (A) the EPCA module, (B) the MF module and (C) the PECA module.

[0016] Figure 3 This is a schematic diagram of the segmentation result of the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0018] This invention proposes a multimodal medical image segmentation model based on modal collaboration, such as... Figure 1 As shown, the input is a set of CT images in different modalities, including modality one and modality two; the output is a schematic diagram of the binarized segmentation result of the corresponding lesion region. The specific steps of this invention are as follows: Step 1: Data preprocessing. This includes interpolating medical images of different modalities to adjust resolution and unify spatial dimensions; then normalizing the interpolated images to unify contrast and intensity range across different modalities.

[0019] Step 1.1: First, the medical images are loaded from their original file formats (such as DICOM or NIfTI) into the processing system using Python's pydicom library reading module. The image data is converted to a matrix format during the loading process for subsequent operations.

[0020] Step 1.2: After data loading, a preset threshold judgment algorithm is used to remove blank or invalid background areas from the image. Specifically, areas with a grayscale value of 0 or very small values ​​are set to NaN or ignored, thus retaining only the image portion containing valid lesions.

[0021] Step 1.3: To address grayscale differences between different scanning devices, an image normalization method is used to scale the pixel values ​​of the image to a uniform range. The specific implementation method is as follows: in, For the original image data, For standardized image data, and These are the minimum and maximum values ​​of the original image data.

[0022] Step 1.4: Generate diverse image data samples using data augmentation methods to improve the robustness of subsequent models. These data augmentation methods include operations such as rotation, flipping, cropping, and scaling.

[0023] Step 2: Establish a multimodal segmentation network based on the U-net model, which is called Mmi-Unet in this invention. Mmi-Unet encodes the features of each modality separately and realizes feature interaction through a cross-modal attention module.

[0024] Step 2.1: As Figure 1 As shown, dual-branch residual blocks are used as U-Net encoders (dual-branch ResNet encoders) to extract features of mode 1 and mode 2 respectively, providing an independent encoding path for each mode, so that the features of each mode can be fully learned in independent branches, avoiding information mixing.

[0025] Step 2.2: As Figure 2As shown in sections (A) and (C), an Enhanced-Plain Scan Cross-Attention Module (EPCA) and a Plain Scan-Enhanced Cross-Attention Module (PECA) are established. EPCA and PECA facilitate interaction between modal features, using features from modality one to guide features from modality two, and vice versa, thus achieving cross-modal feature alignment. Specific operations include: Step 2.2.1: Let the input features be respectively ,in The spatial dimensions of the feature map. Number of channels; Step 2.2.2: Divide the input features into blocks, assuming each block size is... , together formed There are 1 patch blocks, each denoted as _ . All block features of each of the two modalities Flattened into a one-dimensional vector, we obtain: and ; then it is linearly mapped to dimension D.

[0026] Step 2.2.3: Generate the patch block from mode one. After linear mapping, it becomes the query vector. And the generation of mode two Mapped to key vector AND value vector And utilize a multi-head attention mechanism (assuming the number of attention heads is ). Then the dimension of each head is Calculate the attention score matrix: .

[0027] Step 2.2.4: Apply the patch generated by mode 2 After linear mapping, it becomes the query vector. And the mode one generated Mapped to key vector AND value vector Similarly, the attention score matrix is ​​calculated using a multi-head attention mechanism: .

[0028] The core of the above steps lies in the calculation of attention scores. This is achieved by using the features of different modalities as query vectors (Q) in attention calculation, and then mapping the features to serve as key vectors (K) and value vectors (V) in attention calculation.

[0029] Step 2.3: To address the instability of local feature distribution in patch-based feature extraction, this invention employs an information entropy-based method to process the features of each patch; specific operations include: Step 2.3.1: First, process each patch The probability distribution is obtained using Softmax. Step 2.3.2: Calculate information entropy Step 2.3.3: Then normalize all entropy values ​​to The normalized entropy vector This is used to weight the attention scores. The input consists of two modalities, therefore, we obtain... and .

[0030] Step 2.3.4: Combine value vectors Connecting with the residual yields the final output features. Combined value vector Connecting with the residual yields the final output features. Step 3: Design the modal fusion (MF) module, such as Figure 2 As shown in section (B), this invention effectively integrates multi-source features from the EPCA and PECA modules. It achieves cross-modal information integration by weighted fusion of multi-modal features through a dynamic modal fusion module; specifically as follows: Step 3.1: Set trainable parameters and The weights are converted into dynamic weights by the Softmax function. and , Step 3.2: and After performing element-wise multiplication with the corresponding weights and then adding them together, the final fused feature vector is obtained. Step 4: After modal interaction at different encoder stages, the fused features are passed back to the ResNet decoders of their respective modalities using U-Net skip connections. This maintains the independence of modal information while fully utilizing complementary features to enhance feature representation. During decoding, each modality retains its key information and benefits from cross-modal enhancements, thereby improving segmentation accuracy and robustness, ultimately generating a more accurate segmentation prediction mask.

[0031] Step 5: Design a combined loss function based on the Dice coefficient and cross-entropy to train and optimize the model.

[0032] Step 5.1: The Dice coefficient is a metric for measuring the similarity between two sets, particularly useful in semantic segmentation tasks for evaluating the overlap between model predictions and ground truth labels. Its calculation formula is as follows: in It is the pixel-level probability of the model's predicted value. It is the real label after binarization.

[0033] Cross-entropy loss is used to measure the difference between the predicted distribution and the true distribution, and its formula is as follows: Step 5.2: Combine the two losses. In this example, the combination coefficient is... Set them all to 0.5. Finally, the overall loss function is used. Train the model.

[0034] The final trained segmentation network model is used to perform high-precision segmentation of target lesions in the medical images preprocessed in step S1. Figure 3 This is a schematic diagram of the segmentation result of the present invention, where A is the location of the lesion in the original CT, B is the lesion mask image of the original CT, and C is the segmented lesion image of the present invention.

[0035] The aforementioned medical image segmentation model is primarily used for high-precision segmentation of target lesions in medical images. By using this segmentation model, this invention can significantly improve segmentation accuracy, reduce cumbersome human-computer interaction operations, and possess strong versatility. This invention implements an end-to-end algorithm flow, effectively assisting doctors in improving diagnostic accuracy.

[0036] It should be noted that the medical images to be segmented can be acquired through scanning by various imaging systems or transmitted through internal or external storage systems such as Picture Archiving and Communication Systems (PACS). Common imaging modalities include, but are not limited to, single or combined modalities such as magnetic resonance imaging (MRI), magnetic resonance angiography (MRA), computed tomography (CT), and positron emission tomography (PET).

[0037] In the embodiments, the quantitative comparison of the segmentation results of the proposed method (referred to as Mmi-Unet in the table) and the other four different segmentation methods on the publicly available medical image dataset (BraTS) for the 2021 brain tumor segmentation task is shown in Table 1. The bolded values ​​indicate the highest values ​​obtained.

[0038] Table 1 lists four different models: SwinUnetr, a deep learning architecture based on the Swin Transformer (moving window transform); SegResNet, which combines a Residual Network (ResNet) and an encoder-decoder architecture; Enn-UNet, which extends the nn-UNet architecture and won first place in the BraTS2021 competition; and 3D Unet, which uses 3D convolutional layers. The main evaluation metrics include the Dice coefficient and Hausdorff distance 95% (HD95). The Dice coefficient measures the similarity between the model's prediction and the ground truth label; a higher value indicates better segmentation. HD95 reflects the error between the predicted boundary and the ground truth boundary; a lower value indicates higher accuracy. WT (Whole Tumor), TC (Tumor Core), and ET (Enhancing Tumor) represent different tumor regions in the same image.

[0039] Table 1 The data shows that the Mmi-Unet proposed in this invention performs best in Dice coefficient, and is superior to other models in WT, TC, and ET, indicating that it has the highest segmentation accuracy in different tumor regions. In terms of the HD95 metric, Mmi-Unet performs best in ET, while TC and WT also show good stability. Overall, Mmi-Unet possesses stronger medical image segmentation capabilities, providing more accurate tumor region identification results and offering reliable support for clinical diagnosis and treatment.

[0040] To verify the universality of the segmentation method proposed in this invention, we conducted experiments on another set of colorectal cancer CT datasets. The model input included two modalities of CT images: enhanced CT and plain CT. The main evaluation metrics were Dice coefficient, IoU coefficient, and recall. The IoU coefficient was used to evaluate the degree of overlap between the segmentation results and the ground truth annotations, while the recall rate reflected the model's ability to identify positive class samples. The results are shown in Table 2.

[0041] Table 2 As shown in Table 2, the Mmi-Unet proposed in this invention demonstrates significant advantages in key evaluation metrics such as Dice coefficient, IoU coefficient, and recall. Whether in the enhanced or normal modality, Mmi-Unet exhibits superior segmentation performance compared to other models, particularly in recall and IoU coefficient. This indicates that the Mmi-Unet proposed in this invention not only possesses high segmentation accuracy but also strong robustness and broad adaptability, providing consistent and excellent performance across various medical image datasets, and thus has broad application prospects.

[0042] In summary, this method first preprocesses medical images of different modalities, including spatial alignment and contrast normalization. Then, it uses a ResNet encoder to extract features for each modality and introduces a bidirectional cross-attention mechanism to achieve cross-modal feature interaction. During this process, the calculation of attention scores is optimized by information entropy to enhance the distribution consistency of feature regions. Next, a dynamic modality fusion module is used to achieve adaptive weighted integration of features from different modalities, thereby improving the expressive power of multimodal features. Finally, combined with an improved Mmi-Unet model, high-precision segmentation is achieved through multi-scale feature fusion and joint optimization.

[0043] Experimental results demonstrate that the Mmi-Unet model proposed in this invention exhibits significant advantages in integrating multimodal information. By fusing the local feature extraction capabilities of ResNet with the global modeling capabilities of Transformer, and combining information entropy guidance and dynamic fusion strategies, the model not only effectively improves the collaborative representation capability of multimodal images, but also achieves higher accuracy and robustness in tumor medical image segmentation tasks, validating its application potential in complex medical scenarios.

Claims

1. A tumor medical image segmentation method based on multimodal information interaction, characterized in that, Includes the following steps: Step S1: Preprocess the medical images of different modalities, including using interpolation methods to adjust the resolution to unify the size of the images of different modalities and normalizing the interpolated images; Step S2: Based on the Unet model, a multimodal segmentation network called Mmi-Unet network is established. Features of each modality are independently encoded, and feature interaction is achieved through the enhancement-flat scan cross-modal attention module and the flat scan-enhancement cross-modal attention module. The Mmi-Unet network uses dual-branch residual blocks as the encoder of the U-net model, providing an independent encoding path for each modality, so that the features of each modality are learned in an independent branch, avoiding information mixing. Step S3: Weighted fusion of multimodal features is performed through the dynamic modal fusion module to achieve cross-modal information integration; Step S4: Combine the Mmi-Unet network from step S2 to decode the fused multimodal features and generate high-precision segmentation results; Step S5: Train and optimize the Mmi-Unet network using a combined loss function based on Dice coefficients and cross-entropy to obtain a segmentation network model for high-precision segmentation of target lesions in the medical images preprocessed in Step S1.

2. The tumor medical image segmentation method based on multimodal information interaction according to claim 1, characterized in that, Step S2 includes the following sub-steps: Step S2.1: Use a dual-branch residual block as the Unet encoder to extract features of mode one and mode two respectively, while maintaining the independence of modal features; Step S2.2: Interaction between modal features is achieved through an enhanced-flat-scan cross-modal attention module and a flat-scan-enhanced cross-modal attention module, guiding the features of modality one and modality two to achieve cross-modal feature alignment; the method is as follows: Step 2.2.1: Let the input features of Mode 1 and Mode 2 be respectively... ,in Where C is the spatial dimension of the feature map, and C is the number of channels; Step 2.2.2: Divide the input features into blocks, assuming each block is P×P in size, forming a total of L patches. Let represent the feature vector corresponding to the i-th patch block, where ; All block features of each of the two modes Flattened into a one-dimensional vector, we obtain: and ; then linearly mapped to dimension D; Step 2.2.3: Generate mode 1 After linear mapping, it becomes the query vector. And the generation of mode two Mapped to key vector AND value vector And use a multi-head attention mechanism to calculate the attention score matrix. Let the number of attention heads be . Then the dimension of each head is ; Step 2.2.4: Generate mode 2 After linear mapping, it becomes the query vector. And the mode one generated Mapped to key vector AND value vector Similarly, the attention score matrix is ​​calculated using a multi-head attention mechanism. : ； Step S2.3: Introduce patch information entropy weights into the attention mechanism to enhance attention to high-information regions. The method is as follows: Step 2.3.1: First, process each patch block. The probability distribution is obtained using the Softmax function. in This represents the normalized weight of the j-th feature component in the i-th patch block; Step 2.3.2: Calculate information entropy Step 2.3.3: Then normalize all information entropy values ​​H to and Let these represent the maximum and minimum values ​​of the information entropy, respectively. The above operations are performed on the features of mode one and mode two respectively, yielding the normalized entropy vector for mode one. And the normalized entropy vector in mode 2 ; Step 2.3.4: The normalized entropy vector is used to weight the attention scores; Combined value vector Connecting to the residuals yields the final output features of the enhanced-flat-scan cross-modal attention module. ; Combined value vector Connecting the residuals yields the final output features of the flat-scan-enhanced cross-modal attention module. .

3. The tumor medical image segmentation method based on multimodal information interaction according to claim 2, characterized in that, Step S3 includes the following sub-steps: Step S3.1: In the dynamic modality fusion module, dynamically weight the modal features and set two trainable model parameters. and Then, dynamic weights are generated using the Softmax function. and This ensures that the importance of multimodal features is adaptively adjusted. Step S3.2: Modal features and The final multimodal fusion feature vector is obtained by performing element-wise multiplication with the dynamically trained weights and then adding them together. .

4. The tumor medical image segmentation method based on multimodal information interaction according to claim 1, characterized in that, Step S4 utilizes the skip connections of U-Net to pass the fused features back to the ResNet decoders of their respective modalities, specifically including: Step S4.1: The decoder adopts a symmetric U-Net structure, combining multi-scale features and global contextual information; Step S4.2: Retain key modal features through residual connections to refine the boundaries of the segmentation results.

5. The tumor medical image segmentation method based on multimodal information interaction according to claim 1, characterized in that, The combined loss function described in step S5 is: It is the Dice coefficient. It is the cross-entropy loss function. and These are the weighting coefficients for different loss terms.

Images