Incomplete modal brain tumor MR image segmentation method, system and device based on local-global modeling structure and medium

Abstract

The invention discloses an incomplete modal brain tumor MR image segmentation method, system and device based on a local-global modeling structure and a medium, and the method comprises the steps: preprocessing a brain tumor MR image, including registration, normalization, data amplification and simulation modal deletion; an SD4M-Net network is constructed and trained, the SD4M-Net network comprises a segmentation coding module, a four-cascade Mama module (M4M), a space deformable convolution module (SDM) and a segmentation decoding module, the M4M captures three-dimensional long-range dependence, and the SDM enhances local features; evaluating the model by using a Dice coefficient; outputting a segmentation result; the system comprises a data preprocessing module, a feature extraction and fusion module and a segmentation output module. The equipment comprises an image collector, a program processor and a display, and incomplete modal brain tumor MR image segmentation is realized based on the method. The method can cope with modal deficiency, is high in segmentation precision, is superior to a mainstream method in the BraTS 2020 data set, and provides reliable support for clinic.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer-aided diagnosis technology, specifically to a method, system, device, and medium for segmenting incomplete modal brain tumor MR images based on a local-global modeling structure; in particular, this invention proposes a brain tumor segmentation solution that integrates four-cascaded Mamba modules (M4M) and spatially deformable convolution to address the common modality loss problem in clinical practice. Background Technology

[0003] Early and accurate diagnosis of brain tumors is crucial for improving patient survival rates. The current clinical diagnostic process comprises three stages: imaging examination, neurological examination, and biopsy. Among these, magnetic resonance imaging (MRI) technology, with its superior soft tissue resolution, has become the most commonly used imaging method for brain tumor diagnosis. Multimodal MRI (including T1, T1ce, T2, and FLAIR sequences) can provide complementary pathological information, offering important evidence for physicians' diagnoses. However, traditional diagnostic processes rely on physicians manually analyzing multimodal images, which is not only time-consuming and labor-intensive but also prone to misinterpretation due to physician subjectivity and fatigue. Given the uneven distribution of medical resources, the development of reliable auxiliary diagnostic tools is particularly urgent.

[0004] Early computer-aided diagnostic systems were mostly based on traditional machine learning algorithms, which had limited accuracy when processing complex and varied brain tumor images. Breakthroughs in deep learning technology, particularly the successful application of convolutional neural networks in image segmentation, have provided new solutions for automatic brain tumor segmentation. However, existing deep learning methods still have significant limitations when dealing with incomplete modalities commonly found in clinical practice: model performance often drops significantly when some modal data is missing. This is mainly because existing models largely rely on complementary information between complete modalities and lack robust design for incomplete data.

[0005] Currently, methods for handling modal missingness mainly fall into three categories: feature completion methods (such as the U-HVED model proposed by Dorrent et al., corresponding to the reference Dorrent R, Joutard S, Modat M, et al. Hetero-ModalVariational Encoder-decoder for Joint Modality Completion and Segmentation[C]. Medical Image Computing and Computer Assisted Intervention. Shenzhen, China: Springer, 2019: 74-82.) compensate by generating features of the missing modalities, but the generation quality is difficult to guarantee in cases of severe missingness; feature alignment methods (such as the self-supervised adversarial training strategy proposed by Shen et al., corresponding to the reference Shen Y, Gao M. Brain Tumor Segmentation on MRI with Missing Modalities[C]. Information Processing in Medical Imaging. Hong Kong, China: Springer, (2019:417-428.) Aligning feature distributions through adversarial training and other methods requires training multiple models for different missing combinations, resulting in high computational costs. Feature fusion methods (such as the HeMIS model proposed by Havaei et al., corresponding to the reference Havaei M, Guizard N, Chapados N, et al. Hemis: Hetero-modal Image Segmentation[C]. MedicalImage Computing and Computer-Assisted Intervention. Athens, Greece: Springer,2016: 469-477.) map existing modalities to a shared space for fusion, but often fails to fully exploit long-range dependencies in three-dimensional space and is not adaptable enough to the irregular morphology of brain tumors.

[0006] Therefore, there is an urgent need to develop a brain tumor image segmentation method that can effectively address modality loss and combines high accuracy with high efficiency to meet the needs of practical clinical applications. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention aims to provide a method, system, device, and medium for segmenting incomplete modal brain tumor MR images based on a local-global modeling structure. By introducing a local-global modeling structure that integrates a four-cascaded Mamba module (M4M) with a spatially deformable convolutional module, an innovative segmentation network, SD4M-Net, is constructed. This not only effectively solves the feature representation problem under incomplete modalities and achieves efficient capture of three-dimensional long-range semantic dependencies through the M4M module, but also enhances the adaptability to irregular tumor morphology with the help of the SDM module.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A method for segmenting incomplete modal brain tumor MR images based on local-global modeling structure includes the following steps: Step 1: Preprocessing brain tumor MR images: including image registration and intensity normalization, dataset image augmentation, and dataset segmentation and simulation of modality loss. Step 2: Construct and train the SD4M-Net segmentation network model. The network model adopts an encoder-decoder architecture, including a segmentation encoding module, a four-tiered Mamba module (M4M), a spatially deformable convolutional module (SDM), and a segmentation decoding module. The segmentation encoding module is responsible for image data feature extraction and downsampling. The four-tiered Mamba module (M4M) performs global long-range dependency modeling on the deep features of the segmentation encoding module. The spatially deformable convolutional module performs local adaptive enhancement on the features of each layer of the segmentation encoding module. The segmentation decoding module outputs the image segmentation result through upsampling and feature fusion. The preprocessed image is used as input to train the network model. Step 3: Evaluate the performance of the trained SD4M-Net network model using evaluation metrics; Step 4: Use the trained SD4M-Net network model to segment the input brain tumor MR image and output the segmentation results.

[0009] The simulation of modality missing situations in step one specifically involves systematically masking the input data of one or more modalities during model training and testing using a preset binary mask matrix, in order to simulate various incomplete modal scenarios that may occur in clinical practice.

[0010] In step two, the four-cascaded Mamba module (M4M) processes the feature map using a three-dimensional sequence modeling method, as follows: The four-level Mamba module (M4M) strengthens long-range dependencies in three-dimensional space through sequence modeling operations; First, multi-plane decomposition is performed to reorganize the 3D feature map into input tensors of three orthogonal planes: HW, HD, and WD. Subsequently, the input sequences of each plane are input into independent four-level Mamba modules (M4M) for long-range dependency modeling, capturing the correlation information between distant pixels / voxels in the image, thereby understanding the overall structure, semantic connections and contextual relationships of brain tumors in three-dimensional space; Then, the outputs of each four-level Mamba module (M4M) are fused across the plane, including extracting local context using 3×3×1 convolution, cross-plane addition, and 1×3×3 convolution; Finally, the fused features are processed again through a four-level Mamba module (M4M) to extract global information, and then output after adjusting the number of channels by a 1×1×1 convolution.

[0011] In step two, the spatially deformable convolutional module (SDM) employs a dual-path convolutional structure: One path is a spatially adaptive convolution path, which generates spatial context guidance information through a channel attention mechanism and dynamically adjusts the convolution parameters to achieve spatially adaptive convolution operation on the input feature map; The other path is a 3D deformable convolution path, which learns the offset of sampling points through additional convolutional layers, enabling the standard convolutional kernel to adapt to the irregular shape of the tumor. Finally, the output features of the two paths are added together and fused.

[0012] In step two, the network model uses a combination of cross-entropy loss and Dice loss as the loss function type and applies it to each decoder layer. The total loss function of the SD4M-Net network is composed of a combination of the deep supervision loss function for the local modeling structure and the loss function for the global modeling structure, proportionally based on the type of combined loss function. The overall loss function of the SD4M-Net network is calculated using the following formula: In the formula: This represents the total loss function; The depth-supervised loss function represents the local modeling structure; The loss function represents the global modeling structure; This indicates the weights assigned to the deep supervised loss function; The combined loss function of the cross-entropy loss function and the Dice loss function is calculated as follows: In the formula: Represents the combined loss function; Denotes the cross-entropy loss function of the i-th layer decoder; Denotes the Dice loss function of the i-th layer decoder; This indicates a five-layer decoder; The cross-entropy loss is used to measure the difference between the predicted result and the true label in pixel classification, and its calculation formula is as follows: In the formula: Number of categories; For the first Class weight; For the first The predicted probability value of the model; For the first The binary representation of the class based on the global real label; The Dice loss is used to measure the degree of overlap between the predicted segmented region and the actual segmented region, and its calculation formula is as follows: In the formula: This represents the number of pixels in the predicted segmentation mask; This represents the number of pixels in the global true segmentation mask; This represents the intersection of the predicted segmentation mask and the global ground truth segmentation mask. Used to avoid zero-split situations; Each decoder layer includes four upsampling levels in the segmentation decoding module and a final output layer. Each level outputs the corresponding segmentation prediction result and participates in the loss calculation.

[0013] The evaluation index used in step three is the Dice similarity coefficient, which is calculated using the following formula: In the formula: This represents the number of pixels in the predicted segmentation mask; This represents the number of pixels in the globally true segmentation mask labeled by the physician. This represents the intersection of the predicted segmentation mask and the global ground truth segmentation mask.

[0014] A segmentation system for incomplete modal brain tumor MR images based on a local-global modeling structure, comprising: The data preprocessing module is used for registration, normalization, data amplification, and modality loss simulation of brain tumor MR images in step one. The feature extraction and fusion module includes the segmentation coding module, the four-cascaded Mamba module (M4M), and the spatially deformable convolution module (SDM), which are used to perform feature extraction, long-range dependency modeling, and local feature enhancement on the input image in step two. The segmentation output module, including the segmentation decoding module, is used to upsample and decode the fused features in steps two and four to restore the segmentation result image of the original image size.

[0015] A segmentation device for incomplete modal brain tumor MR images based on a local-global modeling structure, comprising: Image acquisition device used to acquire multimodal MR images of brain tumors; A program processor is used to store a computer program and execute the computer program to implement the incomplete modality brain tumor MR image segmentation method based on local-global modeling structure as described in any one of steps one to four. A monitor used to display the segmentation results of MR images of brain tumors.

[0016] A computer-readable storage medium storing a computer program that, when executed by a processor, enables the segmentation of incomplete modal brain tumor MR images based on a local-global modeling structure, according to the method described in any one of steps one to four.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Excellent robustness to modal missing features: This invention systematically simulates multiple modal missing combinations during network training and efficiently extracts complementary information from available modalities using the M4M module, enabling the SD4M-Net model to easily cope with common clinical modal incompleteness situations without relying on cumbersome feature completion or training multiple models for different missing features.

[0018] 2. Powerful 3D long-range dependency modeling capability: By introducing the M4M module based on the advanced selective state space model (SSM), this invention can efficiently and effectively capture the global contextual information and long-range semantic dependencies of brain tumors in 3D space, overcoming the shortcomings of limited receptive field of traditional convolutional neural networks and high computational complexity of Transformer-type models.

[0019] 3. Precise ability to segment irregular shapes: The SDM module designed in this invention combines spatial adaptive convolution and 3D deformable convolution, enabling the network to adaptively adjust the receptive field and convolution kernel shape, more accurately fitting the irregular boundaries of brain tumors, especially when segmenting tumor cores and enhancing structurally complex regions such as tumors.

[0020] 4. Structural Design for Local and Global Co-optimization: By organically combining the M4M module responsible for global modeling and the SDM module responsible for local feature enhancement, and supplemented by a deep supervision mechanism, this invention achieves effective fusion and co-optimization of multi-scale and multi-level features from local details to global semantics, significantly improving overall segmentation performance. Experiments demonstrate that on the BraTS 2020 dataset, the proposed SD4M-Net outperforms mainstream methods on multiple evaluation metrics, confirming its effectiveness and advancement.

[0021] In summary, this invention successfully constructed the innovative segmentation network SD4M-Net by introducing a local-global modeling structure that integrates a four-level concatenated Mamba module (M4M) with a spatially deformable convolutional module. This approach not only effectively solves the feature representation problem under incomplete modalities and achieves efficient capture of 3D long-range semantic dependencies through the M4M module, but also enhances adaptability to irregular tumor morphologies with the help of the SDM module. Experimental results show that this invention exhibits superior robustness to 15 modality loss cases on the BraTS 2020 dataset, and its overall performance significantly outperforms existing mainstream methods (such as SRMNet and mmFormer), with particularly outstanding improvements in segmentation accuracy for key regions such as tumors and tumor cores. Furthermore, the network structure maintains high efficiency while possessing good engineering deployment potential, providing reliable technical support for clinical applications and playing an important role in promoting the development of precision medicine. Attached Figure Description

[0022] Figure 1 This is a network structure diagram of SD4M-Net of the present invention.

[0023] Figure 2 This is a structural diagram of the M4M module of the present invention.

[0024] Figure 3 This is a structural diagram of the SDM module of the present invention.

[0025] Figure 4 This is a diagram of the SC convolution structure of the present invention.

[0026] Figure 5 This is a diagram of the 3D deformable convolution structure of the present invention.

[0027] Figure 6 This is a visualization of the segmentation results of different models in this invention. Detailed Implementation

[0028] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. These embodiments are implemented based on the technical solution of the present invention, providing detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0029] Example 1: A method for segmenting incomplete modal brain tumor MR images based on local-global modeling structure See Figure 1 The SD4M-Net network provided in this embodiment is a deep convolutional neural network based on an encoder-decoder architecture. Its core innovation lies in the introduction of a local-global modeling structure, which consists of four cascaded Mamba modules (M4M) and spatially deformable convolutional modules (SDM), specifically designed for processing brain tumor MR image segmentation tasks with incomplete modalities. The specific steps are as follows: Step 1: Preprocessing brain tumor MR images This step aims to convert the original DICOM format MR images into a uniform format suitable for network training and to enhance data diversity.

[0030] 1) Image registration and intensity normalization: For the same patient... Images of the four modalities were registered to the same standard space. Subsequently, each image was normalized by subtracting the mean from the voxel intensity value of each modality and dividing by the standard deviation to eliminate the influence of scanner differences.

[0031] 2) Data Augmentation: To improve the model's generalization ability and prevent overfitting, online augmentation of 3D image patches is performed during the training phase. Specific operations include: 2.1) Random cropping: Randomly crop an image block of size 128×128×128 voxels from the registered image; 2.2) Random flipping: Random flipping is performed along the axial, coronal, or sagittal plane with a probability of 0.5; 2.3) Random rotation: Random rotation is performed within an angle range of ±10°; 3) Dataset partitioning and modality missing simulation: The BraTS 2020 public dataset, containing training data from 369 patients, was used. This dataset was divided into training, validation, and test sets in a ratio of 219:50:100.

[0032] To simulate modality loss, a common clinical scenario, a four-dimensional mask matrix was defined at the network input layer. , respectively corresponding Four modes. If a mode is missing, the input data of its corresponding channel is set to zero, and the mask bit is set to 0. This embodiment simulates a total of 15 different mode combinations.

[0033] Step 2: Construct the SD4M-Net network model: The SD4M-Net model structure is as follows: Figure 1 As shown, it mainly includes the following modules: 1) Segmentation and encoding module: ... Four modalities of brain tumor images are input into the corresponding segmentation and coding modules, where downsampling is performed through various segmentation and coding layers. Simultaneously, a uniform mask matrix is ​​set in each segmentation and coding layer to simulate different modalities of the images and generate fused feature maps. and ( In this context, the subscript numbers represent the levels of the segmentation coding layer, and m represents different modal combinations generated by the mask matrix. This represents the set of all modes. Then, Input into the M4M module in step 2) to extract features.

[0034] 2) Four-level Mamba module (M4M): The M4M module enhances the feature representation capability of the modal encoder by mining and utilizing the correlations between modes. Its structure is as follows: Figure 2 As shown: 2.1) Three-dimensional feature serialization: First, perform multi-plane decomposition to convert the input features... Reshape it using the dimension transformation operation into the following three orthogonal plane input tensors: input to the HW plane. Input of HD plane and WD plane input Then, by using the corresponding Mamba-based modules: M_HW, M_HD, and M_WD, long semantic information in three different planes is captured. The process of the M_HW module capturing long text information is as follows: Figure 2 As shown, in the M_HW module, the input is first processed through the Flatten operation. Flatten it into a sequence tensor of size B×L×C (L is of size H×W). After the above serialization operation, it can be provided as input to the Mamba structure; 2.2) Selective State-Space Model Processing: In the Mamba architecture, the input tensor is first normalized, then linearly projected through a linear layer and passed through a one-dimensional convolutional layer. After the SiLU activation function layer, the tensor is processed through a selective SSM layer. Finally, the output tensor is obtained through a residual connection, then passed through a linear layer again, and the input tensor is restored through a dimension transformation operation. The selective SSM layer is the core component of the Mamba architecture, enhancing its ability to handle long text messages. This selective SSM layer is designed based on the SSM layer, whose structure is as follows: In the formula: This is a continuously hidden state over time. and For the input and output at time; The state matrix; and It belongs to the projection matrix. To adapt to the discretized data in computers, a zero-order preservation method is used with a step size of... The structure of the discretized SSM layer is as follows: In the formula: represent Discretized state transition matrix; It is the identity matrix; represent The discretized projection matrix. The final discretized recursive formula is: In the formula: Representing the The hidden state at each moment; Representing the The hidden state at each moment; Representing the Input at any given moment; Representing the The output at each time step. The SSM layer uses a dynamic matrix... and And a parallel scanning algorithm to obtain a selective SSM layer; 2.3) Multi-plane feature fusion: First, let the output tensors of the M_HW, M_HD, and M_WD modules be... , and Local contextual information is extracted using a 3×3×1 convolutional block. Then, the feature maps from these three different planes are added across the plane and fused using a 1×3×3 convolutional block to enhance local convolutional features. The fused features are then input again into the Mamba module M_HWD to enhance global consistency and further extract global information. Finally, the resulting feature tensor is passed through a 1×1×1 convolutional block to obtain the output tensor of the M4M module. The M4M module proposed in this invention applies the Mamba module to 3D image segmentation. In order to fully utilize the advantages of the Mamba module, the 3D image is decomposed into multiple planes in the network of this invention. It is hoped that the Mamba module can capture rich 3D information of brain tumors, and interact with the information of each plane through separable convolution. Finally, the global information is extracted again through the Mamba module and 1×1×1 convolution. 3) Spatial Deformable Convolutional Module (SDM): This module is crucial for local modeling and employs dual-path convolution: a Spatially Adjusted (SC) convolutional module and a 3D deformable convolutional module to process the input image. The specific structure of the SDM module is as follows: Figure 3 As shown; 3.1) Spatial Adaptive Convolution Path: This is a dual-path input structure, where one path obtains the SC module input through ordinary 3D convolution. Another path requires input. Tensor. For The input image is transposed, convolved, and average pooled to obtain channel-level statistics. Then, a two-layer fully connected network is used to learn the channel importance weights to obtain the channel weights. Then, channel-level calibration of the input features is performed using weights, and the data is generated using the Softmax function and adaptive average pooling (HW direction). Tensors. Among them, the SC convolution module uses... Tensor completes the input to the SC module Spatial information adjustment. The structure of the SC convolutional module is as follows: Figure 4 As shown. In Figure 4 middle, Represents the input tensor. Represents the output tensor. A-weights represents adjusting the weights in the 2D convolution. A-bias represents adjusting the bias in the 2D convolution. The SC convolution module uses... Tensors provide spatial context information to regulate the input tensor. The convolution operation allows for the acquisition of more effective spatial information about brain tumors from the input image. The input image is then processed through the SC convolution module. The depth and channel information are compressed into the same dimension to obtain a deformable tensor of size 1×C*D×H×W. Then, a two-dimensional convolution is performed on the deformable tensor. Therefore, it is necessary to... Tensors provide the weights and biases that guide the two-dimensional convolution operation. To obtain these parameters, first... The dimensionality is increased to obtain a tensor with dimensions B×D×1×H(1)×W(1), and the weights A-weights are adjusted by setting them. (in Represents the number of output channels, Represents the number of input channels, and The size of the convolution kernel is assigned to the tensor after dimensionality increase, resulting in new spatial information weights. To accommodate two-dimensional convolution, the dimensions of the weights are adjusted while retaining the weights required for convolution. The bias adjustment in convolution parameters is similar to the weight adjustment; both involve assigning and adjusting dimensions to ultimately obtain the bias. The obtained weights and biases are then used to perform a 2D convolution operation on the deformed tensor of dimension 1×C*D×H×W. Finally, the tensor is... Dimensionality restoration yields the output tensor. This output tensor is then added to the output of the 3D deformable convolution module to obtain the output of the SC module. The structure of the 3D deformable convolution module is as follows: Figure 5 As shown; 3.2) 3D deformable convolution path: such as Figure 5 As shown, 3D deformable convolution increases its spatial receptive field through learnable biases, thereby enhancing its spatial modeling capabilities, making it particularly suitable for irregularly shaped objects such as brain tumors. Input The bias is generated through 3D convolution, and the bias receptive field has a total of 3×N values ​​along the output channel dimension (3N represents the deformation value of the sampling grid corresponding to depth, height, and width; in a 3×3×3 convolution, N=27 is the size of the sampling grid). The learned lateral bias... and longitudinal offset The deformation of the sampling mesh is used to guide the generation of an adaptive mesh. Finally, the adaptive mesh is used to produce the output features. 3D deformable convolution can be represented as: In the formula: This represents the weights at the corresponding positions of the convolution kernel; This represents a 3×3×3 convolution sampling grid; This represents the element value at the corresponding position on the input feature map; This represents any point in the input feature map; Indicates the output feature map The element value at that position; yes The first in One value; express The Middle The offset corresponding to each value.

[0035] 4) Segmentation and Decoding Module: The segmentation and decoding module includes convolution operations, feature concatenation, and upsampling operations. The spatial resolution of the feature map is gradually restored through the segmentation and decoding module, and multi-stage feature information is fused to finally output the brain tumor region segmentation result.

[0036] Step 3: Train and evaluate the SD4M-Net network model; 1) Loss Function: In this embodiment, a total loss function is used to optimize the model, which is a proportional combination of the deep supervision loss function of the local modeling structure and the loss function of the global modeling structure. 2) Training settings: 2.1) Experimental platform: The operating system is Ubuntu 22.04, the PyTorch 2.0.0 deep learning framework is used, the GPU is NVIDIA GeForce RTX 4090D, and the CPU is AMD R9 9950X 16-Core Processor; 2.2) Optimizer: The Adam optimizer is used, with parameters set to β1=0.9, β2=0.999, and weight decay is... ; 2.3) Learning Rate Strategy: An exponential learning rate strategy was adopted, which is expressed as: In the formula: This represents the current learning rate; The representative size is The initial learning rate; The variable is set to 800 rounds; To adjust the exponent and set its value to 0.9; This represents the current training epoch. This strategy scientifically adjusts the learning rate, reducing the risk of overfitting and accelerating the convergence of the model's loss function curve. 3) Evaluation Metrics: The Dice similarity coefficient was used to evaluate the segmentation effectiveness of brain tumor images. The Dice similarity coefficient is used to calculate the similarity between the predicted tumor region and the actual tumor region, and is a major determinant of segmentation accuracy. The expression is: In the formula: This represents the number of pixels in the predicted segmentation mask; This represents the number of pixels in the globally true segmentation mask labeled by the physician. (represents the intersection of the predicted segmentation mask and the global ground truth segmentation mask). When the model achieves a segmentation Dice coefficient of over 89.56% in the entire tumor region, over 81.58% in the tumor core region, and over 77.23% in the enhanced tumor region, the model is considered to meet the standards for clinical application.

[0037] Step 4: Output brain tumor region segmentation results: To more intuitively compare the segmentation performance of different models on brain tumor images, segmentation labels were generated by inferring from the test set of images using different models. Spatial slicing was then performed on the segmentation labels to obtain segmented labeled images of brain tumors at different spatial locations. The segmentation results of SRMNet, mmFormer, MFTrans, and the proposed SD4M-Net model were visualized, as shown in the figure below. Figure 6 As shown. Because the test set experiment generated segmentation labels for 15 different modalities, the visualization results of the segmentation results of different models include 15 modal slice images at different spatial locations. These modal images are: a) T2, b) T1ce, c) T1, d) Flair, e) T1ce T2, f) T1ce T1, g) Flair T1, h) T1 T2, i) Flair T2, j) Flair T1ce, k) Flair T1ce T1, l) Flair T1 T2, m) Flair T1ce T2, n) T1ce T1 T2, o) Flair T1ce T1 T2.

[0038] exist Figure 6 In the diagram, the red area represents the necrotic and non-enhancing area, i.e., the tumor core (TC); the green area represents the edematous area, i.e., the whole tumor area (WT); and the blue area corresponds to the enhancing tumor area (ET). According to... Figure 6 It can be seen that the segmentation results of models such as SRMNet, mmFormer, MFTrans, and SD4M-Net are closer to the true segmentation labels as the number of modality combinations increases. In terms of specific segmentation details, the mmFormer and SD4M-Net models are closer to the true segmentation labels for tumors, while SRMNet and MFTrans models are inferior to the first two models in terms of segmentation details for brain tumors, specifically failing to accurately segment the contours of the ET region of brain tumors. However, when the number of modalities is small, especially in the case of a single modality, the mmFormer model exhibits missegmentation problems in the TC and ET regions of brain tumors, while the SD4M-Net model proposed in this invention shows fewer missegmentation issues. In summary, the visualization results of the SD4M-Net model are superior to the first three models, proving the effectiveness of the model proposed in this invention.

[0039] Experimental Results and Analysis To verify the effectiveness of this invention, ablation experiments and comparative experiments were conducted on the BraTS 2020 test set.

[0040] Ablation experiments (Table 1): Ablation experiments were conducted based on the BraTS 2020 test set to verify the performance of the designed algorithm. In the ablation experiments, the local-global model was used as the baseline model, and the following ablation experiments were designed: 1) Baseline (baseline model), 2) Baseline combined with the M4M module (Baseline+M4M), 3) Baseline combined with the SDM module (Baseline+SDM), and 4) Baseline combined with both the M4M and SDM modules (Baseline+M4M+SDM), which served as the SD4M-Net model proposed in this invention. The detailed experimental results are listed in Table 1. In the table, black circles ● represent existing modal images, and white circles ○ represent modal images that are masked by the mask matrix.

[0041] Table 1 As shown in Table 1, on the BraTS 2020 dataset, the baseline model achieved multimodal average Dice similarity coefficients of 60.2%, 87.1%, and 80.1% in the enhanced tumor (ET), whole tumor (WT), and tumor core (TC) regions, respectively. These good multimodal average Dice similarity coefficients demonstrate the feasibility of improving the baseline model. After introducing the M4M module into the baseline (Baseline+M4M), the model achieved multimodal average Dice similarity coefficients of 60.1%, 87.9%, and 80.6% in the ET, WT, and TC regions, respectively. Specifically, the multimodal average Dice similarity coefficients in the WT and TC regions improved by 0.8% and 0.5% respectively compared to the baseline model. Introducing the M4M module into the baseline model improved the overall segmentation performance. The M4M module operates on the global structure; by using orthogonal planes and combining them with the Mamba structure, it possesses powerful analytical capabilities for long text information, enhancing the baseline model's ability to extract global information features of brain tumors during the encoding stage.

[0042] After introducing the SDM module into the baseline (Baseline+SDM), the model achieved multimodal average Dice similarity coefficients of 60.5%, 87.8%, and 80.5% in the ET, WT, and TC regions, respectively. The multimodal average Dice similarity coefficients in the ET, WT, and TC regions were improved by 0.3%, 0.6%, and 0.4% compared to the baseline model, respectively. Introducing the SDM module into the baseline further enhanced the overall performance of the model's segmentation metrics. The SDM module primarily operates on the local structure of the network, employing a dual-path convolutional structure of spatially modulated convolution and 3D deformable convolution. Spatially modulated convolution improves the model's ability to extract local spatial information, while 3D deformable convolution enhances its ability to recognize brain tumor regions with irregular shapes. It is precisely this dual-convolutional structure of SDM that improves the overall performance of the baseline model in brain tumor image segmentation.

[0043] Integrating all the above-mentioned improved modules into the baseline model (Baseline+M4M+SDM), we obtain the SD4M-Net model designed in this invention. The SD4M-Net model achieves multimodal average Dice similarity coefficients of 61.4%, 87.7%, and 80.5% in the ET, WT, and TC regions, respectively. The multimodal average Dice similarity coefficients in the ET, WT, and TC regions are improved by 1.2%, 0.6%, and 0.4% respectively compared to the baseline model. After adding all the improved modules, the baseline model achieves the best segmentation performance. The results of the ablation experiments verify the effectiveness of the proposed SD4M-Net model in brain tumor image segmentation under incomplete modality image conditions, and also demonstrate the effectiveness of the improved modules SDM and M4M in the experiments.

[0044] Comparative Experiments (Table 2): To further verify the effectiveness of the SD4M-Net model proposed in this invention, comparative experiments were conducted between the SD4M-Net model and four other models designed based on architectural methods: SRMNet, mmFormer, and MFTrans. Table 2 shows the results of the comparative experiments on the BraTS 2020 test set.

[0045] Table 2 Table 2 shows that the SD4M-Net model achieved multimodal average Dice similarity coefficients of 61.5%, 87.7%, and 80.4% in the ET, WT, and TC regions, respectively. In the comparative experiments, the SRMNet model achieved multimodal average Dice similarity coefficients of 60.2%, 87.0%, and 79.9% in the ET, WT, and TC regions, respectively. However, the SD4M-Net model improved the multimodal average Dice similarity coefficients in the ET, WT, and TC regions by 1.3%, 0.7%, and 0.5% compared to SRMNet, respectively. In the comparative experiments, mmFormer achieved multimodal average Dice similarity coefficients of 60.2%, 87.8%, and 80.1% in the ET, WT, and TC regions, respectively. Compared to mmFormer, while the SD4M-Net model has a slightly lower multimodal average Dice similarity coefficient (0.1%) in the WT region, it improves by 1.3% and 0.3% in the ET and TC regions, respectively, indicating that the overall segmentation performance of the SD4M-Net model is superior to mmFormer. In comparative experiments, the MFTrans model achieved multimodal average Dice similarity coefficients of 58.5%, 85.7%, and 77.3% in the ET, WT, and TC regions, respectively. The SD4M-Net model significantly improves the average Dice similarity coefficient compared to MTCNet, with improvements of 3.0%, 2.0%, and 3.1% in the ET, WT, and TC regions, respectively.

[0046] In the segmentation of 15 sub-modal images, the SD4M-Net model achieved the best segmentation results on 13 modalities of the ET region, 7 modalities of the WT region, and 12 modalities of the TC region. Whether considering the average segmentation value or the single-modal segmentation performance, the SD4M-Net model proposed in this invention demonstrates significant advantages in the segmentation task of incomplete modal brain tumor images.

[0047] In summary, this invention proposes a brain tumor segmentation network, SD4M-Net, based on a local-global modeling structure by introducing M4M and SDM modules. This network effectively improves the segmentation performance of incomplete modal brain tumor images and provides a more effective solution for brain tumor image segmentation in the case of incomplete modal images.

Claims

1. A method for segmenting incomplete modal brain tumor MR images based on local-global modeling structure, characterized in that, Includes the following steps: Step 1: Preprocessing brain tumor MR images: including image registration and intensity normalization, dataset image augmentation, and dataset segmentation and simulation of modality loss. Step 2: Construct and train the SD4M-Net segmentation network model. The network model adopts an encoder-decoder architecture, including a segmentation encoding module, a four-tiered Mamba module (M4M), a spatially deformable convolutional module (SDM), and a segmentation decoding module. The segmentation encoding module is responsible for image data feature extraction and downsampling. The four-tiered Mamba module (M4M) performs global long-range dependency modeling on the deep features of the segmentation encoding module. The spatially deformable convolutional module performs local adaptive enhancement on the features of each layer of the segmentation encoding module. The segmentation decoding module outputs the image segmentation result through upsampling and feature fusion. The preprocessed image is used as input to train the network model. Step 3: Evaluate the performance of the trained SD4M-Net network model using evaluation metrics; Step 4: Use the trained SD4M-Net network model to segment the input brain tumor MR image and output the segmentation results.

2. The method for segmenting incomplete modal brain tumor MR images based on local-global modeling structure according to claim 1, characterized in that: The simulation of modality missing situations in step one specifically involves systematically masking the input data of one or more modalities during model training and testing using a preset binary mask matrix, in order to simulate various incomplete modal scenarios that may occur in clinical practice.

3. The method for segmenting incomplete modal brain tumor MR images based on local-global modeling structure according to claim 1, characterized in that: In step two, the four-cascaded Mamba module (M4M) processes the feature map using a three-dimensional sequence modeling method, as follows: The four-level Mamba module (M4M) strengthens long-range dependencies in three-dimensional space through sequence modeling operations; First, multi-plane decomposition is performed to reorganize the 3D feature map into input tensors of three orthogonal planes: HW, HD, and WD. Subsequently, the input sequences of each plane are input into independent four-level Mamba modules (M4M) for dependency modeling, capturing the correlation information between distant pixels / voxels in the image, thereby understanding the overall structure, semantic connections and contextual relationships of brain tumors in three-dimensional space; Then, the outputs of each four-level Mamba module (M4M) are fused across the plane, including extracting local context using 3×3×1 convolution, cross-plane addition, and 1×3×3 convolution; Finally, the fused features are processed again through a four-level Mamba module (M4M) to extract global information, and then output after adjusting the number of channels by a 1×1×1 convolution.

4. The method for segmenting incomplete modal brain tumor MR images based on local-global modeling structure according to claim 1, characterized in that: In step two, the spatially deformable convolutional module (SDM) employs a dual-path convolutional structure: One path is a spatially adaptive convolution path, which generates spatial context guidance information through a channel attention mechanism and dynamically adjusts the convolution parameters to achieve spatially adaptive convolution operation on the input feature map; The other path is a 3D deformable convolution path, which learns the offset of sampling points through additional convolutional layers, enabling the standard convolutional kernel to adapt to the irregular shape of the tumor. Finally, the output features of the two paths are added together and fused.

5. The method for segmenting incomplete modal brain tumor MR images based on local-global modeling structure according to claim 1, characterized in that: In step two, the network model uses a combination of cross-entropy loss function and Dice loss function as the loss function type used by the network and applies it to each decoder layer; while the total loss function of the SD4M-Net network is composed of the deep supervision loss function of the local modeling structure and the loss function of the global modeling structure combined in proportion to the type of combined loss function. The overall loss function of the SD4M-Net network is calculated using the following formula: In the formula: This represents the total loss function; The depth-supervised loss function represents the local modeling structure; The loss function represents the global modeling structure; This indicates the weights assigned to the deep supervised loss function; The combined loss function of the cross-entropy loss function and the Dice loss function is calculated as follows: In the formula: Represents the combined loss function; Indicates the first The cross-entropy loss function of the layer decoder; Indicates the first The Dice loss function of the layer decoder; This indicates a five-layer decoder; The cross-entropy loss is used to measure the difference between the predicted result and the true label in pixel classification, and its calculation formula is as follows: In the formula: Number of categories; For the first Class weight; For the first The predicted probability value of the model; For the first The binary representation of the class based on the global real label; The Dice loss is used to measure the degree of overlap between the predicted segmented region and the actual segmented region, and its calculation formula is as follows: In the formula: This represents the number of pixels in the predicted segmentation mask; This represents the number of pixels in the global true segmentation mask; This represents the intersection of the predicted segmentation mask and the global ground truth segmentation mask. Used to avoid zero-split situations; Each decoder layer includes four upsampling levels in the segmentation decoding module and a final output layer. Each level outputs the corresponding segmentation prediction result and participates in the loss calculation.

6. The method for segmenting incomplete modal brain tumor MR images based on local-global modeling structure according to claim 1, characterized in that: The evaluation index used in step three is the Dice similarity coefficient, which is calculated using the following formula: In the formula: This represents the number of pixels in the predicted segmentation mask; This represents the number of pixels in the globally true segmentation mask labeled by the physician. This represents the intersection of the predicted segmentation mask and the global ground truth segmentation mask.

7. A brain tumor MR image segmentation system based on the segmentation method according to any one of claims 1 to 6, characterized in that, include: The data preprocessing module is used for registration, normalization, data amplification, and modality loss simulation of brain tumor MR images in step one. The feature extraction and fusion module includes the segmentation coding module, the four-cascaded Mamba module (M4M), and the spatially deformable convolution module (SDM), which are used to perform feature extraction, long-range dependency modeling, and local feature enhancement on the input image in step two. The segmentation output module, including the segmentation decoding module, is used to upsample and decode the fused features in steps two and four to restore the segmentation result image of the original image size.

8. A brain tumor MR image segmentation device based on the segmentation method according to any one of claims 1 to 6, characterized in that, include: Image acquisition device used to acquire multimodal MR images of brain tumors; A program processor is configured to store a computer program and, when executing the computer program, implement the incomplete modality brain tumor MR image segmentation method based on local-global modeling structure as described in any one of claims 1 to 6. A monitor used to display the segmentation results of MR images of brain tumors.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it can perform incomplete modal brain tumor MR image segmentation based on the method of any one of claims 1 to 6.

Images