Multimodal magnetic resonance image brain glioma segmentation method and application thereof

Abstract

The application provides a multi-modal nuclear magnetic resonance image brain glioma segmentation method and application thereof, and belongs to the field of medical image processing.The M2ES-UNet network is provided in view of the problems of insufficient space feature utilization, single multi-modal fusion mechanism and cross-level semantic loss of existing segmentation methods.The method extracts multi-view anatomical information through a multi-plane feature coordination module;orthogonal dimension fusion convolution and mode introspection-coordination modules are used to realize differentiated fusion of shallow and deep features;and the encoding information smooth transmission module is used to realize progressive jump transmission of features.The application can effectively mine multi-plane and multi-modal complementary information, significantly improve the boundary precision and robustness of brain glioma segmentation, and assist clinical diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing and computer-aided diagnosis, in particular to a multi-modal magnetic resonance imaging (MRI) brain glioma automatic segmentation method based on deep learning and application thereof. BACKGROUND

[0002] Brain glioma is the most common primary malignant tumor in the central nervous system, with characteristics of high invasiveness, variable anatomical location and strong internal structural heterogeneity. Magnetic resonance imaging (MRI) has become a key basis for brain glioma diagnosis and treatment planning due to its high soft tissue contrast and multi-sequence imaging capabilities (such as T1, T2, T1ce, FLAIR).

[0003] At present, deep learning-based automatic segmentation methods (such as UNet, TransUNet, etc.) have made certain progress in the field of medical image segmentation. However, existing technologies still face the following severe challenges when dealing with brain glioma segmentation tasks:

[0004] Firstly, insufficient utilization of spatial features. Existing segmentation networks mostly rely on general 3D convolution operations, ignoring the habit of clinicians to comprehensively verify the information on coronal, axial and sagittal planes during reading, making it difficult to effectively capture the unique anatomical structure information of each plane and limiting the accurate understanding of tumor spatial morphology.

[0005] Secondly, single multi-modal fusion mechanism. Current fusion strategies (such as simple channel concatenation or pixel addition) are mostly linear fusion, which is difficult to capture the complex nonlinear complementary relationship between different modalities (such as FLAIR's edema high signal and T1ce's tumor enhancement area). Especially in the processing of shallow features, the asymmetric growth pattern of tumors in different spatial dimensions is ignored; in the processing of deep features, there is a lack of adaptive suppression mechanism for feature conflicts between modalities, resulting in insufficient discriminability of fused features.

[0006] Finally, there is a semantic gap in cross-level information transmission. Traditional skip connection directly concatenates deep features of the encoder with shallow features of the decoder, which often introduces noise due to the huge difference in semantic level and resolution between the two, resulting in the loss of deep key semantic information during transmission, ultimately causing the segmentation result boundary to be fuzzy, especially in the definition of tumor infiltration area. SUMMARY

[0007] The embodiment of the present application provides a multi-modal magnetic resonance image brain glioma segmentation method and application thereof, and aims at the problems in the prior art, such as incomplete spatial feature extraction caused by neglecting multi-plane anatomical perspective, poor feature complementarity caused by linear simple multi-modal fusion, and fuzzy boundary segmentation caused by cross-level semantic gap.

[0008] The core technology of the present application is mainly to propose a M2ES-UNet segmentation network, which extracts multi-view anatomical information through a multi-plane feature coordination module (MPS), combines a differentiated mid-fusion strategy of shallow orthogonal dimension fusion (ODFC) and deep modal introspection coordination (MICB), and uses a progressive encoding information smoothing transmission module (EIST) to replace the traditional jump connection, thereby realizing high-precision segmentation of multi-modal MRI brain glioma.

[0009] In a first aspect, the present application provides a multi-modal magnetic resonance image brain glioma segmentation method, which comprises the following steps:

[0010] Obtaining multi-modal brain MRI images of a patient to be segmented;

[0011] Constructing a segmentation network model, which comprises an encoder, a decoder and a feature transmission module connecting the two;

[0012] Using the encoder to independently encode each modality of the input MRI image, and in the encoding process, performing multi-plane feature coordination processing on the feature maps of each modality: splitting the three-dimensional feature map in the channel dimension, respectively mapping to the coronal plane, axial plane and sagittal plane for independent plane feature extraction, and then coupling and fusing the extracted plane features to obtain enhanced multi-plane features corresponding to each modality;

[0013] Performing mid-feature fusion on the enhanced multi-plane features of each modality output by the encoder, which comprises:

[0014] For the enhanced multi-plane features of each modality at the shallow layer of the encoder, an orthogonal dimension fusion strategy is adopted to capture and fuse the feature correlation in the height, width and depth three orthogonal dimensions respectively;

[0015] For the enhanced multi-plane features of each modality at the deep layer of the encoder, a modal introspection and coordination strategy is adopted, which first performs feature introspection enhancement within the single modality, and then calculates the importance weight of each modality based on the cross-modality attention mechanism to perform weighted coordination fusion of multi-modal features;

[0016] Using the feature transmission module to perform encoding information smoothing transmission, and gradually transmitting the fused deep layer encoding features to the decoder through step-by-step upsampling and fusion with the features of adjacent scales;

[0017] The received features are decoded and reconstructed by the decoder to output a segmentation result of the brain glioma.

[0018] Further, in the multi-plane feature collaborative processing, splitting the three-dimensional feature map in the channel dimension specifically includes:

[0019] Considering the anatomical symmetry of the axial plane in brain MRI diagnosis, half of the number of channels of the input feature map is allocated to the axial plane feature extraction branch, and the remaining number of channels is evenly allocated to the coronal plane feature extraction branch and the sagittal plane feature extraction branch.

[0020] Further, in the multi-plane feature collaborative processing, the independent feature extraction of each plane includes:

[0021] Through convolution operation and dimension reshaping, the input three-dimensional feature is converted into a query vector, a key vector and a value vector containing current plane information, wherein the key vector and the value vector integrate position encoding information;

[0022] The attention map of the current plane is calculated by using the query vector and the key vector, and the value vector is weighted by using the attention map to obtain the significant feature of the plane;

[0023] The significant feature is connected in residual with the input feature to obtain the output feature of the plane.

[0024] Further, the complementary coupling and fusion of the extracted plane features specifically include:

[0025] The independently extracted coronal plane feature, axial plane feature and sagittal plane feature are spliced in the channel dimension to obtain multi-plane spliced features;

[0026] The multi-plane spliced features are linearly transformed by using 1x1x1 convolution to promote information interaction between different planes;

[0027] The transformed features are processed in sequence by applying group normalization and LeakyRelu activation function to output enhanced multi-plane features.

[0028] Further, the orthogonal dimension fusion strategy for the encoder shallow layer specifically includes:

[0029] The enhanced multi-plane features of each modality are spliced in the channel dimension;

[0030] The spliced feature map is respectively reshaped in the height, width and depth directions to obtain three groups of feature tensors of different perspectives;

[0031] Convolution kernels are used to perform convolution operations on the three groups of feature tensors to extract context structure information specific to each dimension;

[0032] The three groups of feature tensors after convolution are restored to the original shape and spliced in the channel dimension, and the final shallow fusion features are generated through dimension reduction convolution and a feedforward neural network.

[0033] Further, the modal introspection and collaborative strategy for the deep layers of the encoder includes a single-modal introspection stage and a modal collaboration stage.

[0034] The single-modal introspection stage includes: dividing the enhanced multi-plane features of each modality into multiple windows, performing multi-head self-attention calculation within the windows, and then performing feature enhancement through window interaction and depth separable convolution.

[0035] The modal collaboration stage includes: taking the features after modal introspection as query vectors, and taking the spliced results of all modal features as shared key vectors and value vectors; calculating the correlation scores between the query vectors of each modality and the shared key vectors, and weighting the shared value vectors accordingly to obtain collaborative features that fuse global modal information.

[0036] Further, the smooth transmission of encoded information specifically includes:

[0037] A double-input structure is adopted, the first input comes from the encoder fusion features of the current level, and the second input comes from the output of the transmission module of a deeper level.

[0038] The first input is subjected to a transpose convolution operation to match the scale.

[0039] Spatial attention mechanisms and channel attention mechanisms are applied to the two input features respectively to perform feature screening.

[0040] The screened two features are fused, part of the fusion results are transmitted to the decoder of the same level, and the other part of the fusion results are taken as the input of the transmission module of the next level.

[0041] In a second aspect, the present application provides a multi-modal magnetic resonance image glioma segmentation device, comprising:

[0042] An image acquisition module is configured to acquire multi-modal brain MRI images of a patient to be segmented.

[0043] An encoding module is configured with multiple single-modal encoding paths and is configured to extract multi-scale features of each modality and includes a multi-plane feature collaboration unit configured to split the features into coronal, axial and sagittal planes in the channel dimension for independent extraction and complementary fusion, and output enhanced multi-plane features corresponding to each modality.

[0044] A feature fusion module is configured to fuse the enhanced multi-plane features of each modality output by the encoding module, including an orthogonal dimension fusion unit for shallow features and a modal introspection and collaboration unit for deep features.

[0045] The feature transmission module is configured to adopt a progressive upsampling strategy to smoothly transmit the fused deep semantic information to the decoder.

[0046] The decoding module is configured to receive the transmitted features and generate a brain glioma segmentation image.

[0047] In a third aspect, the present application provides an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the above-mentioned multi-modal magnetic resonance image brain glioma segmentation method.

[0048] In a fourth aspect, the present application provides a readable storage medium, the readable storage medium storing a computer program, the computer program comprising program codes for controlling a process to perform the process, and the process comprising the above-mentioned multi-modal magnetic resonance image brain glioma segmentation method.

[0049] The main contributions and innovations of the present application are as follows:

[0050] 1. Significantly improves the expression ability of spatial features: By introducing a multi-plane feature coordination module (MPS), the present application simulates the multi-view reading logic of clinicians, independently extracts and fuses the features of axial, coronal and sagittal planes, effectively integrates heterogeneous information in three-dimensional space, and enhances the understanding of the overall spatial position and morphological structure of the tumor by the model.

[0051] 2. Realize the deep complementarity and anti-interference of multi-modal features: The present application designs a differentiated fusion architecture. In the shallow layer, the orthogonal dimension fusion convolution (ODFC) effectively captures the asymmetric infiltration growth pattern of brain glioma in different directions by decoupling the three orthogonal dimensions, improving the accuracy of boundary division; in the deep layer, the modal introspection-coordination module (MICB) not only purifies single-modal features, but also adaptively adjusts the weight of each modality (such as focusing on FLAIR in the edema area and focusing on T1ce in the enhanced area), effectively solving the feature conflict between modalities and improving the robustness of fusion.

[0052] 3. Optimizes the boundary segmentation accuracy and detail restoration: The encoding information smooth transmission module (EIST) is used instead of the traditional skip connection, and the progressive strategy of step-by-step upsampling and fusion is adopted, avoiding the semantic loss and noise introduction caused by direct large-scale upsampling, ensuring that the deep semantic features can be smoothly and high-fidelity transmitted to the decoder, significantly reducing the Hausdorff distance (HD95), and making the segmentation boundary (especially the junction of the tumor core and the edema area) clearer and closer to the real anatomical structure.

[0053] 4. Excellent comprehensive performance: experimental data show that the Dice coefficient of the application on the BraTS2021 dataset reaches 89.17%, and the HD95 is reduced to 7.79 pixels, and the segmentation accuracy of the whole tumor, tumor core and enhanced tumor area is better than that of the mainstream model TransBTS, nn-UNet, etc., and has high clinical application value.

[0054] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a further understanding of the other features, objectives and advantages of the application. BRIEF DESCRIPTION OF DRAWINGS

[0055] The accompanying drawings illustrated herein are part of the specification and, together with the description, serve to explain the application. The illustrative embodiments of the application and their description serve to explain the application without unduly limiting it. In the drawings:

[0056] Figure 1 is the overall architecture diagram of the multi-modal nuclear magnetic resonance image glioma segmentation method according to the embodiment of the application;

[0057] Figure 2 is the MPS architecture diagram according to the embodiment of the application;

[0058] Figure 3 is the MICB architecture diagram according to the embodiment of the application;

[0059] Figure 4 is the ODFC architecture diagram according to the embodiment of the application;

[0060] Figure 5 is the EIST architecture diagram according to the embodiment of the application;

[0061] Figure 6 is the M 2 ES-UNet and SOTA segmentation model qualitative comparison diagram on BraTS2021 dataset;

[0062] Figure 7 is a schematic diagram of the hardware structure of the electronic device according to the embodiment of the application. DETAILED DESCRIPTION

[0063] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. Unless otherwise indicated, the same numbers in the different drawings indicate the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with one or more embodiments of the present specification. Rather, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of the present specification as detailed in the appended claims.

[0064] It should be noted that the steps of the respective methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the steps included in its method can be more or less than described in this specification. In addition, a single step described in this specification can be broken down into multiple steps for description in other embodiments; while multiple steps described in this specification can be combined into a single step for description in other embodiments.

[0065] Embodiment one: a brain glioma segmentation method based on multi-modal magnetic resonance images

[0066] As Figure 1 shown, the present embodiment provides a M2ES-UNet network model for automatic segmentation of brain glioma. The method as a whole adopts the "encoder-decoder" architecture, but has innovative improvements in the encoding stage, multi-modal fusion stage and skip connection stage. The specific implementation steps are as follows:

[0067] Step S1: data acquisition and preprocessing

[0068] Multi-modal brain MRI image data of the patient to be segmented is obtained. The multi-modal includes T1 (T1 weighted imaging or longitudinal relaxation time image), T2 (T2 weighted imaging or transverse relaxation time image), T1ce (enhanced T1 weighted image) and Flair (fluid attenuated inversion recovery image). After necessary preprocessing (such as registration, skull removal, normalization, etc.), these images are used as the input of the network. The shape of the input tensor is , where C is the number of single modal channels (usually 1), H, W, D are the height, width and depth of the spatial dimension respectively.

[0069] Step S2: single modal independent encoding and multi-planar feature synergy processing

[0070] Unlike traditional methods that directly splice multi-modal images, the present embodiment first extracts features from T1, T2, T1ce and Flair through four independent encoder paths. Each encoding stage includes a residual block (ResBlock) and a multi-planar feature synergy module (MPS).

[0071] As Figure 2 shown, the specific processing process of the multi-planar feature synergy module (MPS) is as follows:

[0072] 1. Channel splitting: considering that the axial plane (D direction, which horizontally cuts the brain from top to bottom) has the best left-right symmetry in clinical diagnosis and is the main basis for tumor diagnosis, the present embodiment splits the input feature map Non-uniform split is performed. Half (0.5C) of the feature channels C is assigned to the axial plane feature extraction branch, and the remaining features are evenly divided (0.25C each) and assigned to the coronal plane (W direction, which vertically and longitudinally cuts the brain from back to front) and the sagittal plane (H direction, which vertically and longitudinally cuts the brain from left to right) feature extraction branches.

[0073] 2. Planar Features Independent Extraction (PFIE):

[0074] In PFIE, L1, L2, L3 represent three planes, and feature extraction is performed on the L3 plane, L3 can be H, W or D, and L1, L2 become the other two planes, respectively. A three-dimensional feature L1xL2xL3 is reshaped into a two-dimensional feature (L1xL2)xL3 containing a certain plane and feature extraction is performed on that plane.

[0075] Each modality will use the MPS module separately, aiming to achieve complementary and synergistic enhancement of features in coronal, axial and sagittal views, and to prepare for subsequent multi-modal fusion.

[0076] In this embodiment, PFIE aims to process single-plane slice information, mine the relationship between slices, and extract features unique to the plane. The original input will be divided into three groups on channel C and input into PFIE for coronal, axial and sagittal feature extraction. Taking the depth direction (axial plane D) as an example, the input is :

[0077]

[0078] wherein C1=0.5C, i.e. half of the original number of channels.

[0079] (1) Query vector Q, key vector K and value vector V are generated by three independent 1x1x1 three-dimensional convolution and dimension reshaping (Reshape) operations:

[0080]

[0081]

[0082]

[0083] wherein PE represents position encoding, which marks the position information of each feature, and Reshape reshapes the features into the corresponding shape; K and V are stacked with position encoding (PE) after convolution through the broadcast mechanism to preserve spatial position information. PE is a learnable feature matrix, and the initial feature values of the matrix satisfy a normal distribution with a mean of 0 and a variance of 1.

[0084] (2) Use scaled dot-product attention to calculate the results of the separate extraction of this planar feature :

[0085] where the softmax formula is , Y is the last dimension of the feature vector, Y i and Y j are the feature values in dimension Y. This step uses attention mechanism to mine the dependency between slices.

[0086] (3) Residual connection of the extracted features with the input: .

[0087] The coronal and sagittal branches perform the same operation, respectively obtaining and :

[0088]

[0089] where C2=0.25C, C3=0.25C, respectively, are one-fourth of the original channel number C.

[0090] Through the above innovative design, PFIE fully refines the information from each plane, focusing on learning the most discriminative features of each plane, avoiding interference between different plane features, which ensures that the fusion stage deals with highly abstract and semantic features, thus enabling more efficient cut surface information fusion.

[0091] 3. Planar Features Complementary Fusion (PFCF): PFCF aims to adaptively adjust the importance of different planes and fuse the essential features of each plane. First, the information extracted from the three planes is concatenated in the channel dimension as the input of this module :

[0092]

[0093] The fusion process is:

[0094] Then, the concatenated features are linearly transformed by 1x1x1 convolution to promote information interaction between different planes, and finally pass through GroupNorm and LeakyRelu activation functions in turn to obtain the enhanced multi-planar features of this modality (the final output ).

[0095] This process simulates the way of comprehensive judgment of clinicians based on multi-planar information, through independent modeling and interactive fusion of each planar feature, effectively integrating the left-right symmetric information of axial plane, the front-back relationship of sagittal plane and the up-down structure of coronal plane, so as to realize consistent understanding of the overall shape of the tumor in the spatial dimension, and enhance the trust and acceptance of the doctor to the model diagnosis process.

[0096] Step S3: Differentiated mid-term feature fusion

[0097] In different levels of the encoder, different fusion strategies are adopted in this embodiment to adapt to the characteristic differences of shallow features and deep features.

[0098] In this embodiment, the innovative architecture of Modality Introspective-Cooperative Block (MICB) and Orthogonal Dimension Fusion Convolution (ODFC) is mainly adopted to replace the traditional simple linear fusion method. The architecture deploys ODFC and MICB at the output end of the encoder at different levels. ODFC acts on the output of the shallow encoder, focusing on realizing the structured feature interaction between modalities at the detail level; compared with the ordinary 3D convolution kernel which is symmetrical in its three spatial dimensions (H, W, D), the infiltrative growth of glioma leads to completely different and asymmetric growth patterns in each dimension, so the ordinary convolution blurs the most discriminative features in a particular direction, while ODFC captures the unique expansion patterns in H, W and D directions through three independent paths, decoupling the information in different dimensions. At the boundary blur, the model can compare the features in the three dimensions to determine the most likely dominant direction of infiltration, thus making more accurate boundary division and flexibly coping with the heterogeneous growth of tumors in different dimensions. Secondly, after dimension permutation, 1x1x1 convolution is also used to overcome the irregularity problem of features whose shapes are not cubes. 1x1x1 convolution can accept irregularly shaped features as input and does not change the feature size.

[0099] MICB acts on the output of the deepest encoder, and perceives and fuses the high-level semantic information of each modality through an adaptive weight mechanism. The two work together to effectively learn the importance weight of different modalities, realize the organic integration of multi-modal features from local to global, and thus significantly improve the robustness and expression ability of fusion under the premise of maintaining data interpretability.

[0100] In this embodiment, the following is specifically implemented:

[0101] 1. Shallow fusion: Orthogonal Dimension Fusion Convolution (ODFC) is used for high-resolution features output by the shallow layers of the encoder, using an ODFC module as shown in Figure 4

[0102] (1) The enhanced multi-planar features of the four modalities are concatenated in the channel dimension to obtain a feature map , where B is the number of batches.

[0103] (2) After the feature map is preliminarily processed by a depthwise separable convolution, the feature tensor is reshaped along the height, width, and depth orthogonal dimensions, respectively, to obtain , . For example, focuses on the structural correlation in the H dimension.

[0104] (3) 1x1x1 convolution is used to extract the unique context of each dimension for . Since glioma growth has asymmetry (such as infiltration along white matter fiber tracts), this decoupling process can effectively capture the infiltration features in a specific direction. The 1x1x1 convolution has input-output shape invariance, which can extract the unique context of each dimension while preserving the structural information.

[0105] (4) The features in the three dimensions are restored to the original shape and concatenated, and then compressed by a 1x1x1 convolution to fuse the multi-dimensional information. Finally, a feedforward neural network (FFN, including LayerNorm, a linear projection layer, a GELU activation function, a second linear projection layer) is used to output the fused features, which complete the linear and nonlinear transformation of the features.

[0106] Unlike traditional 3D convolution, which captures local correlation in fixed (C, H, W, D) dimensional relationships, ODFC analyzes the same 3D tensor from three orthogonal perspectives through dimension permutation, thereby achieving explicit modeling of the global relationship between dimensions. This design enables the model to more comprehensively capture the structural characteristics of tumors in multi-dimensional space, providing richer feature representations for precise segmentation and diagnosis.

[0107] 2. Deep fusion: Modality introspection-collaborative block (MICB)

[0108] For low-resolution, high-semantic features output by the deepest layers of the encoder, a MICB module as shown in Figure 3 is used:

[0109] ​(1) Single-Modal introspection stage (SMIM): First, each modality is processed independently. The feature map is divided into non-overlapping windows, and multi-head self-attention (MSA) is calculated within the window to purify the internal features of the modality. Then, window restoration is performed, and a depth separable convolution (DWConv) is applied to further extract features. Specifically:

[0110] Input feature First, it is converted into a window sequence with a shape of by window partition (WP), where B is the number of window partitions, N is the sequence length of each window, and C is the number of channels. Then, layer normalization function (LayerNorm, LN) and multi-head self-attention (Multi-head Self-Attention, MSA) are applied to each window. MSA projects the input to query (Q), key (K), and value (V) through linear projection and calculates the scaled dot product . Finally, the resulting output of this stage is obtained by connecting the result with the original window input in residual connection , and the formula is as follows:

[0111]

[0112] Then, the is restored to the spatial size by window reverse (Window-Reverse, WR) operation. Then, layer normalization is performed, and the depth separable convolution (Depthwise Separable Convolution, DWConv) module is used for further feature extraction. First, a 1x1x1 convolution is used for linear transformation and channel interaction, and then a depth convolution is used for efficient feature extraction in the spatial dimension. After that, an attention branch is introduced to weight the features to strengthen key information and suppress redundant components.

[0113] Finally, a 1x1x1 convolution is used to complete the cross-channel information integration, and the output result of this stage is obtained. The processing flow is as follows:

[0114]

[0115] It can be seen that the single-modal introspection module provides an independent context learning space for each modality. By calculating the global dependency between all feature points within a single modality, the module realizes self-purification and enhancement of the features within the modality, thereby more systematically and purely highlighting the pathological presentation unique to the modality. For example, the T1ce modality focuses more on depicting the enhanced area after the blood-brain barrier is destroyed, and the Flair modality can capture the boundary of the peritumoral edema. The design of this module explicitly associates the physical meaning of the modality with the discriminative basis of the model, significantly enhancing the rationality and interpretability of the model decision-making process.

[0116] (2) Modality Coordination Module (MCM): aims to learn the importance weights of different modalities. The key is that when different modalities present conflicting features at the same location, it can learn and assign importance weights based on the context information of all modalities, thereby suppressing the interference caused by feature exclusion and forming consistent and reliable fusion representations.

[0117] For the input feature maps of the four different modalities Apply MCM to the 4 inputs, the process is as follows:

[0118] Query (Query) construction: features after introspection from the current modality (such as T1ce) After linear projection .

[0119] Key / Value construction: concatenate the features of all modalities (T1, T2, T1ce, Flair) to generate shared K and V after linear projection.

[0120] Cross-attention calculation: calculate Similarity with global K. For example, in the tumor enhancement area, the matching degree of T1ce's Q with global K is high, and the weight is amplified; in the edema area, the weight of Flair's Q is amplified.

[0121] Weighted fusion: use the calculated weight to perform weighted summation on global V to obtain the fusion feature. This makes the output of each modality fuse the information of all modalities, but retains their respective emphases.

[0122] Specifically, the input feature maps of the four different modalities are windowed and split into four inputs Concatenate them in the spatial feature dimension to get Then and are obtained through linear projection layers to get and :

[0123]

[0124]

[0125]

[0126] respectively, and K T , i.e. the query of a single modality is computed with K, which contains all modality information. A weight is assigned to each position on the concatenated feature of all modalities. Since the query of each modality is independent but the key and value are shared, it forces the model to consider consistency across modalities when fusing. In the region where the boundary is ambiguous, the query result score of T2 and Flair will be higher, while in the enhanced tumor region, the query result of T1ce is higher. Then pass through the activation function and do dot product with

[0127]

[0128] where the softmax formula is , 4N is the feature dimension number, N i is the feature value on the dimension 4N. This formula will amplify the features with high query scores in the last step and suppress the low score features.

[0129] For each modality , the output is the weighted sum of the value V (all modality features), and the weight is determined by the similarity of the query of the modality and the key of all modalities. Therefore, the output of each modality fuses the information of all modalities, but the weight of fusion is determined by the interaction of the modality with all modalities. This allows the model to retain its own modality features while absorbing useful information from other modalities, allowing each modality's features to reference all modalities' global information when fusing, thus adaptively adjusting the importance of each modality at each position, mitigating modality conflicts, and achieving more effective multi-modal feature fusion.

[0130] Subsequently, the is subjected to WR operation for feature remodeling to restore the original spatial dimension, and the is subjected to residual connection to obtain the output .

[0131] The obtained by MCM is passed into LayerNorm for layer normalization, and finally DWConv is used for feature extraction to obtain the final result.

[0132] ​​​​In summary, the two-stage architecture of the MICB module—first single-modal introspection, then multi-modal collaboration—enables to independently extract the discriminative features of each modality, and then perform evidence-based weighting and integration. This architecture avoids the inherent limitations of traditional fusion methods in contribution balancing and conflict resolution.

[0133] Step S4: Smooth Transmission of Encoded Information (EIST)

[0134] like Figure 5 As shown, this embodiment uses an EIST module to replace the traditional skip connections, passing the fused encoded features to the decoder. The EIST module has two inputs: input A comes from the encoder fused features of the current level (such as the previous progressive layer), and input B comes from the output of the EIST module of a deeper level (such as the previous deep layer), that is, it receives two inputs from the encoder and the deeper EIST module.

[0135] 1. Performing a transposed convolution on the input from the encoder doubles the spatial resolution while halving the number of channels to achieve scale alignment with another input. For example, performing a transposed convolution on input A doubles its spatial resolution and halves its number of channels to align it with the scale of input B.

[0136] 2. Spatial attention and channel attention mechanisms are applied in parallel to the two feature paths to enhance key features and suppress redundant information.

[0137] Channel attention: Global average pooling -> MLP -> Sigmoid -> Weighted.

[0138] Spatial attention: Channel average pooling -> Convolution -> Sigmoid -> Weighted.

[0139] 3. Perform dual-path fusion and delivery:

[0140] Path 1: The features processed by the two channels are concatenated, fused through a 1×1×1 convolution, and then passed to the same level decoder (layer).

[0141] Path 2: The features processed by spatial attention from both paths are added element-wise and passed to the shallower EIST module. This step-by-step upsampling and fusion avoids the semantic gap caused by direct large-scale upsampling and achieves a smooth transition from deep semantics to shallow semantics.

[0142] As can be seen, the EIST module workflow is as follows:

[0143] (1) Starting from the deepest layer of the encoder, the feature map is upsampled to make it the same size as the feature map of the next level encoder layer;

[0144] (2) The up-sampled feature map is fused with the feature map of the next encoder layer to form a new feature map;

[0145] (3) The new feature map is up-sampled again and fused with the feature map of the next lower encoder layer, and this is repeated until the corresponding level of the decoder is reached. The information of the encoder is transmitted level by level, and each level is only fused with the features of the adjacent scale, because each level of fusion is between adjacent scales, the semantic gap and resolution gap of the feature map are smaller, and the change of the fused feature is more smooth.

[0146] It can be seen that the EIST module overcomes the information loss and noise introduction problems caused by large up-sampling and down-sampling in traditional skip connections by constructing a progressive information transmission path. The EIST module promotes sufficient interaction between different levels of encoders, realizes cross-level feature complementation, and ensures the integrity and availability of key discriminative information in the transmission process.

[0147] Step S5: The decoding and output decoder receives the features transmitted from the EIST module and the up-sampled features of the previous decoder, and gradually restores the resolution through convolution and up-sampling operations. Finally, the pixel-level classification results are output through the segmentation head, and the necrotic core, peritumoral edema and enhanced tumor regions are identified.

[0148] To verify the effectiveness of the present application, experiments were conducted on the public brain tumor segmentation dataset BraTS2021. Following the common division ratio of 8:2, the data was divided into a training set containing 1000 samples and a test set containing 251 samples. This division strategy ensures the sufficiency of the data for model training, while the test set is also large enough for comprehensive evaluation.

[0149] To evaluate the comprehensive performance of the present application, the present application was compared with a variety of mainstream medical image segmentation models, including nn-UNet, Attention-UNet, TransBTS, S 2 CA-Net and VT-Unet. The experiment used two commonly used segmentation performance evaluation indexes, Dice Similarity Coefficient (DSC) and 95% Hausdorff Distance (HD95), to quantitatively analyze from two aspects of regional overlap precision and boundary segmentation quality. The higher the Dice value represents the better overlap, and the lower the HD95 value represents the smaller boundary error.

[0150] The annotation of each sample in the BraTS2021 dataset contains three types of tumor sub-regions: necrotic core (label value 1), peritumoral edema (label value 2), and enhanced tumor (label value 4). As shown in the quantitative comparison results in Table 1, the performance of each model is reported on the following three key tumor regions: whole tumor region (WT) containing labels 1, 2, and 4; tumor core (TC) containing labels 1 and 4; and enhanced tumor (ET) containing only label 4.

[0151] Table 1 M 2 Segmentation performance of ES-UNet and SOTA segmentation models on the BraTS2021 dataset

[0152]

[0153] Experiments on the BraTS2021 dataset of brain glioma MRI image segmentation (as shown in Table 1) show that the M 2 ES-UNet has a significant advantage in both Dice coefficient and HD95 evaluation indicators. In terms of overall segmentation accuracy, M 2 ES-UNet achieves the best average Dice value of 89.17%, which is higher than the other five mainstream segmentation models, demonstrating its accuracy in identifying the overall tumor region. At the same time, in the HD95 indicator that measures the boundary segmentation quality, M 2 ES-UNet significantly outperforms the other five mainstream models with an average of 7.79 pixels, indicating that its segmentation results are closer to the true label in spatial boundaries.

[0154] From each tumor sub-region, M 2 ES-UNet leads in overall Dice coefficient in WT, TC, and ET three key regions, especially in the complex tumor core region. In terms of boundary accuracy, this method achieves the lowest HD95 value in each region, demonstrating excellent boundary positioning consistency and structure integrity maintenance capability.

[0155] To more intuitively evaluate the segmentation effect, the present application performs visual comparison and analysis on part of the samples on the BraTS2021 dataset, as shown in Figure 6 It can be observed that the existing mainstream methods have obvious limitations in many aspects. nn-UNet and Attention-UNet are prone to miss segmentation when dealing with small target tumors; S 2 CA-UNet generated segmentation boundaries lack smoothness and have structure detail loss; TransBTS and VT-UNet are difficult to completely restore the overall morphological structure of the tumor. In contrast, the M2 The ES-UNet is significantly superior to the above-mentioned five mainstream segmentation methods in visual performance, and has excellent performance in boundary definition, structural integrity preservation, detail restoration capability and sensitivity to small size tumors.

[0156] To further verify the effectiveness of the innovative modules proposed in the application in the brain glioma MRI image segmentation task and their contribution to the overall model performance, the application designs a systematic ablation experiment, as shown in Table 2. By integrating MPS, MICB, ODFC and EIST on the basis of the baseline model respectively, the influence of each module on the segmentation accuracy, boundary discrimination ability and model robustness is comprehensively evaluated.

[0157] Table 2 Ablation experiment results of different improved modules

[0158]

[0159] It can be seen that ODFC significantly improves the boundary quality, especially the HD95 of the TC region is reduced from 12.89 to 8.90, MICB further improves the overall segmentation accuracy, the WT Dice breaks through 91%, the average Dice Dice is further improved by 0.85%, and the HD95 of the TC region is further optimized to 7.25. The MPS module slightly improves the average Dice by 0.14%, but the boundary consistency is better. The average HD95 of EIST is greatly reduced by 1.18 pixels, realizing the best boundary segmentation. Overall, the Dice is improved from 86.97% of the Baseline to 89.17% of the final model, with an absolute improvement of 2.2%. The HD95 is reduced from 12.85 pixels to 7.79 pixels, showing that the boundary positioning is significantly optimized.

[0160] Embodiment Two: A brain glioma segmentation device based on multi-modal magnetic resonance images

[0161] Based on the same concept, the application further proposes a multi-modal magnetic resonance image brain glioma segmentation device, comprising:

[0162] An image acquisition module is configured to acquire T1, T2, T1ce and Flair MRI images of four modalities.

[0163] An encoding module comprises four parallel encoding paths. Each path is integrated with a multi-plane feature coordination unit and is configured to perform the channel splitting, PFIE and PFCF operations described in Embodiment One, and output enhanced multi-plane features of each modality.

[0164] A feature fusion module comprises:

[0165] An orthogonal dimension fusion unit is connected to the shallow output end of the encoding module and is configured to perform the ODFC operation described in Embodiment One.

[0166] The modal introspection and coordination unit is connected to the deep output end of the encoding module and is configured to perform the MICB operation described in Embodiment One.

[0167] The feature transfer module is composed of a plurality of cascaded EIST units and is configured to receive the encoded features and the deep transferred features, perform transpose convolution and double attention screening, and shunt the features to the decoder and the next EIST unit.

[0168] The decoding module is used to reconstruct and output the segmentation mask according to the received feature map.

[0169] Embodiment Three

[0170] The embodiment also provides an electronic device, which refers to Figure 7 The electronic device includes a memory 404 and a processor 402, the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0171] Specifically, the processor 402 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0172] The memory 404 can include a mass storage that stores data or instructions. For example, and without limitation, the memory 404 can include a Hard Disk Drive (HDD), a floppy disk drive, a Solid State Drive (SSD), a flash drive, a Compact Disc Read Only Memory (CD-ROM), a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. The memory 404 can be removable and / or non-removable (or fixed) as appropriate. The memory 404 can be internal or external as appropriate. In particular embodiments, the memory 404 is a Non-Volatile memory. In particular embodiments, the memory 404 includes a Read-Only Memory (ROM) and a Random Access Memory (RAM). The ROM can be a mask-programmed ROM, a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), an Electrically Alterable ROM (EAROM), or a FLASH memory, or a combination of two or more of these, as appropriate. The RAM can be a Static Random-Access Memory (SRAM) or a Dynamic Random Access Memory (DRAM), which can be a Fast Page Mode Dynamic Random Access Memory (FPMDRAM), an Extended Data Output Dynamic Random Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), or the like, as appropriate.

[0173] The memory 404 can be used to store or buffer various data files needed for processing and / or communication, and possible computer program instructions executed by the processor 402.

[0174] The processor 402 implements any one of the above-mentioned multi-modal magnetic resonance image brain glioma segmentation methods by reading and executing the computer program instructions stored in the memory 404.

[0175] Optionally, the above-mentioned electronic device can further include a transmission device 406 connected with the processor 402 and an input / output device 408 connected with the processor 402.

[0176] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network can include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (NIC) which can be connected with other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module which is used to communicate with the Internet in a wireless manner.

[0177] The input / output device 408 is used to input or output information.

[0178] Embodiment Four

[0179] The embodiment also provides a readable storage medium, and the readable storage medium stores a computer program. The computer program includes program codes for controlling a process to execute the process. The process includes the multi-modal magnetic resonance image brain glioma segmentation method according to the first embodiment.

[0180] It should be noted that specific examples in the embodiment can refer to examples described in the above-mentioned embodiments and optional implementation manners, and the embodiment will not be described here again.

[0181] Generally, various embodiments can be implemented in hardware or special-purpose circuitry, software, logic or any combination thereof. Some aspects of the application can be implemented in hardware, while other aspects can be implemented by firmware or software executed by a controller, microprocessor or other computing device, but the application is not limited thereto. Although various aspects of the application can be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein can be implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0182] Embodiments of the application can be implemented by computer software executable by a data processor of the mobile device such as in the processor entity, or by hardware, or by a combination of software and hardware. Computer software or program, also called program product, including software routines, applets and / or macros, can be stored in any apparatus-readable data storage medium and they include program instructions to implement certain tasks. The program product can include one or more computer-executable components tangibly embodied in a computer- readable medium, when executed, for implementing one or more embodiments of the present application. The one or more computer-executable components can be one or more of: a process; a function; a routine; a sag; a subroutine; a plug-in; an app; or a combination of the foregoing. The one or more computer-executable components can each include at least one program instruction. The one or more computer-executable components can each include at least one program instruction. The one or more computer-executable components can each include at least one program instruction. The program instructions of the computer program product can include at least one of a machine code, assembly code, an object code, a source code, or a combination of the foregoing. The program instructions can be executable by the data processor.

[0183] Those skilled in the art should understand that each technical feature of the above embodiments can be combined with any other technical feature, and in order to make the description simple, each technical feature in the above embodiments has not been described in all possible combinations, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present disclosure.

[0184] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as the limitation of the scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for brain glioma segmentation from multi-modal magnetic resonance images, characterized in that, The method comprises the following steps: obtaining multi-modal brain MRI images of a patient to be segmented; constructing a segmentation network model, the segmentation network model comprising an encoder, a decoder and a feature transmission module connected therebetween; independently encoding each modality of the input MRI images using the encoder, and in the encoding process, performing multi-plane feature collaborative processing on the feature maps of each modality: splitting the three-dimensional feature map in the channel dimension, respectively mapping to the coronal plane, the axial plane and the sagittal plane for independent plane feature extraction, and then complementarily coupling and fusing the extracted plane features to obtain enhanced multi-plane features corresponding to each modality; performing mid-term feature fusion on the enhanced multi-plane features of each modality output by the encoder, the mid-term feature fusion comprising: for the enhanced multi-plane features of each modality at the shallow layer of the encoder, adopting an orthogonal dimension fusion strategy to capture and fuse feature correlations in the height, width and depth three orthogonal dimensions; for the enhanced multi-plane features of each modality at the deep layer of the encoder, adopting a modality introspection and collaboration strategy, first performing feature introspection enhancement within a single modality, then calculating the importance weight of each modality based on a cross-modality attention mechanism, and performing weighted collaborative fusion of multi-modality features; using the feature transmission module to perform encoding information smooth transmission, gradually transmitting the fused deep layer encoding features to the decoder by means of progressive upsampling and fusion with features of adjacent scales; using the decoder to decode and reconstruct the received features, and outputting a segmentation result of brain glioma.

2. The multi-modal magnetic resonance image brain glioma segmentation method of claim 1, wherein, In the multi-plane feature collaborative processing, splitting the three-dimensional feature map in the channel dimension specifically comprises: Considering the anatomical symmetry of the axial plane in brain MRI diagnosis, half of the number of channels of the input feature map is allocated to the axial plane feature extraction branch, and the remaining number of channels is evenly allocated to the coronal plane feature extraction branch and the sagittal plane feature extraction branch.

3. The multi-modal magnetic resonance image brain glioma segmentation method of claim 1, wherein, In the multi-plane feature collaborative processing, the independent feature extraction of each plane comprises: Through convolution operation and dimension reshaping, the input three-dimensional feature is converted into a query vector, a key vector and a value vector containing current plane information, wherein the key vector and the value vector fuse position encoding information; using the query vector and the key vector to calculate the attention map of the current plane, and using the attention map to weight the value vector to obtain the significant feature of the plane; performing residual connection on the significant feature and the input feature to obtain the output feature of the plane.

4. The multi-modal magnetic resonance image brain glioma segmentation method of claim 1, wherein, The complementary coupling fusion of the extracted plane features specifically comprises: splicing the independently extracted coronal plane feature, axial plane feature and sagittal plane feature in the channel dimension to obtain multi-plane splicing features; using 1x1x1 convolution to linearly transform the multi-plane splicing features to promote information interaction between different planes; applying group normalization and LeakyRelu activation function in sequence to process the transformed features, and output the enhanced multi-plane features.

5. The multi-modal magnetic resonance image brain glioma segmentation method of claim 1, wherein, The orthogonal dimension fusion strategy for the shallow layer of the encoder specifically comprises: splicing the enhanced multi-plane features of each modality in the channel dimension; The concatenated feature maps are respectively dimensionally reshaped along the height, width and depth directions to obtain three groups of feature tensors with different perspectives; The three groups of feature tensors are respectively convolved using convolution kernels to extract context structure information specific to each dimension; The three groups of convolved feature tensors are restored to the original shape and concatenated in the channel dimension to generate the final shallow fusion features through dimension reduction convolution and feedforward neural network.

6. The multi-modal magnetic resonance image brain glioma segmentation method of claim 1, wherein, The modal introspection and collaboration strategy for the deep layers of the encoder includes a single-modal introspection stage and a modal collaboration stage; The single-modal introspection stage includes dividing the enhanced multi-plane features of each modality into multiple windows, performing multi-head self-attention calculation within the windows, and then performing feature enhancement through window interaction and depth separable convolution; The modal collaboration stage includes taking the features after introspection of each modality as a query vector and taking the concatenation result of all modality features as shared key and value vectors; calculating the correlation score between the query vector of each modality and the shared key vector, and weighting the shared value vector accordingly to obtain collaborative features that fuse global modality information.

7. The multi-modal magnetic resonance image brain glioma segmentation method of any one of claims 1 to 6, wherein, The smooth transfer of encoded information specifically includes: A double-input structure is adopted, with the first input coming from the encoder fusion features of the current level and the second input coming from the output of the transfer module of a deeper level; The first input is subjected to transpose convolution operation to match the scale; Spatial attention mechanism and channel attention mechanism are applied to the two input features respectively to perform feature screening; The screened two features are fused, with part of the fusion result being passed to the decoder of the same level and the other part of the fusion result being used as the input of the next level transfer module.

8. An apparatus for implementing the method of claim 1, characterized in that, It includes: An image acquisition module for acquiring multi-modal brain MRI images of a patient to be segmented; An encoding module configured with multiple single-modal encoding paths for extracting multi-scale features of each modality, and containing a multi-plane feature collaboration unit for splitting the features into coronal, axial and sagittal planes in the channel dimension for independent extraction and complementary fusion, and outputting enhanced multi-plane features corresponding to each modality; A feature fusion module for fusing the enhanced multi-plane features of each modality output by the encoding module, including an orthogonal dimension fusion unit for shallow features and a modal introspection and collaboration unit for deep features; A feature transfer module for using a progressive upsampling strategy to smoothly transfer the fused deep semantic information to the decoder; A decoding module for receiving the transferred features and generating a brain glioma segmentation image. 9.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to run the computer program to perform the multi-modal nuclear magnetic resonance image brain glioma segmentation method of any one of claims 1-7.

10. A readable storage medium, characterized by, The readable storage medium stores a computer program, and the computer program includes program code for controlling a process to perform the process, and the process includes the multi-modal nuclear magnetic resonance image brain glioma segmentation method of any one of claims 1-7.

Images