Glioma segmentation method of multimodal fusion network based on anatomical symmetry guidance

Abstract

The invention belongs to the technical field of medical image processing, and particularly relates to a glioma segmentation method based on a multimodal fusion network guided by anatomical symmetry, which comprises the following steps of: jointly inputting an FLAIR image, a T2 image, a T1 image and a T1c image of the same glioma into a trained image segmentation model, and outputting a predicted segmentation image by the trained image segmentation model, the prediction segmentation image is a glioma MRI image with three segmentation areas obtained through prediction, and the three segmentation areas are an edema area, an enhanced tumor area and a necrosis area respectively; the image segmentation model comprises an encoder, a jump connection part and a decoder; the encoder comprises an ASG module, the jump connection part comprises an IMP module, and the decoder comprises a CMF module. Through a three-module cooperation mechanism, the performance of tumor localization, cross-modal fusion, subregion segmentation and the like is improved, and a reliable image basis is provided for glioma operation plan formulation, prognosis evaluation and personalized treatment decision.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing, and particularly relates to a glioma segmentation method based on an anatomic symmetry guided multi-modal fusion network. BACKGROUND

[0002] Glioma is one of the most invasive primary tumors in the central nervous system (CNS). Accurate delineation of the boundary and sub-region (including edema region, enhanced tumor region and necrosis region) of glioma is crucial for surgical planning, prognosis evaluation and personalized treatment decision.

[0003] Glioma segmentation usually needs to rely on four complementary MRI modalities: FLAIR, T2, T1 and T1c. Among them, FLAIR and T2 sequences are particularly sensitive to tissue fluid content, T2 sequence can show the edema region, and FLAIR sequence can more clearly present the peritumoral lesions by suppressing the cerebrospinal fluid signal; T1 sequence provides clear anatomical structure details, and T1c can highlight the areas of abnormal blood-brain barrier permeability (such as actively proliferating tumor tissue).

[0004] In the field of brain tumor segmentation, existing methods can be roughly divided into three categories. The first category is based on traditional image processing techniques, such as graph cut methods based on fuzzy kernel seed selection, morphological-based methods, etc. The second category is based on machine learning methods, which rely on hand-crafted features (such as texture, intensity and shape) to train classifiers. The third category is based on deep learning models. Compared with traditional image processing techniques and machine learning-based methods, deep learning-based models have the following advantages: automatic feature extraction without the need for manual design, which can capture complex abstract image features to improve accuracy; strong learning ability to handle diverse data and adapt to different patient tumor differences for better segmentation performance; efficient processing with GPU acceleration for real-time processing of large amounts of images to meet the clinical demand for rapid and accurate segmentation; good multi-modal integration to effectively integrate multi-modal data and mine patterns to improve segmentation accuracy and robustness, etc. Therefore, deep learning-based models have become a research hotspot.

[0005] However, deep learning-based models still have some key technical defects: first, the brain has natural anatomic symmetry, and the growth of glioma will destroy this symmetry, but existing deep learning-based models often fail to fully utilize this anatomic prior knowledge, making it difficult to accurately locate the asymmetric regions caused by tumors; second, the four MRI modalities contain complementary information, but existing methods simply concatenate or weight the multi-modal features, failing to fully exploit the synergistic effect of different modalities, resulting in poor fusion results; finally, the edema, enhanced tumor and necrosis regions of glioma have fuzzy boundaries and complex features on images, and existing methods have limited segmentation accuracy for these sub-regions, making it difficult to meet the clinical demand for fine segmentation.

[0006] The document (Brain tumor segmentation with deep convolutional symmetric neural network [J]. Neurocomputing, 2020, 392: 305-313.) adopts a deep convolutional neural network combined with symmetry to automatically segment brain tumors. The neural network is called deep convolutional symmetric neural network (DCSNN), which extends the segmentation network based on deep convolutional neural network (DCNN) by adding left-right symmetric masks in multiple layers. The left-right symmetric mask (LRSM) is calculated by the similarity measure between the left part of the input image and the right part of the same input image, which reflects the symmetric position information of the input image. Although certain achievements have been made, this method still cannot effectively fuse multi-party complementary information data.

[0007] Therefore, although the deep learning-based model has great potential in the field of brain tumor segmentation, given the many deficiencies currently existing and the stringent requirements of clinical application for segmentation accuracy and effect, it is necessary to further improve and optimize the deep learning-based model to break through the technical bottlenecks and promote the development of brain tumor segmentation technology towards more accurate and efficient directions, and to provide more reliable technical support for the diagnosis and treatment of brain tumors. SUMMARY

[0008] The purpose of the present application is to at least solve the problem of the deep learning-based model, and provide a glioma segmentation method based on an anatomic symmetry guided multi-modal fusion network.

[0009] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0010] A glioma segmentation method based on an anatomic symmetry guided multi-modal fusion network, wherein the FLAIR image, T2 image, T1 image and T1c image of the same glioma are input into the trained image segmentation model, and a predicted segmentation image is output from the image segmentation model, wherein the predicted segmentation image is a predicted glioma MRI image with three segmentation regions, and the three segmentation regions are edema region, enhanced tumor region and necrosis region.

[0011] The image segmentation model comprises an encoder, a jump connection part and a decoder.

[0012] The encoder comprises an ASG module (an anatomic symmetry guided module), and the input of the ASG module is a feature , wherein C represents the number of channels, D represents the depth, H represents the height, and W represents the width, and the output is a feature , , wherein the ASG module is obtained by ​ The steps are as follows:

[0013] (a) the is obtained through a max-pooling layer , , and the is obtained through a convolution layer with a convolution kernel of 3x3x3 , ;

[0014] (b) the is respectively obtained through four hollow convolution layers with expansion rates of 1, 2, 3 and 4, and then spliced, and then obtained through a convolution layer with a convolution kernel of 1x1x1 , , and the is obtained through mirror reflection , , and the and are respectively obtained through an IMP module (i.e. an intra-modal purification module) to obtain and , , , and the and are element-wise subtracted and then obtained through a max-pooling layer , ;

[0015] (c) the and are spliced and then obtained through a convolution layer with a convolution kernel of 1x1x1 .

[0016] As a preferred technical solution:

[0017] The IMP module of the glioma segmentation method based on anatomical symmetry guided multi-modal fusion network has an input of feature , and an output of feature , , and the process of obtaining from is as follows: first, the size of is adjusted to obtain , or , and the adjustment means that if the size of exceeds 48x48x48, then is obtained through a max-pooling layer , otherwise, no operation is performed, and then is obtained through two parallel fully connected layers and then matrix multiplication to obtain initial channel correlation weights , , then is refined to get the final channel related weight , then is element-wise multiplied with the weight to get the refined feature , or , then is interpolated to get , , finally is element-wise added with to get ;

[0018] , , , represents element-wise multiplication, represents the identity matrix, represents the pair-wise dependency relationship between the channel in and the channel .

[0019] The glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network as described above, the working process of the encoder is: after the FLAIR image, the T2 image, the T1 image and the T1c image of the same glioma pass through the double convolution module, each ASG module is used to obtain feature map A1, feature map A2, feature map A3 and feature map A4;

[0020] Then, the feature map A1, the feature map A2, the feature map A3 and the feature map A4 pass through an ASG module respectively to obtain feature map B1, feature map B2, feature map B3 and feature map B4;

[0021] Then, the feature map B1, the feature map B2, the feature map B3 and the feature map B4 pass through an ASG module respectively to obtain feature map C1, feature map C2, feature map C3 and feature map C4;

[0022] Then, the feature map C1, the feature map C2, the feature map C3 and the feature map C4 pass through an ASG module respectively to obtain feature map D1, feature map D2, feature map D3 and feature map D4;

[0023] Then, the feature map D1, the feature map D2, the feature map D3 and the feature map D4 are directly spliced to obtain feature map E.

[0024] The working process of the skip connection part is as follows: the feature map A1, the feature map A2, the feature map A3 and the feature map A4 are spliced after each passing through an IMP module to obtain a feature map A;

[0025] Meanwhile, the feature map B1, the feature map B2, the feature map B3 and the feature map B4 are spliced after each passing through an IMP module to obtain a feature map B;

[0026] Meanwhile, the feature map C1, the feature map C2, the feature map C3 and the feature map C4 are spliced after each passing through an IMP module to obtain a feature map C;

[0027] Meanwhile, the feature map D1, the feature map D2, the feature map D3 and the feature map D4 are spliced after each passing through an IMP module to obtain a feature map D.

[0028] The decoder of the glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network comprises a CMF module (i.e., a cross membrane state fusion module), the input of the CMF module is the feature , , and the output is the feature , , and the process of obtaining the feature from the feature is as follows:

[0029] The channels corresponding to the FLAIR image, the T2 image, the T1 image and the T1c image in the feature are respectively denoted as , , and , , , , ;

[0030] The feature passes through a channel-wise max pooling layer to obtain , ;

[0031] The feature passes through a convolution layer with a convolution kernel of 1x1x1 to obtain , ;

[0032] The feature passes through a convolution layer with a convolution kernel of 3x3x3 to obtain , ; the feature passes through a convolution layer with a convolution kernel of 3x3x3 to obtain , ; and After splicing, sequentially pass through the maximum pooling layer and the convolution kernel 1x1x1 convolution layer to obtain , ; after splicing and , sequentially pass through the maximum pooling layer and the convolution kernel 1x1x1 convolution layer to obtain , ; after splicing and , sequentially pass through the convolution kernel 3x3x3 convolution layer and the Sigmod function operation to obtain , ;

[0033] According to the formula , , wherein represents element-wise multiplication, represents element-wise addition.

[0034] The working process of the decoder of the glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network is as follows: the feature map E is sequentially fused with the feature map D after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. The feature map f1 is obtained;

[0035] Then the feature map f1 is sequentially fused with the feature map C after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. The feature map f2 is obtained;

[0036] Then the feature map f2 is sequentially fused with the feature map B after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. The feature map f3 is obtained;

[0037] Then the feature map f3 is sequentially fused with the feature map A after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. The feature map f4 is obtained;

[0038] The feature map f4 is obtained after passing through the double convolution module.

[0039] The expression of the loss function of the image segmentation model of the glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network is as follows:

[0040] ;

[0041] ;

[0042] In the formula, represents the loss function, , , , a degree of coincidence of a predicted region representing an enhanced tumor region and a true region, a degree of coincidence of a predicted region representing a necrosis region and a true region, a degree of coincidence of a predicted region representing an edema region and a true region, a total number of voxels in the image, a pixel a predicted value whether it is a target class, =1 indicates yes, =0 indicates no, a pixel a true value, is 1e-6, is yes to avoid division by 0.

[0043] The glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network as described above, before the FLAIR image, the T2 image, the T1 image and the T1c image of the same glioma are jointly input into the trained image segmentation model, the preprocessing process is: first, the original image with a spatial resolution of 155*240*240 is uniformly adjusted to 192*192*192 by using the center cropping strategy; then, the Z-score normalization (mean=0, standard deviation=1) is used for standardization to enhance the generalization ability of the model; finally, Gaussian noise (standard deviation of 0.1) is introduced to further improve the robustness of the model.

[0044] The training steps of the image segmentation model in the glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network as described above are as follows:

[0045] (a) collect glioma cases, ≥285, each glioma case has a FLAIR image, a T2 image, a T1 image and a T1c image;

[0046] (b) pre-process the FLAIR image, the T2 image, the T1 image and the T1c image; obtain the real segmentation image of each glioma case, the real segmentation image is the glioma MRI image with the three segmentation regions obtained by manual annotation;

[0047] (c) construct a training set and a test set using the FLAIR image, the T2 image, the T1 image, the T1c image and the real segmentation image corresponding to glioma cases;

[0048] (d) training the image segmentation model using the training set, during the training, taking the FLAIR image, the T2 image, the T1 image and the T1c image as the input of the image segmentation model, taking the real segmentation image as the theoretical output of the image segmentation model, constantly adjusting the weight parameters of the image segmentation model until the image segmentation model converges;

[0049] (e) testing the trained image segmentation model using the test set.

[0050] Principle of the application:

[0051] In view of the problem that the existing deep learning-based model is difficult to effectively model the anatomical symmetry destruction, the ASG module is introduced in the encoder, the module generates symmetric features through mirror flipping of the features, performs element-by-element subtraction after processing by the IMP module, captures the anatomical asymmetry information caused by glioma, and combines the context features extracted by the multi-scale hollow convolution to integrate the symmetry destruction information into the feature extraction process, thereby improving the accuracy of tumor region positioning.

[0052] In view of the problem that the existing deep learning-based model is difficult to realize effective cross-modal fusion, the CMF module is introduced in the decoder, the module classifies and processes the feature channels corresponding to the FLAIR image, the T2 image, the T1 image and the T1c image, enhances the attention mechanism of the student mode (T2, T1) fusion through the teacher mode (FLAIR, T1c), generates adaptive weights to realize dynamic integration of multi-modal features, and solves the problem of inefficient fusion caused by simple splicing.

[0053] In view of the problem that the existing deep learning-based model is difficult to realize fine segmentation in the glioma sub-region, the IMP module is introduced in the jump connection part, the module calculates the channel correlation weight and suppresses the autocorrelation interference, enhances the tumor-related features and suppresses irrelevant information, and simultaneously cooperates with the ASG module in the encoder and the CMF module in the decoder to improve the feature expression ability from three dimensions of anatomical prior, modal complementation and feature purification, and accurately captures the edema region, enhances the boundary and details of the tumor region and the necrotic region.

[0054] Advantages:

[0055] 1、The ASG module in the encoder of the present application generates symmetric features through mirror flipping, and performs element-by-element subtraction after processing by the IMP module, which can effectively capture the brain anatomical asymmetry information caused by glioma, and simultaneously combine the context features extracted by the multi-scale hollow convolution to integrate the symmetry destruction information into the feature extraction process, thereby solving the problem that the existing method is difficult to utilize anatomical prior knowledge, and significantly improving the positioning accuracy of the tumor region.

[0056] 2、The CMF module in the decoder of the application classifies and processes the feature channels corresponding to the FLAIR image, the T2 image, the T1 image and the T1c image, generates adaptive weights through the attention mechanism of the teacher mode (FLAIR, T1c) and the student mode (T2, T1), realizes the dynamic integration of multi-modal complementary information, overcomes the fusion inefficiency problem caused by simple splicing or weighting of the prior art, and fully excavates the synergistic effect of different modes.

[0057] 3、The IMP module of the jump connection part of the application enhances tumor-related features and suppresses irrelevant information by calculating channel correlation weights and suppressing autocorrelation interference; meanwhile, the module cooperates with the ASG module and the CMF module to improve feature expression ability from three dimensions of anatomical prior, modal complementarity and feature purification, effectively solves the problems of blurred boundaries of the edema region, enhanced tumor region and necrotic region of the glioma and limited segmentation accuracy, and meets the needs of clinical sub-regional fine segmentation.

[0058] 4、Compared with the traditional method, the machine learning method and the existing deep learning-based model, the three-module cooperative mechanism of the application realizes performance improvement in tumor positioning, cross-modal fusion and sub-regional segmentation, and provides more reliable image basis for the surgical plan of glioma, prognosis evaluation and personalized treatment decision. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 It is a model structure schematic diagram of the glioma segmentation method of the application;

[0060] Figure 2 It is a model structure schematic diagram of the ASG module of the application;

[0061] Figure 3 It is a model structure schematic diagram of the IMP module of the application;

[0062] Figure 4 It is a model structure schematic diagram of the CMF module of the application;

[0063] Figure 5 It is a model structure schematic diagram of the glioma segmentation method of Comparative Example 1;

[0064] Figure 6 It is a model structure schematic diagram of the glioma segmentation method of Comparative Example 2;

[0065] Figure 7 It is a model structure schematic diagram of the glioma segmentation method of Comparative Example 3;

[0066] Figure 8 It is a model structure schematic diagram of the glioma segmentation method of Comparative Example 4;

[0067] Figure 9A schematic diagram of a model structure of a glioma segmentation method of Comparative Example 5;

[0068] Figure 10 A schematic diagram of a model structure of a glioma segmentation method of Comparative Example 6;

[0069] Figure 11 A schematic diagram of a model structure of a glioma segmentation method of Comparative Example 7;

[0070] Figure 12 A visualization diagram of a module segmentation performance comparison of Example 1 and Comparative Examples 1 to 4 (in the segmentation results of Example 1 and Comparative Examples 1 to 4, the numerical value in the upper left corner is the DSC average value of the necrotic area, the enhanced tumor area and the edema area); the position marked by the purple circle in the diagram is the area where the predicted segmentation image of the different models and the true value image exist differences;

[0071] Figure 13 A visualization diagram of a module segmentation performance comparison of Example 1 and Comparative Examples 1 to 4 (in the segmentation results of Example 1 and Comparative Examples 1 to 4, the numerical value in the upper left corner is the DSC average value of the necrotic area, the enhanced tumor area and the edema area); the position marked by the purple circle in the diagram is the area where the predicted segmentation image of the different models and the true value image exist differences; Figure 1 An enlarged view of the input image and the output image shown in DETAILED DESCRIPTION

[0072] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not used to limit the scope of the application. In addition, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content taught by the application, and these equivalent forms also fall within the scope defined by the claims attached to the application.

[0073] Example 1

[0074] A glioma segmentation method based on an anatomical symmetry guided multi-modal fusion network, the steps are as follows:

[0075] Step 1, establishing an image segmentation model;

[0076] As shown in Figure 1 , the image segmentation model includes an encoder, a skip connection part and a decoder; Figure 1 Figure 13 As shown in , the encoder includes an ASG module, the input of the ASG module is a feature

[0077] , , wherein C represents the number of channels, D represents the depth, H represents the height, and W represents the width, and the output is a feature , As shown in Figure 2 , the feature is obtained from ​The steps are as follows:

[0078] (a) the feature is obtained by passing through a max-pooling layer, , the feature is obtained by passing through a convolutional layer with a convolution kernel of 3x3x3, , ;

[0079] (b) the feature is obtained by passing through four dilated convolutional layers with dilation rates of 1, 2, 3, and 4 respectively, followed by concatenation, and then passing through a convolutional layer with a convolution kernel of 1x1x1, , the feature is obtained by mirror flipping, , the features and are obtained by passing through an IMP module in sequence, and , , the feature is obtained by element-wise subtraction between and , ;

[0080] wherein the input of the IMP module is the feature , and the output is the feature , the process of obtaining from is as follows: as shown in Figure 3 , first, the size of is adjusted to obtain , or , then is passed through two parallel fully connected layers and then matrix multiplication is performed to obtain the initial channel correlation weight , the final channel correlation weight is obtained by refining , the feature is obtained by element-wise multiplication between and the weight , or , then is interpolated to obtain , , and finally is obtained by element-wise addition between and ;

[0081] , , , represents element-wise multiplication, represents an identity matrix, represents middle channel and channel between pairs of dependencies;

[0082] (c) after splicing and , a convolution layer with a convolution kernel of 1x1x1 is obtained ;

[0083] As shown in Figure 1 , the working process of the encoder is: after the FLAIR image, the T2 image, the T1 image and the T1c image of the same glioma pass through the double convolution module, each passes through an ASG module to obtain feature map A1, feature map A2, feature map A3 and feature map A4;

[0084] Then, the feature map A1, the feature map A2, the feature map A3 and the feature map A4 each pass through an ASG module to obtain feature map B1, feature map B2, feature map B3 and feature map B4;

[0085] Then, the feature map B1, the feature map B2, the feature map B3 and the feature map B4 each pass through an ASG module to obtain feature map C1, feature map C2, feature map C3 and feature map C4;

[0086] Then, the feature map C1, the feature map C2, the feature map C3 and the feature map C4 each pass through an ASG module to obtain feature map D1, feature map D2, feature map D3 and feature map D4;

[0087] Then, the feature map D1, the feature map D2, the feature map D3 and the feature map D4 are directly spliced to obtain feature map E;

[0088] The working process of the skip connection part is: after the feature map A1, the feature map A2, the feature map A3 and the feature map A4 each pass through an IMP module, they are spliced to obtain feature map A;

[0089] At the same time, after the feature map B1, the feature map B2, the feature map B3 and the feature map B4 each pass through an IMP module, they are spliced to obtain feature map B;

[0090] At the same time, after the feature map C1, the feature map C2, the feature map C3 and the feature map C4 each pass through an IMP module, they are spliced to obtain feature map C;

[0091] At the same time, feature map D1, feature map D2, feature map D3, and feature map D4 are each passed through an IMP module and then spliced ​​to obtain feature map D;

[0092] The decoder includes a CMF module, the input of which is the feature , , the output is the feature , ,like Figure 4 As shown by get The process is:

[0093] Will The channels corresponding to FLAIR images, T2 images, T1 images, and T1c images are respectively recorded as 、 、 and , , , , ;

[0094] Will After the channel-wise maximum pooling layer, , ;

[0095] Will After the convolution layer with a convolution kernel of 1×1×1, we get , ;

[0096] Will After the convolution layer with a convolution kernel of 3×3×3, we get , ;Will After the convolution layer with a convolution kernel of 3×3×3, we get , ;Will and After splicing, it passes through the maximum pooling layer and the convolution layer with a convolution kernel of 1×1×1 to obtain , ;Will and After splicing, it passes through the maximum pooling layer and the convolution layer with a convolution kernel of 1×1×1 to obtain , ;Will and After splicing, it is sequentially processed through a convolution layer with a convolution kernel of 3×3×3 and a Sigmod function to obtain , ;

[0097] According to the formula Obtained , wherein, represents element-by-element multiplication, represents element-by-element addition;

[0098] As shown in Figure 1 , the working process of the decoder is: the feature map E is sequentially fused with the feature map D after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. Feature map f1;

[0099] Then the feature map f1 is sequentially fused with the feature map C after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. Feature map f2;

[0100] Then the feature map f2 is sequentially fused with the feature map B after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. Feature map f3;

[0101] Then the feature map f3 is sequentially fused with the feature map A after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. Feature map f4;

[0102] The feature map f4 is obtained after passing through the double convolution module. Predicted segmentation image;

[0103] Step two, determine the loss function of the image segmentation model;

[0104] The expression of the loss function of the image segmentation model is as follows:

[0105] ;

[0106] ;

[0107] In the formula, represents the loss function, , , , represents the degree of coincidence between the predicted area and the real area of the enhanced tumor area, represents the degree of coincidence between the predicted area and the real area of the necrotic area, represents the degree of coincidence between the predicted area and the real area of the edema area, represents the total number of voxels in the image, represents the predicted value of whether the pixel is a target category, =1 indicates yes, =0 indicates no, represents the true value of the pixel , 1e-6;

[0108] Step three, training the image segmentation model;

[0109] (a) Collecting 285 glioma cases (the data source is the dataset BraTS 2018 of brain tumor segmentation challenge, which is widely used in brain tumor segmentation task), each of which has FLAIR image, T2 image, T1 image and T1c image;

[0110] (b) Preprocessing the FLAIR image, T2 image, T1 image and T1c image (firstly, the original image with a spatial resolution of 155x240x240 is uniformly adjusted to 192x192x192 by using the center cropping strategy; then, the image is standardized by using the Z-score normalization method (the mean is set to 0 and the standard deviation is set to 1); finally, Gaussian noise with a standard deviation of 0.1 is introduced to the image); obtaining the real segmentation image of each glioma case, which is the glioma MRI image with the three segmentation regions (edema region, enhanced tumor region and necrosis region) obtained by manual annotation;

[0111] (c) Constructing the training set and test set (the number ratio of brain tumor cases in the training set and test set is 8:2) by using the FLAIR image, T2 image, T1 image, T1c image and real segmentation image corresponding to the 285 glioma cases;

[0112] (d) Training the image segmentation model by using the training set, during which the FLAIR image, T2 image, T1 image and T1c image are used as the input of the image segmentation model, the real segmentation image is used as the theoretical output of the image segmentation model, and the weight parameters of the image segmentation model are constantly adjusted until the image segmentation model converges;

[0113] (e) Testing the trained image segmentation model by using the test set; the dice similarity coefficient (DSC) and 95% Hausdorff distance (HD) are used to represent the segmentation accuracy of the trained image segmentation model, and the calculation formula is as follows:

[0114] ;

[0115] In the formula, A and B represent the predicted region (i.e. the whole tumor region, the tumor core region or the enhanced tumor region; the whole tumor region = edema region + enhanced tumor region + necrosis region, the tumor core region = enhanced tumor region + necrosis region) and the real region, represents the intersection size of the predicted region and the real region, and represent the total size of the predicted region and the real region, respectively;

[0116] ;

[0117] wherein X and Y represent the set of boundary points of the predicted region and the set of boundary points of the real region respectively, d(x, y) represents the Euclidean distance between point x and point y, represents the maximum value of the minimum distance from each point x of the predicted region boundary to the real region boundary, represents the maximum value of the minimum distance from each point y of the real region boundary to the predicted region boundary;

[0118] Step four, output the predicted segmentation image;

[0119] The FLAIR image, the T2 image, the T1 image and the T1c image of the same glioma are collectively input into the trained image segmentation model, and a predicted segmentation image is output therefrom, the predicted segmentation image being a glioma MRI image with 3 segmented regions obtained by prediction, the 3 segmented regions being an edema region, an enhanced tumor region and a necrosis region.

[0120] Comparative Example 1

[0121] A glioma segmentation method, which is basically the same as that of Example 1, except that the image segmentation model is as shown in Figure 5 compared with the image segmentation model of Example 1, the encoder does not contain the ASG module, the skip connection part does not contain the IMP module, and the decoder does not contain the CMF module.

[0122] Comparative Example 2

[0123] A glioma segmentation method, which is basically the same as that of Example 1, except that the image segmentation model is as shown in Figure 6 compared with the image segmentation model of Example 1, the skip connection part does not contain the IMP module, and the decoder does not contain the CMF module.

[0124] Comparative Example 3

[0125] A glioma segmentation method, which is basically the same as that of Example 1, except that the image segmentation model is as shown in Figure 7 compared with the image segmentation model of Example 1, the encoder does not contain the ASG module, and the decoder does not contain the CMF module.

[0126] Comparative Example 4

[0127] A glioma segmentation method, which is basically the same as that of Example 1, except that the image segmentation model is as shown in Figure 8 compared with the image segmentation model of Example 1, the encoder does not contain the ASG module, and the skip connection part does not contain the IMP module.

[0128] Comparative Example 5

[0129] A glioma segmentation method, substantially the same as embodiment 1, except that the image segmentation model is as shown in Figure 9 which, relative to the image segmentation model of embodiment 1, the decoder does not contain the CMF module.

[0130] Comparative example 6

[0131] A glioma segmentation method, substantially the same as embodiment 1, except that the image segmentation model is as shown in Figure 10 which, relative to the image segmentation model of embodiment 1, the skip connection part does not contain the IMP module.

[0132] Comparative example 7

[0133] A glioma segmentation method, substantially the same as embodiment 1, except that the image segmentation model is as shown in Figure 11 which, relative to the image segmentation model of embodiment 1, the encoder does not contain the ASG module.

[0134] The test results of embodiment 1 and comparative examples 1-7 are shown in Table 1 (bold represents the best result; - represents that there is no such module in the model; √ represents that there is such module in the model; * represents that statistical significant improvement compared with the baseline method is achieved using Wilcoxon signed rank test (p<0.05)):

[0135] Table 1

[0136]

[0137] As shown in Table 1, the baseline model, Comparison Example 1 (which does not use the ASG, IMP, or CMF modules), achieves the lowest average DSC scores for the entire tumor region, the tumor core region, and the enhanced tumor region, and the highest average HD scores. Comparison Example 2 and Comparison Example 3, which use only the ASG module and the IMP module, achieve significant improvements in DSC scores for the tumor core and enhanced tumor regions. Comparison Example 4, which uses only the CMF module, achieves significant improvements in DSC scores for the entire tumor region and the tumor core region. This demonstrates that the ASG module in the encoder effectively guides the network to focus on asymmetric regions of the image; the IMP module in the skip connection optimizes feature maps across modalities; and the CMF module in the decoder enhances the importance of the teacher modality through modal fusion. Its internal compression-expansion mechanism also effectively represents important information, thereby improving segmentation accuracy. Comparison Example 5, which uses both the ASG and IMP modules, Comparison Example 6, which uses both the ASG and CMF modules, and Comparison Example 7, which uses both the IMP and CMF modules, all perform better overall than the models using only one module. Finally, Example 1, which uses the ASG module, IMP module, and CMF module at the same time, achieved the highest values ​​in DSC in the entire tumor area, tumor core area, and enhanced tumor area, with an average DSC value of 84.4%. It also performed well in HD in these three areas, with the average HD value reduced to 4.27 mm, which is a significant improvement compared to other models. This fully demonstrates that the synergistic effect of the three modules can maximize the glioma segmentation effect.

[0138] To further verify the above conclusions, we conducted a visual analysis of the predicted segmentation images of two brain tumor cases based on the FLAIR images of the BraTS2018 dataset (e.g. Figure 12 In the first case ( Figure 12 In row a of the figure, the baseline model (Comparative Example 1) has the problem of missing detection of necrotic areas. Comparative Example 2, which introduces the ASG module, Comparative Example 3, and Comparative Example 4, which introduce the CMF module, can improve this problem. Among them, Example 1, which integrates the three modules at the same time, has the highest degree of fit with the true value map in the segmentation boundaries of the necrotic area (red), enhanced tumor area (yellow), and edema area (green); in the second case ( Figure 12 In row (b), the baseline model (Comparative Example 1) over-segments the necrotic region. This problem is alleviated after the modules are introduced in Comparative Examples 2-4. The predicted segmented image of Example 1 is highly consistent with the ground-truth image, with minimal differences (marked by purple circles). This intuitive result is consistent with the quantitative indicator trends in Table 1, further confirming the effectiveness of the ASG module, IMP module, CMF module, and their synergy in improving glioma segmentation accuracy.

Claims

1. A glioma segmentation method based on a multimodal fusion network guided by anatomical symmetry, characterized in that: The FLAIR image, T2 image, T1 image, and T1c image of the same glioma are input into the trained image segmentation model, which outputs a predicted segmentation image. The predicted segmentation image is a glioma MRI image with three segmentation regions obtained through prediction. The three segmentation regions are edema region, enhanced tumor region, and necrosis region. The image segmentation model consists of an encoder, a skip connection part, and a decoder; The encoder includes an ASG module, the input of the ASG module is the feature , , where C represents the number of channels, D represents the depth, H represents the height, and W represents the width. The output is the feature , ,Depend on get The steps are as follows: (a) After the maximum pooling layer, we get , , and then After the convolution layer with a convolution kernel of 3×3×3, we get , ; (b) After passing through four dilation rates of 1, 2, 3, and 4 respectively, they are spliced ​​and then passed through a convolution layer with a convolution kernel of 1×1×1 to obtain , , and then Perform mirror flip to obtain , , and then and Get through the IMP module in turn and , , , then and After element-by-element subtraction, we get , ; (c) and After splicing, the convolution layer with a convolution kernel of 1×1×1 is obtained .

2. The glioma segmentation method based on a multimodal fusion network guided by anatomical symmetry according to claim 1, characterized in that: The input of the IMP module is the feature , , the output is the feature , ,Depend on get The process is: first adjust The size of , or , then The initial channel correlation weights are obtained by performing matrix multiplication after two parallel fully connected layers , , then Refine to get the final channel-related weights , and then With weight Perform element-wise multiplication to obtain features , or , then Interpolation is performed to obtain , , and finally and Add element by element to get ; , , , represents element-wise multiplication, represents the identity matrix, represent Middle Channel and channel The pairwise dependencies between them.

3. The glioma segmentation method based on a multimodal fusion network guided by anatomical symmetry according to claim 2, characterized in that: The encoder works as follows: After the FLAIR image, T2 image, T1 image, and T1c image of the same glioma are passed through a double convolution module, each of them is passed through an ASG module to obtain feature maps A1, A2, A3, and A4; Then, feature map A1, feature map A2, feature map A3, and feature map A4 are each passed through an ASG module to obtain feature map B1, feature map B2, feature map B3, and feature map B4; Then, feature map B1, feature map B2, feature map B3, and feature map B4 are each passed through an ASG module to obtain feature map C1, feature map C2, feature map C3, and feature map C4; Then, feature maps C1, C2, C3, and C4 are each passed through an ASG module to obtain feature maps D1, D2, D3, and D4; Then directly concatenate the feature maps D1, D2, D3, and D4 to obtain the feature map E.

4. The glioma segmentation method based on a multimodal fusion network guided by anatomical symmetry according to claim 3, characterized in that: The working process of the jump connection part is: feature map A1, feature map A2, feature map A3, and feature map A4 are each passed through an IMP module and then spliced ​​to obtain feature map A; At the same time, feature map B1, feature map B2, feature map B3, and feature map B4 are each passed through an IMP module and then spliced ​​to obtain feature map B; At the same time, feature map C1, feature map C2, feature map C3, and feature map C4 are each passed through an IMP module and then spliced ​​to obtain feature map C; At the same time, feature map D1, feature map D2, feature map D3, and feature map D4 are each passed through an IMP module and then spliced ​​together to obtain feature map D.

5. The glioma segmentation method based on a multimodal fusion network guided by anatomical symmetry according to claim 4, characterized in that: The decoder includes a CMF module, the input of which is the feature , , the output is the feature , ,Depend on get The process is: Will The channels corresponding to FLAIR images, T2 images, T1 images, and T1c images are respectively recorded as 、 、 and , , , , ; Will After the channel-wise maximum pooling layer, , ; Will After the convolution layer with a convolution kernel of 1×1×1, we get , ; Will After the convolution layer with a convolution kernel of 3×3×3, we get , ;Will After the convolution layer with a convolution kernel of 3×3×3, we get , ;Will and After splicing, it passes through the maximum pooling layer and the convolution layer with a convolution kernel of 1×1×1 to obtain , ;Will and After splicing, it passes through the maximum pooling layer and the convolution layer with a convolution kernel of 1×1×1 to obtain , ;Will and After splicing, it is sequentially processed through a convolution layer with a convolution kernel of 3×3×3 and a Sigmod function to obtain , ; According to the formula get , where represents element-wise multiplication, Stands for element-wise addition.

6. The glioma segmentation method based on a multimodal fusion network guided by anatomical symmetry according to claim 5, characterized in that: The working process of the decoder is as follows: the feature map E is sequentially passed through the CMF module and the transposed convolution module and then fused with the feature map D. The fusion result is passed through the double convolution module to obtain the feature map f1; The feature map f1 is then passed through the CMF module and the transposed convolution module in sequence and fused with the feature map C. The fusion result is passed through the double convolution module to obtain the feature map f2; The feature map f2 is then passed through the CMF module and the transposed convolution module in sequence and fused with the feature map B. The fusion result is passed through the double convolution module to obtain the feature map f3; The feature map f3 is then passed through the CMF module and the transposed convolution module in sequence and fused with the feature map A. The fusion result is passed through the double convolution module to obtain the feature map f4; The feature map f4 is passed through a double convolution module to obtain the predicted segmentation image.

7. The glioma segmentation method based on a multimodal fusion network guided by anatomical symmetry according to claim 6, characterized in that: The loss function of the image segmentation model is expressed as follows: ； ； Where, represents the loss function, , , , represents the degree of overlap between the predicted area and the true area of ​​the enhanced tumor area, represents the degree of overlap between the predicted area and the actual area of ​​necrosis, Represents the degree of overlap between the predicted area and the real area of ​​the edema area, represents the total number of voxels in the image, Represents pixels Is it the predicted value of the target category? =1 means yes, =0 means no, Represents pixels The truth value of is 1e-6.

8. The glioma segmentation method based on a multimodal fusion network guided by anatomical symmetry according to claim 1, characterized in that: The FLAIR, T2, T1, and T1c images of the same glioma were preprocessed before being fed into the trained image segmentation model. The preprocessing process was as follows: first, the original images with a spatial resolution of 155 × 240 × 240 were uniformly resized to 192 × 192 × 192 using a center cropping strategy; then, they were normalized using Z-score normalization; finally, Gaussian noise was introduced.

9. The glioma segmentation method based on a multimodal fusion network guided by anatomical symmetry according to claim 8, characterized in that: The training steps of the image segmentation model are as follows: (a) Collection glioma cases, ≥285, each glioma case had FLAIR images, T2 images, T1 images, and T1c images; (b) performing the preprocessing on the FLAIR image, T2 image, T1 image, and T1c image; obtaining a true segmentation image of each glioma case, where the true segmentation image is a glioma MRI image with the three segmentation regions obtained by manual annotation; (c) Adoption The training and test sets were constructed using FLAIR images, T2 images, T1 images, T1c images, and true segmentation images corresponding to each glioma case. (d) Using the training set to train the image segmentation model, during training, the FLAIR images, T2 images, T1 images, and T1c images are used as the input of the image segmentation model, and the real segmented images are used as the theoretical output of the image segmentation model. The weight parameters of the image segmentation model are continuously adjusted until the image segmentation model converges; (e) Use the test set to test the trained image segmentation model.

Images