A glioma segmentation method based on a multi-modal fusion network guided by anatomical symmetry

Abstract

The present application belongs to the technical field of medical image processing, and particularly relates to a glioma segmentation method based on an anatomic symmetry guided multi-modal fusion network, which comprises the following steps: inputting FLAIR images, T2 images, T1 images and T1c images of the same glioma into a trained image segmentation model, and outputting a predicted segmentation image from the image segmentation model, wherein the predicted segmentation image is a glioma MRI image with three segmentation regions obtained by prediction, and the three segmentation regions are an edema region, an enhanced tumor region and a necrosis region; the image segmentation model comprises an encoder, a skip connection part and a decoder; the encoder comprises an ASG module, the skip connection part comprises an IMP module, and the decoder comprises a CMF module; the present application realizes performance improvement in tumor positioning, cross-modal fusion and sub-region segmentation through a three-module collaborative mechanism, and provides reliable image basis for surgery plan making, prognosis evaluation and personalized treatment decision of glioma.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing, and particularly relates to a glioma segmentation method based on an anatomic symmetry guided multi-modal fusion network. BACKGROUND

[0002] Glioma is one of the most invasive primary tumors in the central nervous system (CNS). Accurate delineation of the boundary and sub-region (including edema region, enhanced tumor region and necrosis region) of glioma is crucial for surgical planning, prognosis evaluation and personalized treatment decision.

[0003] Glioma segmentation usually needs to rely on four complementary MRI modalities: FLAIR, T2, T1 and T1c. Among them, FLAIR and T2 sequences are particularly sensitive to tissue fluid content, T2 sequence can show the edema region, and FLAIR sequence can more clearly present the peritumoral lesions by suppressing the cerebrospinal fluid signal; T1 sequence provides clear anatomical structure details, and T1c can highlight the areas of abnormal blood-brain barrier permeability (such as actively proliferating tumor tissue).

[0004] In the field of brain tumor segmentation, existing methods can be roughly divided into three categories. The first category is based on traditional image processing techniques, such as graph cut methods based on fuzzy kernel seed selection, morphological-based methods, etc. The second category is based on machine learning methods, which rely on hand-crafted features (such as texture, intensity and shape) to train classifiers. The third category is based on deep learning models. Compared with traditional image processing techniques and machine learning-based methods, deep learning-based models have the following advantages: automatic feature extraction without the need for manual design, which can capture complex abstract image features to improve accuracy; strong learning ability to handle diverse data and adapt to different patient tumor differences for better segmentation performance; efficient processing with GPU acceleration for real-time processing of large amounts of images to meet the clinical demand for rapid and accurate segmentation; good multi-modal integration to effectively integrate multi-modal data and mine patterns to improve segmentation accuracy and robustness, etc. Therefore, deep learning-based models have become a research hotspot.

[0005] However, deep learning-based models still have some key technical defects: first, the brain has natural anatomic symmetry, and the growth of glioma will destroy this symmetry, but existing deep learning-based models often fail to fully utilize this anatomic prior knowledge, making it difficult to accurately locate the asymmetric regions caused by tumors; second, the four MRI modalities contain complementary information, but existing methods simply concatenate or weight the multi-modal features, failing to fully exploit the synergistic effect of different modalities, resulting in poor fusion results; finally, the edema, enhanced tumor and necrosis regions of glioma have fuzzy boundaries and complex features on images, and existing methods have limited segmentation accuracy for these sub-regions, making it difficult to meet the clinical demand for fine segmentation.

[0006] The document (Brain tumor segmentation with deep convolutional symmetric neural network [J]. Neurocomputing, 2020, 392: 305-313.) adopts a deep convolutional neural network combined with symmetry to automatically segment brain tumors. The neural network is called deep convolutional symmetric neural network (DCSNN), which extends the segmentation network based on deep convolutional neural network (DCNN) by adding left-right symmetric masks in multiple layers. The left-right symmetric mask (LRSM) is calculated by the similarity measure between the left part of the input image and the right part of the same input image, which reflects the symmetric position information of the input image. Although certain achievements have been made, this method still cannot effectively fuse multi-party complementary information data.

[0007] Therefore, although the deep learning-based model has great potential in the field of brain tumor segmentation, given the many deficiencies currently existing and the stringent requirements of clinical application for segmentation accuracy and effect, it is necessary to further improve and optimize the deep learning-based model to break through the technical bottlenecks and promote the development of brain tumor segmentation technology towards more accurate and efficient directions, and to provide more reliable technical support for the diagnosis and treatment of brain tumors. SUMMARY

[0008] The purpose of the present application is to at least solve the problem of the deep learning-based model, and provide a glioma segmentation method based on an anatomic symmetry guided multi-modal fusion network.

[0009] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0010] A glioma segmentation method based on an anatomic symmetry guided multi-modal fusion network, wherein the FLAIR image, T2 image, T1 image and T1c image of the same glioma are input into the trained image segmentation model, and a predicted segmentation image is output from the image segmentation model, wherein the predicted segmentation image is a predicted glioma MRI image with three segmentation regions, and the three segmentation regions are edema region, enhanced tumor region and necrosis region.

[0011] The image segmentation model comprises an encoder, a jump connection part and a decoder.

[0012] The encoder comprises an ASG module (an anatomic symmetry guided module), and the input of the ASG module is a feature , wherein C represents the number of channels, D represents the depth, H represents the height, and W represents the width, and the output is a feature , , wherein the ASG module is obtained by ​ The steps are as follows:

[0013] (a) the is obtained through a max-pooling layer , , and the is obtained through a convolution layer with a convolution kernel of 3x3x3 , ;

[0014] (b) the is respectively obtained through four hollow convolution layers with expansion rates of 1, 2, 3 and 4, and then spliced, and then obtained through a convolution layer with a convolution kernel of 1x1x1 , , and the is obtained through mirror reflection , , and the and are respectively obtained through an IMP module (i.e. an intra-modal purification module) to obtain and , , , and the and are element-wise subtracted and then obtained through a max-pooling layer , ;

[0015] (c) the and are spliced and then obtained through a convolution layer with a convolution kernel of 1x1x1 .

[0016] As a preferred technical solution:

[0017] The IMP module of the glioma segmentation method based on anatomical symmetry guided multi-modal fusion network has an input of feature , and an output of feature , , and the process of obtaining from is as follows: first, the size of is adjusted to obtain , or , and the adjustment means that if the size of exceeds 48x48x48, then is obtained through a max-pooling layer , otherwise, no operation is performed, and then is obtained through two parallel fully connected layers and then matrix multiplication to obtain initial channel correlation weights , , then is refined to get the final channel related weight , then is element-wise multiplied with the weight to get the refined feature , or , then is interpolated to get , , finally is element-wise added with to get ;

[0018] , , , represents element-wise multiplication, represents the identity matrix, represents the pair-wise dependency relationship between the channel in and the channel .

[0019] The glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network as described above, the working process of the encoder is: after the FLAIR image, the T2 image, the T1 image and the T1c image of the same glioma pass through the double convolution module, each ASG module is used to obtain feature map A1, feature map A2, feature map A3 and feature map A4;

[0020] Then, the feature map A1, the feature map A2, the feature map A3 and the feature map A4 pass through an ASG module respectively to obtain feature map B1, feature map B2, feature map B3 and feature map B4;

[0021] Then, the feature map B1, the feature map B2, the feature map B3 and the feature map B4 pass through an ASG module respectively to obtain feature map C1, feature map C2, feature map C3 and feature map C4;

[0022] Then, the feature map C1, the feature map C2, the feature map C3 and the feature map C4 pass through an ASG module respectively to obtain feature map D1, feature map D2, feature map D3 and feature map D4;

[0023] Then, the feature map D1, the feature map D2, the feature map D3 and the feature map D4 are directly spliced to obtain feature map E.

[0024] The working process of the skip connection part is as follows: the feature map A1, the feature map A2, the feature map A3 and the feature map A4 are spliced after each passing through an IMP module to obtain a feature map A;

[0025] Meanwhile, the feature map B1, the feature map B2, the feature map B3 and the feature map B4 are spliced after each passing through an IMP module to obtain a feature map B;

[0026] Meanwhile, the feature map C1, the feature map C2, the feature map C3 and the feature map C4 are spliced after each passing through an IMP module to obtain a feature map C;

[0027] Meanwhile, the feature map D1, the feature map D2, the feature map D3 and the feature map D4 are spliced after each passing through an IMP module to obtain a feature map D.

[0028] The decoder of the glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network comprises a CMF module (i.e., a cross membrane state fusion module), the input of the CMF module is the feature , , and the output is the feature , , and the process of obtaining the feature from the feature is as follows:

[0029] The channels corresponding to the FLAIR image, the T2 image, the T1 image and the T1c image in the feature are respectively denoted as , , and , , , , ;

[0030] The feature passes through a channel-wise max pooling layer to obtain , ;

[0031] The feature passes through a convolution layer with a convolution kernel of 1x1x1 to obtain , ;

[0032] The feature passes through a convolution layer with a convolution kernel of 3x3x3 to obtain , ; the feature passes through a convolution layer with a convolution kernel of 3x3x3 to obtain , ; and After splicing, sequentially pass through the maximum pooling layer and the convolution kernel 1x1x1 convolution layer to obtain , ; after splicing and , sequentially pass through the maximum pooling layer and the convolution kernel 1x1x1 convolution layer to obtain , ; after splicing and , sequentially pass through the convolution kernel 3x3x3 convolution layer and the Sigmod function operation to obtain , ;

[0033] According to the formula , , wherein represents element-wise multiplication, represents element-wise addition.

[0034] The working process of the decoder of the glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network is as follows: the feature map E is sequentially fused with the feature map D after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. The feature map f1 is obtained;

[0035] Then the feature map f1 is sequentially fused with the feature map C after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. The feature map f2 is obtained;

[0036] Then the feature map f2 is sequentially fused with the feature map B after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. The feature map f3 is obtained;

[0037] Then the feature map f3 is sequentially fused with the feature map A after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. The feature map f4 is obtained;

[0038] The feature map f4 is obtained after passing through the double convolution module.

[0039] The expression of the loss function of the image segmentation model of the glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network is as follows:

[0040] ;

[0041] ;

[0042] In the formula, represents the loss function, , , , a degree of coincidence of a predicted region representing an enhanced tumor region and a true region, a degree of coincidence of a predicted region representing a necrosis region and a true region, a degree of coincidence of a predicted region representing an edema region and a true region, a total number of voxels in the image, a pixel a predicted value whether it is a target class, =1 indicates yes, =0 indicates no, a pixel a true value, is 1e-6, is yes to avoid division by 0.

[0043] The glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network as described above, before the FLAIR image, the T2 image, the T1 image and the T1c image of the same glioma are jointly input into the trained image segmentation model, the preprocessing process is: first, the original image with a spatial resolution of 155*240*240 is uniformly adjusted to 192*192*192 by using the center cropping strategy; then, the Z-score normalization (mean=0, standard deviation=1) is used for standardization to enhance the generalization ability of the model; finally, Gaussian noise (standard deviation of 0.1) is introduced to further improve the robustness of the model.

[0044] The training steps of the image segmentation model in the glioma segmentation method based on the anatomical symmetry guided multi-modal fusion network as described above are as follows:

[0045] (a) collect glioma cases, ≥285, each glioma case has a FLAIR image, a T2 image, a T1 image and a T1c image;

[0046] (b) pre-process the FLAIR image, the T2 image, the T1 image and the T1c image; obtain the real segmentation image of each glioma case, the real segmentation image is the glioma MRI image with the three segmentation regions obtained by manual annotation;

[0047] (c) construct a training set and a test set using the FLAIR image, the T2 image, the T1 image, the T1c image and the real segmentation image corresponding to glioma cases;

[0048] (d) training the image segmentation model using the training set, during the training, taking the FLAIR image, the T2 image, the T1 image and the T1c image as the input of the image segmentation model, taking the real segmentation image as the theoretical output of the image segmentation model, constantly adjusting the weight parameters of the image segmentation model until the image segmentation model converges;

[0049] (e) testing the trained image segmentation model using the test set.

[0050] Principle of the application:

[0051] In view of the problem that the existing deep learning-based model is difficult to effectively model the anatomical symmetry destruction, the ASG module is introduced in the encoder, the module generates symmetric features through mirror flipping of the features, performs element-by-element subtraction after processing by the IMP module, captures the anatomical asymmetry information caused by glioma, and combines the context features extracted by the multi-scale hollow convolution to integrate the symmetry destruction information into the feature extraction process, thereby improving the accuracy of tumor region positioning.

[0052] In view of the problem that the existing deep learning-based model is difficult to realize effective cross-modal fusion, the CMF module is introduced in the decoder, the module classifies and processes the feature channels corresponding to the FLAIR image, the T2 image, the T1 image and the T1c image, enhances the attention mechanism of the student mode (T2, T1) fusion through the teacher mode (FLAIR, T1c), generates adaptive weights to realize dynamic integration of multi-modal features, and solves the problem of inefficient fusion caused by simple splicing.

[0053] In view of the problem that the existing deep learning-based model is difficult to realize fine segmentation in the glioma sub-region, the IMP module is introduced in the jump connection part, the module calculates the channel correlation weight and suppresses the autocorrelation interference, enhances the tumor-related features and suppresses irrelevant information, and simultaneously cooperates with the ASG module in the encoder and the CMF module in the decoder to improve the feature expression ability from three dimensions of anatomical prior, modal complementation and feature purification, and accurately captures the edema region, enhances the boundary and details of the tumor region and the necrotic region.

[0054] Advantages:

[0055] 1、The ASG module in the encoder of the present application generates symmetric features through mirror flipping, and performs element-by-element subtraction after processing by the IMP module, which can effectively capture the brain anatomical asymmetry information caused by glioma, and simultaneously combine the context features extracted by the multi-scale hollow convolution to integrate the symmetry destruction information into the feature extraction process, thereby solving the problem that the existing method is difficult to utilize anatomical prior knowledge, and significantly improving the positioning accuracy of the tumor region.

[0056] 2、The CMF module in the decoder of the application classifies and processes the feature channels corresponding to the FLAIR image, the T2 image, the T1 image and the T1c image, generates adaptive weights through the attention mechanism of the teacher mode (FLAIR, T1c) and the student mode (T2, T1), realizes the dynamic integration of multi-modal complementary information, overcomes the fusion inefficiency problem caused by simple splicing or weighting of the prior art, and fully excavates the synergistic effect of different modes.

[0057] 3、The IMP module of the jump connection part of the application enhances tumor-related features and suppresses irrelevant information by calculating channel correlation weights and suppressing autocorrelation interference; meanwhile, the module cooperates with the ASG module and the CMF module to improve feature expression ability from three dimensions of anatomical prior, modal complementarity and feature purification, effectively solves the problems of blurred boundaries of the edema region, enhanced tumor region and necrotic region of the glioma and limited segmentation accuracy, and meets the needs of clinical sub-regional fine segmentation.

[0058] 4、Compared with the traditional method, the machine learning method and the existing deep learning-based model, the three-module cooperative mechanism of the application realizes performance improvement in tumor positioning, cross-modal fusion and sub-regional segmentation, and provides more reliable image basis for the surgical plan of glioma, prognosis evaluation and personalized treatment decision. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 It is a model structure schematic diagram of the glioma segmentation method of the application;

[0060] Figure 2 It is a model structure schematic diagram of the ASG module of the application;

[0061] Figure 3 It is a model structure schematic diagram of the IMP module of the application;

[0062] Figure 4 It is a model structure schematic diagram of the CMF module of the application;

[0063] Figure 5 It is a model structure schematic diagram of the glioma segmentation method of Comparative Example 1;

[0064] Figure 6 It is a model structure schematic diagram of the glioma segmentation method of Comparative Example 2;

[0065] Figure 7 It is a model structure schematic diagram of the glioma segmentation method of Comparative Example 3;

[0066] Figure 8 It is a model structure schematic diagram of the glioma segmentation method of Comparative Example 4;

[0067] Figure 9A schematic diagram of a model structure of a glioma segmentation method of Comparative Example 5;

[0068] Figure 10 A schematic diagram of a model structure of a glioma segmentation method of Comparative Example 6;

[0069] Figure 11 A schematic diagram of a model structure of a glioma segmentation method of Comparative Example 7;

[0070] Figure 12 A visualization diagram of a module segmentation performance comparison of Example 1 and Comparative Examples 1 to 4 (in the segmentation results of Example 1 and Comparative Examples 1 to 4, the numerical value in the upper left corner is the DSC average value of the necrotic area, the enhanced tumor area and the edema area); the position marked by the purple circle in the diagram is the area where the predicted segmentation image of the different models and the true value image exist differences;

[0071] Figure 13 A visualization diagram of a module segmentation performance comparison of Example 1 and Comparative Examples 1 to 4 (in the segmentation results of Example 1 and Comparative Examples 1 to 4, the numerical value in the upper left corner is the DSC average value of the necrotic area, the enhanced tumor area and the edema area); the position marked by the purple circle in the diagram is the area where the predicted segmentation image of the different models and the true value image exist differences; Figure 1 An enlarged view of the input image and the output image shown in DETAILED DESCRIPTION

[0072] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not used to limit the scope of the application. In addition, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content taught by the application, and these equivalent forms also fall within the scope defined by the claims attached to the application.

[0073] Example 1

[0074] A glioma segmentation method based on an anatomical symmetry guided multi-modal fusion network, the steps are as follows:

[0075] Step 1, establishing an image segmentation model;

[0076] As shown in Figure 1 , the image segmentation model includes an encoder, a skip connection part and a decoder; Figure 1 Figure 13 As shown in , the encoder includes an ASG module, the input of the ASG module is a feature

[0077] , , wherein C represents the number of channels, D represents the depth, H represents the height, and W represents the width, and the output is a feature , As shown in Figure 2 , the feature is obtained from ​The steps are as follows:

[0078] (a) the feature is obtained by passing through a max-pooling layer, , the feature is obtained by passing through a convolutional layer with a convolution kernel of 3x3x3, , ;

[0079] (b) the feature is obtained by passing through four dilated convolutional layers with dilation rates of 1, 2, 3, and 4 respectively, followed by concatenation, and then passing through a convolutional layer with a convolution kernel of 1x1x1, , the feature is obtained by mirror flipping, , the features and are obtained by passing through an IMP module in sequence, and , , the feature is obtained by element-wise subtraction between and , ;

[0080] wherein the input of the IMP module is the feature , and the output is the feature , the process of obtaining from is as follows: as shown in Figure 3 , first, the size of is adjusted to obtain , or , then is passed through two parallel fully connected layers and then matrix multiplication is performed to obtain the initial channel correlation weight , the final channel correlation weight is obtained by refining , the feature is obtained by element-wise multiplication between and the weight , or , then is interpolated to obtain , , and finally is obtained by element-wise addition between and ;

[0081] , , , represents element-wise multiplication, represents an identity matrix, represents middle channel and channel between pairs of dependencies;

[0082] (c) after splicing and , a convolution layer with a convolution kernel of 1x1x1 is obtained ;

[0083] As shown in Figure 1 , the working process of the encoder is: after the FLAIR image, the T2 image, the T1 image and the T1c image of the same glioma pass through the double convolution module, each passes through an ASG module to obtain feature map A1, feature map A2, feature map A3 and feature map A4;

[0084] Then, the feature map A1, the feature map A2, the feature map A3 and the feature map A4 each pass through an ASG module to obtain feature map B1, feature map B2, feature map B3 and feature map B4;

[0085] Then, the feature map B1, the feature map B2, the feature map B3 and the feature map B4 each pass through an ASG module to obtain feature map C1, feature map C2, feature map C3 and feature map C4;

[0086] Then, the feature map C1, the feature map C2, the feature map C3 and the feature map C4 each pass through an ASG module to obtain feature map D1, feature map D2, feature map D3 and feature map D4;

[0087] Then, the feature map D1, the feature map D2, the feature map D3 and the feature map D4 are directly spliced to obtain feature map E;

[0088] The working process of the skip connection part is: after the feature map A1, the feature map A2, the feature map A3 and the feature map A4 each pass through an IMP module, they are spliced to obtain feature map A;

[0089] At the same time, after the feature map B1, the feature map B2, the feature map B3 and the feature map B4 each pass through an IMP module, they are spliced to obtain feature map B;

[0090] At the same time, after the feature map C1, the feature map C2, the feature map C3 and the feature map C4 each pass through an IMP module, they are spliced to obtain feature map C;

[0091] Simultaneously, feature maps D1, D2, D3, and D4 are each processed through an IMP module and then concatenated to obtain feature map D;

[0092] The decoder includes a CMF module, whose input is features. , The output is the feature. , ,like Figure 4 As shown, by get The process is as follows:

[0093] Will The channels corresponding to the FLAIR image, T2 image, T1 image, and T1c image are respectively denoted as follows: , , and , , , , ;

[0094] Will After obtaining the maximum pooling layer of the channel... , ;

[0095] Will After passing through a convolutional layer with a kernel of 1×1×1, we obtain , ;

[0096] Will After passing through a convolutional layer with a kernel size of 3×3×3, the following is obtained: , ;Will After passing through a convolutional layer with a kernel size of 3×3×3, the following is obtained: , ;Will and After concatenation, the layers are sequentially passed through a max pooling layer and a convolutional layer with a 1×1×1 kernel to obtain the desired result. , ;Will and After concatenation, the layers are sequentially passed through a max pooling layer and a convolutional layer with a 1×1×1 kernel to obtain the desired result. , ;Will and After concatenation, the result is obtained by sequentially passing the convolutional layer with a 3×3×3 kernel and performing a Sigmoid function operation. , ;

[0097] According to the formula Obtained , wherein, represents element-by-element multiplication, represents element-by-element addition;

[0098] As shown in Figure 1 , the working process of the decoder is: the feature map E is sequentially fused with the feature map D after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. Feature map f1;

[0099] Then the feature map f1 is sequentially fused with the feature map C after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. Feature map f2;

[0100] Then the feature map f2 is sequentially fused with the feature map B after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. Feature map f3;

[0101] Then the feature map f3 is sequentially fused with the feature map A after passing through the CMF module and the transposed convolution module, and the fusion result is obtained after passing through the double convolution module. Feature map f4;

[0102] The feature map f4 is obtained after passing through the double convolution module. Predicted segmentation image;

[0103] Step two, determine the loss function of the image segmentation model;

[0104] The expression of the loss function of the image segmentation model is as follows:

[0105] ;

[0106] ;

[0107] In the formula, represents the loss function, , , , represents the degree of coincidence between the predicted area and the real area of the enhanced tumor area, represents the degree of coincidence between the predicted area and the real area of the necrotic area, represents the degree of coincidence between the predicted area and the real area of the edema area, represents the total number of voxels in the image, represents the predicted value of whether the pixel is a target category, =1 indicates yes, =0 indicates no, represents the true value of the pixel , 1e-6;

[0108] Step three, training the image segmentation model;

[0109] (a) Collecting 285 glioma cases (the data source is the dataset BraTS 2018 of brain tumor segmentation challenge, which is widely used in brain tumor segmentation task), each of which has FLAIR image, T2 image, T1 image and T1c image;

[0110] (b) Preprocessing the FLAIR image, T2 image, T1 image and T1c image (firstly, using the center cropping strategy, the original image with a spatial resolution of 155x240x240 is uniformly adjusted to 192x192x192; then, using the Z-score normalization method (the mean is set to 0 and the standard deviation is set to 1) to standardize the image; finally, introducing Gaussian noise with a standard deviation of 0.1 to the image); obtaining the real segmentation image of each glioma case, which is the glioma MRI image with the three segmentation regions (edema region, enhanced tumor region and necrosis region) obtained by manual annotation;

[0111] (c) Constructing the training set and test set (the number ratio of brain tumor cases in the training set and test set is 8:2) using the FLAIR image, T2 image, T1 image, T1c image and real segmentation image corresponding to the 285 glioma cases;

[0112] (d) Training the image segmentation model using the training set, during which the FLAIR image, T2 image, T1 image and T1c image are used as the input of the image segmentation model, the real segmentation image is used as the theoretical output of the image segmentation model, and the weight parameters of the image segmentation model are constantly adjusted until the image segmentation model converges;

[0113] (e) Testing the trained image segmentation model using the test set; using the dice similarity coefficient (DSC) and 95% Hausdorff distance (HD) to represent the segmentation accuracy of the trained image segmentation model, the calculation formula is as follows:

[0114] ;

[0115] In the formula, A and B represent the predicted region (i.e. the whole tumor region, the tumor core region or the enhanced tumor region; the whole tumor region = edema region + enhanced tumor region + necrosis region, the tumor core region = enhanced tumor region + necrosis region) and the real region, represents the intersection size of the predicted region and the real region, and respectively represent the total size of the predicted region and the real region;

[0116] ;

[0117] wherein X and Y represent the set of boundary points of the predicted region and the set of boundary points of the real region respectively, d(x, y) represents the Euclidean distance between point x and point y, represents the maximum value of the minimum distance from each point x of the predicted region boundary to the real region boundary, represents the maximum value of the minimum distance from each point y of the real region boundary to the predicted region boundary;

[0118] Step four, output the predicted segmentation image;

[0119] The FLAIR image, the T2 image, the T1 image and the T1c image of the same glioma are collectively input into the trained image segmentation model, and a predicted segmentation image is output therefrom, the predicted segmentation image being a glioma MRI image with 3 segmented regions obtained by prediction, the 3 segmented regions being an edema region, an enhanced tumor region and a necrosis region.

[0120] Comparative Example 1

[0121] A glioma segmentation method, which is basically the same as that of Example 1, except that the image segmentation model is as shown in Figure 5 compared with the image segmentation model of Example 1, the encoder does not contain the ASG module, the skip connection part does not contain the IMP module, and the decoder does not contain the CMF module.

[0122] Comparative Example 2

[0123] A glioma segmentation method, which is basically the same as that of Example 1, except that the image segmentation model is as shown in Figure 6 compared with the image segmentation model of Example 1, the skip connection part does not contain the IMP module, and the decoder does not contain the CMF module.

[0124] Comparative Example 3

[0125] A glioma segmentation method, which is basically the same as that of Example 1, except that the image segmentation model is as shown in Figure 7 compared with the image segmentation model of Example 1, the encoder does not contain the ASG module, and the decoder does not contain the CMF module.

[0126] Comparative Example 4

[0127] A glioma segmentation method, which is basically the same as that of Example 1, except that the image segmentation model is as shown in Figure 8 compared with the image segmentation model of Example 1, the encoder does not contain the ASG module, and the skip connection part does not contain the IMP module.

[0128] Comparative Example 5

[0129] A glioma segmentation method, substantially the same as embodiment 1, except that the image segmentation model is as shown in Figure 9 which, relative to the image segmentation model of embodiment 1, the decoder does not contain the CMF module.

[0130] Comparative example 6

[0131] A glioma segmentation method, substantially the same as embodiment 1, except that the image segmentation model is as shown in Figure 10 which, relative to the image segmentation model of embodiment 1, the skip connection part does not contain the IMP module.

[0132] Comparative example 7

[0133] A glioma segmentation method, substantially the same as embodiment 1, except that the image segmentation model is as shown in Figure 11 which, relative to the image segmentation model of embodiment 1, the encoder does not contain the ASG module.

[0134] The test results of embodiment 1 and comparative examples 1-7 are shown in Table 1 (bold represents the best result; - represents that there is no such module in the model; √ represents that there is such module in the model; * represents that statistical significant improvement compared with the baseline method is achieved using Wilcoxon signed rank test (p<0.05)):

[0135] Table 1

[0136]

[0137] As shown in Table 1, Comparative Example 1 (without ASG, IMP, or CMF modules) as the baseline model has the lowest average DSC and the highest average HD in the entire tumor region, tumor core region, and enhanced tumor region. Comparative Example 2 (using only the ASG module) and Comparative Example 3 (using only the IMP module) show significant improvements in DSC in the tumor core region and enhanced tumor region; Comparative Example 4 (using only the CMF module) shows significant improvements in DSC in the entire tumor region and tumor core region. This indicates that the ASG module in the encoder can effectively guide the network to focus on asymmetric regions of the image; the IMP module in the skip connection part can optimize the feature maps in the modality; and the CMF module in the decoder enhances the importance of the teacher modality through modality fusion, and its internal compression-expansion mechanism can more effectively express important information, thereby improving segmentation accuracy. The overall performance of Comparative Example 5 (using both ASG and IMP modules), Comparative Example 6 (using both ASG and CMF modules), and Comparative Example 7 (using both IMP and CMF modules) is better than that of using only a single module. Finally, Example 1, which uses the ASG module, IMP module, and CMF module simultaneously, achieved the highest DSC values ​​in the entire tumor region, the tumor core region, and the enhanced tumor region, with an average DSC value of 84.4%. It also performed well in HD in these three regions, with the average HD value reduced to 4.27 mm. Compared with other models, it achieved a significant improvement, which fully demonstrates that the synergistic effect of the three modules can maximize the glioma segmentation effect.

[0138] To further verify the above conclusions, we conducted a visualization analysis of the predicted segmentation images of two brain tumor cases based on FLAIR images from the BraTS2018 dataset (e.g., Figure 12 As shown). In the first case ( Figure 12 In the first case (a), the baseline model (Comparative Example 1) had the problem of missing necrotic areas. However, Comparative Example 2, which introduced the ASG module, Comparative Example 3, and Comparative Example 4, which introduced the IMP module, all improved this phenomenon. Among them, Example 1, which integrated all three modules, showed the highest consistency with the ground truth map in the segmentation boundaries of the necrotic area (red), the enhanced tumor area (yellow), and the edema area (green). In the second case (a) Figure 12 In the example (b), the baseline model (Comparative Example 1) over-segmented the necrotic region. This problem was alleviated after the introduction of the module in Comparative Examples 2-4. The predicted segmentation image in Example 1 closely resembled the ground truth image, with the fewest differences (marked by purple circles). This intuitive result is consistent with the trend of the quantitative indicators in Table 1, further confirming the effectiveness of the ASG module, IMP module, CMF module, and their synergistic effect in improving the segmentation accuracy of gliomas.

Claims

1. A glioma segmentation method based on an anatomically symmetry-guided multimodal fusion network, characterized in that, The FLAIR, T2, T1 and T1c images of the same glioma are input into the trained image segmentation model, which outputs a predicted segmented image. The predicted segmented image is the glioma MRI image with three segmentation regions, namely the edema region, the enhanced tumor region and the necrosis region. Image segmentation models include an encoder, a skip connection part, and a decoder; The encoder includes an ASG module, whose input is a feature. , Where C represents the number of channels, D represents the depth, H represents the height, and W represents the width, the output is the feature. , ,Depend on get The steps are as follows: (a) will After the maximum pooling layer is obtained , Then After passing through a convolutional layer with a kernel size of 3×3×3, the following is obtained: , ; (b) will After passing through four dilation rates of 1, 2, 3, and 4 respectively, the layers are stitched together, and then passed through a convolutional layer with a kernel of 1×1×1 to obtain the desired result. , Then Mirror flipping to obtain , Then and They are obtained sequentially through the IMP module. and , , And then and After element-wise subtraction, the result is obtained through a max pooling layer. , ; The input of the IMP module is features , The output is the feature. , ,Depend on get The process is as follows: First adjust The dimensions are obtained , or Then The initial channel correlation weights are obtained by matrix multiplication after passing through two parallel fully connected layers. , And then The final channel-related weights are obtained by refining the process. Then With weight Element-wise multiplication yields the features , or Then on Interpolation is performed to obtain , Finally and Element-by-element addition yields ; , , , This represents element-wise multiplication. Represents the identity matrix. represent Central Channel and channels Pair dependency relationships between them; (c) will and After splicing, the result is obtained through a convolutional layer with a kernel of 1×1×1. .

2. The glioma segmentation method based on an anatomically symmetry-guided multimodal fusion network according to claim 1, characterized in that, The encoder works as follows: FLAIR, T2, T1 and T1c images of the same glioma are processed through a double convolution module, and then each is processed through an ASG module to obtain feature maps A1, A2, A3 and A4. Then, feature maps A1, A2, A3, and A4 are each processed through an ASG module to obtain feature maps B1, B2, B3, and B4. Then, feature map B1, feature map B2, feature map B3, and feature map B4 are each processed through an ASG module to obtain feature map C1, feature map C2, feature map C3, and feature map C4. Then, feature maps C1, C2, C3, and C4 are each processed through an ASG module to obtain feature maps D1, D2, D3, and D4. Then, feature maps D1, D2, D3, and D4 are directly spliced ​​together to obtain feature map E.

3. The glioma segmentation method based on an anatomically symmetry-guided multimodal fusion network according to claim 2, characterized in that, The skip connection process is as follows: feature map A1, feature map A2, feature map A3, and feature map A4 are each passed through an IMP module and then concatenated to obtain feature map A; Simultaneously, feature map B1, feature map B2, feature map B3, and feature map B4 are each processed through an IMP module and then concatenated to obtain feature map B; Simultaneously, feature maps C1, C2, C3, and C4 are each processed through an IMP module and then concatenated to obtain feature map C; Simultaneously, feature maps D1, D2, D3, and D4 are each processed through an IMP module and then concatenated to obtain feature map D.

4. The glioma segmentation method based on an anatomically symmetry-guided multimodal fusion network according to claim 3, characterized in that, The decoder includes a CMF module, whose input is features. , The output is the feature. , ,Depend on get The process is as follows: Will The channels corresponding to the FLAIR image, T2 image, T1 image, and T1c image are respectively denoted as follows: , , and , , , , ; Will After obtaining the maximum pooling layer of the channel... , ; Will After passing through a convolutional layer with a kernel of 1×1×1, we obtain , ; Will After passing through a convolutional layer with a kernel size of 3×3×3, the following is obtained: , ;Will After passing through a convolutional layer with a kernel size of 3×3×3, the following is obtained: , ;Will and After concatenation, the layers are sequentially passed through a max pooling layer and a convolutional layer with a 1×1×1 kernel to obtain the desired result. , ;Will and After concatenation, the layers are sequentially passed through a max pooling layer and a convolutional layer with a 1×1×1 kernel to obtain the desired result. , ;Will and After concatenation, the result is obtained by sequentially passing the convolutional layer with a 3×3×3 kernel and performing a Sigmoid function operation. , ; According to the formula get In the formula, This represents element-wise multiplication. This represents element-wise addition.

5. The glioma segmentation method based on an anatomically symmetry-guided multimodal fusion network according to claim 4, characterized in that, The decoder works as follows: feature map E is passed through the CMF module and the transposed convolution module in sequence and then fused with feature map D. The fused result is then passed through the double convolution module to obtain feature map f1. The feature map f1 is then passed through the CMF module and the transposed convolution module in sequence and fused with the feature map C. The fused result is then passed through the double convolution module to obtain the feature map f2. The feature map f2 is then passed through the CMF module and the transposed convolution module in sequence and fused with the feature map B. The fused result is then passed through the double convolution module to obtain the feature map f3. The feature map f3 is then passed through the CMF module and the transposed convolution module in sequence and fused with the feature map A. The fused result is then passed through the double convolution module to obtain the feature map f4. Feature map f4 is processed by a double convolution module to obtain the predicted segmentation image.

6. The glioma segmentation method based on an anatomically symmetry-guided multimodal fusion network according to claim 5, characterized in that, The expression for the loss function of the image segmentation model is as follows: ； ； In the formula, Represents the loss function. , , , This represents the degree of overlap between the predicted and actual tumor regions. The degree of overlap between the predicted and actual areas representing necrotic regions. This represents the degree of overlap between the predicted and actual edema areas. This represents the total number of voxels in the image. Represents pixels Is it a predicted value for the target category? =1 means yes, =0 indicates no. Represents pixels The truth value of , It is 1e-6.

7. The glioma segmentation method based on an anatomically symmetry-guided multimodal fusion network according to claim 1, characterized in that, Before the FLAIR, T2, T1, and T1c images of the same glioma are input into the trained image segmentation model, they are preprocessed. The preprocessing process is as follows: First, the original image with a spatial resolution of 155×240×240 is uniformly adjusted to 192×192×192 using a center cropping strategy; then, it is normalized using Z-score normalization; finally, Gaussian noise is introduced.

8. The glioma segmentation method based on an anatomically symmetry-guided multimodal fusion network according to claim 7, characterized in that, The training steps for the image segmentation model are as follows: (a) Collection One case of glioma, ≥285, each glioma case had FLAIR images, T2 images, T1 images and T1c images; (b) Perform the aforementioned preprocessing on FLAIR, T2, T1, and T1c images; obtain the true segmentation image for each glioma case, which is the glioma MRI image with the three segmentation regions obtained through manual annotation; (c) Adopt The training and test sets were constructed using FLAIR images, T2 images, T1 images, T1c images, and ground truth segmentation images corresponding to each glioma case. (d) The image segmentation model is trained using the training set. During training, the FLAIR image, T2 image, T1 image, and T1c image are used as the input of the image segmentation model, and the real segmented image is used as the theoretical output of the image segmentation model. The weight parameters of the image segmentation model are continuously adjusted until the image segmentation model converges. (e) Test the trained image segmentation model using a test set.

Images