Brain tumor segmentation method based on uncertainty estimation

Abstract

The invention belongs to the technical field of medical image analysis, and relates to a brain tumor segmentation method based on uncertainty estimation, and the method comprises the steps: inputting T1, T1c, T2 and Flair images of a brain tumor into a trained image segmentation model which comprises four encoder modules, one decoder module, one Monte Carlo simulation module and one image processing module; the Monte Carlo module repeatedly calls the encoder module and the decoder module for sampling; and the image processing module processes the output to obtain a mean value image, a random uncertainty image, a cognitive uncertainty image, an entropy uncertainty image and a mutual information uncertainty image, thereby realizing accurate segmentation and analysis of the brain tumor. The method improves the segmentation accuracy and the diagnosis reliability, is high in expansibility, and is suitable for various medical image analysis tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image analysis, and in particular relates to a brain tumor segmentation method based on uncertainty estimation. Background Art

[0002] In the vast field of medical diagnosis and treatment planning, brain tumor segmentation plays a vital role. It is not only the basis for doctors to assess the size, location and morphology of tumors, but also the key basis for formulating personalized treatment plans, predicting patient prognosis and monitoring treatment effects. Accurate brain tumor segmentation can significantly improve the accuracy of surgery, reduce unnecessary tissue damage, and optimize the formulation of radiotherapy plans to ensure that the radiation dose accurately covers the tumor area while maximizing the protection of surrounding normal tissues. Therefore, improving the accuracy and reliability of brain tumor segmentation is of immeasurable value in improving the quality of life of patients and prolonging their survival.

[0003] Traditional single-modality segmentation methods, such as segmentation techniques based on MRI modalities such as T1, T2, and FLAIR, have shown certain application value in specific situations, but their inherent limitations cannot be ignored. The diversity and complexity of brain tumors require segmentation methods to be able to capture subtle characteristic changes of tumors in different modalities. However, single-modality segmentation methods only rely on information from a single modality and are difficult to fully reflect the overall picture of the tumor, especially when the tumor boundaries are blurred, the morphology is irregular, or the modality information is missing. The segmentation results are often unsatisfactory. This may not only lead to the omission or misjudgment of the tumor area, but may also affect the accuracy and effectiveness of subsequent treatment plans, increasing the risk of clinical decision-making.

[0004] In order to overcome the limitations of single-modality segmentation methods, researchers have begun to explore multimodal data fusion methods. By fusing images of multiple MRI modalities such as T1, T2, and FLAIR, multimodal segmentation methods can comprehensively utilize the information provided by different modalities to more comprehensively capture the characteristics of the tumor and improve the accuracy and stability of segmentation. This fusion strategy can not only enhance the ability to identify tumor boundaries, but also reveal subtle structural changes inside the tumor, providing doctors with richer diagnostic information.

[0005] Although multimodal segmentation methods have made significant progress in improving segmentation accuracy, existing methods generally lack uncertainty estimation mechanisms. Uncertainty estimation is a method for evaluating the reliability of model prediction results. It can provide additional information about the confidence of the prediction results, which helps doctors understand the credibility of the segmentation results more comprehensively. Existing uncertainty estimation techniques are mostly combined with single-modal segmentation methods, lack effective processing and fusion of multimodal features, and are difficult to accurately reflect the uncertainty of multimodal data. In addition, some uncertainty estimation methods have high computational complexity and are difficult to implement in real time in actual clinical environments.

[0006] Therefore, it is of great significance to develop a brain tumor segmentation method based on uncertainty estimation, which not only helps to improve the accuracy and stability of the segmentation results, but also provides doctors with more reliable diagnostic basis and further promotes progress in the field of brain tumor treatment. Summary of the invention

[0007] The purpose of the present invention is to solve the above problems and provide a brain tumor segmentation method based on uncertainty estimation.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] A brain tumor segmentation method based on uncertainty estimation, which inputs T1 image, T1c image, T2 image and Flair image of the same brain tumor into a trained image segmentation model, and outputs a mean image, a random uncertainty image, a cognitive uncertainty image, an entropy uncertainty image and a mutual information uncertainty image;

[0010] The image segmentation model includes 4 encoder modules, 1 decoder module, 1 Monte Carlo simulation module, and 1 image processing module;

[0011] Both the encoder and decoder modules contain dropout layers;

[0012] The four encoder modules downsample the T1 image, T1c image, T2 image, and Flair image respectively, and the one decoder module upsamples the downsampled T1 image, T1c image, T2 image, and Flair image to obtain image X;

[0013] The Monte Carlo simulation module repeatedly calls four encoder modules and one decoder module to perform Monte Carlo sampling image X, =10-50;

[0014] Image processing module The image X is processed to obtain the mean image, random uncertainty image, cognitive uncertainty image, entropy uncertainty image and mutual information uncertainty image;

[0015] Mean Image The calculation formula is as follows:

[0016] ;

[0017] In the formula, It is Monte Carlo sampling image X;

[0018] Random uncertainty image The calculation formula is as follows:

[0019] ;

[0020] In the formula, Represents the model parameters The expectation of the posterior distribution of (given data D), Indicates that given the model parameters Conditional prediction The variance of

[0021] Epistemic uncertainty image The calculation formula is as follows:

[0022] ;

[0023] In the formula, Represents the model parameters The variance of the posterior distribution of (given data D), Indicates that given the model parameters Prediction under conditions Expected value;

[0024] Entropy uncertainty image The calculation formula is as follows:

[0025] ;

[0026] In the formula, is the probability mass function of the random variable X at x;

[0027] Mutual Information Uncertainty Image The calculation formula is as follows:

[0028] ;

[0029] Where P(x) and P(y) are the marginal probability mass functions of the random variables X and Y, respectively, and P(x,y) is the joint probability mass function of X and Y evaluated at x and y;

[0030] The mean image is the predicted segmentation image, which is the brain tumor MRI image with three segmentation regions obtained through prediction. The three segmentation regions are the complete tumor region, the tumor core region, and the enhancement tumor region.

[0031] The confidence of the segmentation results in the predicted segmentation image can be seen from the random uncertainty image, cognitive uncertainty image, entropy uncertainty image and mutual information uncertainty image.

[0032] The present invention innovatively combines the encoder module and the decoder module with the uncertainty estimation module (composed of a Monte Carlo simulation module and an image processing module), and introduces uncertainty information in the encoding and decoding process to achieve the synchronization of multi-scale feature extraction and uncertainty assessment. Such a combination can achieve unexpected results, not only dynamically adjusting the model's attention to high uncertainty areas, but also improving the segmentation accuracy of boundaries and complex areas.

[0033] Monte Carlo sampling is applied at the bottom of the network to generate multiple segmentation results, and the mean and variance of these segmentation results are calculated. These segmentation results are then used to generate different types of uncertainty maps, including Aleatoric, Epistemic, Entropy, and Mutual Information, which can provide insights into the credibility of the segmentation results, making the model more effective when highly reliable predictions are required.

[0034] As the preferred technical solution:

[0035] In the above-mentioned brain tumor segmentation method based on uncertainty estimation, the discard rate of each discard layer is 0.3.

[0036] In the above-mentioned brain tumor segmentation method based on uncertainty estimation, the decoder module includes a multimodal fusion module a and a multimodal fusion module b, and the workflows of the two modules are as follows:

[0037] (a) Concatenate T2 and T1 images to form the student mode , connect the Flair image and T1c image to form the teacher modality ;

[0038] (b) Apply a convolutional block operation of size 3×3×3, Operation to obtain student attention weight ;

[0039] Sequentially Apply a convolutional block operation of size 5×5×5, Operation to obtain student attention weight ;

[0040] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the student weighted feature representation ;

[0041] Sequentially Apply a convolutional block operation of size 3×3×3, Operation to get the teacher's attention weight ;

[0042] Sequentially Apply a convolutional block operation of size 5×5×5, Operation to get the teacher's attention weight ;

[0043] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the teacher weighted feature representation ;

[0044] Will and Concatenate to form spatial feature representation ;

[0045] Respectively and After applying the average pooling operation and concatenating the results, the multi-layer perceptron operation and Operation to get attention weight ;

[0046] Respectively and After applying the max pooling operation and concatenating the results, the multi-layer perceptron operation and Operation to get attention weight ;

[0047] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the student weighted feature representation ;

[0048] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the teacher weighted feature representation ;

[0049] Will and Concatenate to form channel attention features ;

[0050] (c) and Concatenate to form fused feature representation .

[0051] In order to learn cross-modal complementary information, this method proposes a multimodal teacher-student fusion (MTSF) strategy, which includes a spatial attention fusion module (SAFM) and a channel attention fusion module (CAFM), such as Figure 2 As shown in Figure 2. The former is used to learn the fused feature representation from a spatial perspective, and the latter is used to extract the feature representation from a channel perspective. Initially, Flair and T1c are concatenated to form the teacher modality. Similarly, T2 and T1 are concatenated to obtain the student modality.

[0052] In the spatial attention fusion module, the student modality is and teacher mode Apply convolutional blocks of size 3×3×3 and 5×5×5, each followed by a This process can obtain the student attention weight , , and the teacher attention weight , Then, these weights are combined with the original student model and teacher mode By performing element-wise multiplication and addition, we can obtain the weighted feature representation. and Subsequently, the student spatial feature representation and the teacher spatial feature representation are concatenated to produce the final spatial feature representation In this way, the spatial attention fusion module ensures that the fused feature representation captures the most informative spatial details in each modality.

[0053] In the channel attention module, the student mode and teacher mode The average pooling (GAP, GlobalAverage Pooling) and maximum pooling (GMP, Global Max Pooling) operations are applied respectively. The results of each pooling operation are concatenated and then multi-layer perceptron and Function to get attention weight and Next, these weights are compared with the original student model and teacher mode Multiply and add to get the weighted feature representation of students and teachers and Subsequently, these feature representations are concatenated to achieve the final channel-attention feature Therefore, the channel attention fusion module promotes the extraction of complementary information by emphasizing important channels and suppressing irrelevant channels.

[0054] Finally, the spatial feature representation and channel attention features are connected to achieve fusion feature representation By combining these two attention modules, the proposed multimodal teacher-student fusion strategy is able to effectively learn comprehensive and complementary feature representations. This dual attention approach exploits the advantages of both spatial and channel information and improves the overall performance of the multimodal fusion process.

[0055] In the above-mentioned brain tumor segmentation method based on uncertainty estimation, the structures of the four encoder modules are the same, and all include a down-sampling initial block, two down-sampling convolution normalized activation modules b, and two convolution normalized activation discarding blocks connected in sequence;

[0056] The downsampling initial block consists of a standard convolutional normalized activation module and a downsampling convolutional normalized activation module a connected in sequence;

[0057] Both convolution-normalized activation dropout blocks consist of a downsampled convolution-normalized activation module a and a dropout layer connected in sequence;

[0058] The output of the last convolution normalization activation drop block is the output of the encoder module;

[0059] The decoder module also includes a first up-sampling fusion convolution discard block, a second up-sampling fusion convolution discard block, a first up-sampling fusion convolution block, a second up-sampling fusion convolution block, a third up-sampling fusion convolution block, a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, a first up-sampling module b, a second up-sampling module b, a third up-sampling module b, a fourth up-sampling module b, a first element-by-element addition module, a second element-by-element addition module, a third element-by-element addition module, and a fourth element-by-element addition module;

[0060] The multimodal fusion module b, the first upsampling fusion convolution discard block, the second upsampling fusion convolution discard block, the first upsampling fusion convolution block, the second upsampling fusion convolution block, the third upsampling fusion convolution block, the fifth convolution block, and the fourth element-by-element addition module are connected in sequence;

[0061] The first upsampling fusion convolution discarding block, the first convolution block, the first upsampling module b, the first element-by-element addition module, the second upsampling module b, the second element-by-element addition module, the third upsampling module b, the third element-by-element addition module, the fourth upsampling module b, and the fourth element-by-element addition module are connected in sequence;

[0062] The second upsampling fusion convolution discard block, the second convolution block, and the first element-by-element addition module are connected in sequence;

[0063] The first upsampling fusion convolution block, the third convolution block, and the second element-by-element addition module are connected in sequence;

[0064] The second upsampling fusion convolution block, the fourth convolution block, and the third element-by-element addition module are connected in sequence;

[0065] The first upsampling fusion convolution discard block and the second upsampling fusion convolution discard block are both composed of an upsampling module a, a multimodal fusion module a, a standard convolution normalization activation module, and a discard layer connected in sequence;

[0066] The first upsampling fusion convolution block, the second upsampling fusion convolution block and the third upsampling fusion convolution block are all composed of an upsampling module a, a multimodal fusion module a, and a standard convolution normalization activation module connected in sequence;

[0067] The output of the fourth element-by-element addition module is the output of the decoder;

[0068] The decoder part consists of multiple layers of upsampling and convolution modules, which are designed to gradually restore the feature maps compressed by the encoder; the feature maps are processed by upsampling (green module), convolution (blue module) and multimodal fusion modules (purple module) to gradually increase the resolution of the feature maps and finally reconstruct them to the original size; the decoder path also integrates a deep supervision mechanism (yellow and green modules) to facilitate the generation of multi-scale segmentation results and enhance the learning ability of the model during training;

[0069] The standard convolutional normalization activation module consists of a convolutional layer, an instance normalization layer, and a LeakyReLU activation function layer connected in sequence;

[0070] The downsampling convolution normalization activation module a and the downsampling convolution normalization activation module b are composed of a convolution layer with a step size of 2, an instance normalization layer, and a LeakyReLU activation function layer connected in sequence;

[0071] All sequential connections refer to sequential connections from front to back according to the data flow direction;

[0072] In the present invention, "a" and "b" are used to distinguish whether a module is an independent module or constitutes a module together with other modules. For example, the downsampling convolution normalization activation module a refers to the downsampling convolution normalization activation module that constitutes a module together with other modules, while the downsampling convolution normalization activation module b refers to the downsampling convolution normalization activation module that is an independent module.

[0073] As described above, a brain tumor segmentation method based on uncertainty estimation, the loss function of the image segmentation model is The calculation formula is as follows:

[0074] ;

[0075] ;

[0076] ;

[0077] ;

[0078] ;

[0079] In the formula, , is a hyperparameter, = =1, represents the number of voxels in the image, Representing voxels The predicted value of Representing voxels The true value of is a constant used to prevent division by zero. For 10 -5 , Indicates Monte Carlo sampling of voxels The predicted values ​​of , other unexplained symbols can be eliminated during the calculation process, they should not affect the accuracy and validity of the results;

[0080] This is a hybrid loss function that combines the traditional Dice loss (L Dice ) and the uncertainty-based penalty term (L Uncertainty ) and further enhance the performance of the network in segmentation tasks by adding uncertainty information into the loss function.

[0081] In a brain tumor segmentation method based on uncertainty estimation as described in any of the above items, the T1 image, T1c image, T2 image and Flair image of the same brain tumor are preprocessed before being input into the trained image segmentation model. The preprocessing is to perform grayscale normalization, pixel adjustment, and tensor quantization in sequence.

[0082] In the above-mentioned brain tumor segmentation method based on uncertainty estimation, the training steps of the image segmentation model are as follows:

[0083] (a) Collection Brain tumor cases, ≥285, each brain tumor case had T1 images, T1c images, T2 images, and Flair images;

[0084] (b) performing the above-mentioned preprocessing on the T1 image, T1c image, T2 image and Flair image; obtaining a true segmentation image of each brain tumor case, where the true segmentation image is a brain tumor MRI image with the above-mentioned three segmentation regions obtained by manual annotation;

[0085] (c) Adoption The training and test sets were constructed using T1 images, T1c images, T2 images, Flair images, and true segmentation images corresponding to each brain tumor case.

[0086] (d) Use the training set to train the image segmentation model. During training, use the T1 image, T1c image, T2 image, and Flair image as the input of the image segmentation model, use the real segmentation image as the theoretical output of the image segmentation model, and continuously adjust the weight parameters of the image segmentation model (the weight and bias parameters of the convolutional layer, and the instance normalization layer parameters) until the image segmentation model converges (that is, the loss function value gradually decreases and tends to a stable value);

[0087] (e) Use the test set to test the trained image segmentation model.

[0088] Beneficial effects:

[0089] A brain tumor segmentation method based on uncertainty estimation of the present invention improves the accuracy of segmentation by introducing uncertainty estimation: existing brain tumor segmentation methods usually ignore the uncertainty of model output, which leads to easy errors when processing complex or fuzzy boundary areas; in contrast, the present invention introduces uncertainty estimation, which can not only accurately capture the uncertain areas of the model, but also improve the overall segmentation accuracy, especially in the segmentation of tumor boundaries and irregular shapes.

[0090] The present invention provides a brain tumor segmentation method based on uncertainty estimation, which improves the reliability of diagnosis by quantifying uncertainty: traditional segmentation methods only output result images and lack quantitative evaluation of segmentation accuracy; by introducing uncertainty estimation, the present invention provides an uncertainty measurement for each pixel, which can help doctors better understand the credibility level of the model in different areas, and then more accurately judge the location and size of the tumor, providing reliable auxiliary information for clinical diagnosis.

[0091] The brain tumor segmentation method based on uncertainty estimation of the present invention has strong scalability and is suitable for a variety of medical image analysis tasks: the uncertainty estimation method adopted by the present invention is not only limited to brain tumor segmentation, but also has strong versatility and can be extended to other medical image analysis tasks, such as automatic segmentation and diagnosis of organs such as the heart, liver, and lungs, and has broad clinical application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 and Figure 2 A schematic diagram of the overall network architecture of a brain tumor segmentation method based on uncertainty estimation in Example 1;

[0093] Figure 3 for Figure 2 Schematic diagram of the network architecture of the purple module in the figure;

[0094] Figure 4 A schematic diagram of the overall network architecture of a brain tumor segmentation method in Comparative Example 1;

[0095] Figure 5 A schematic diagram of the overall network architecture of a brain tumor segmentation method in Comparative Example 2;

[0096] Figure 6 A schematic diagram of the overall network architecture of a brain tumor segmentation method in Comparative Example 3;

[0097] Figure 7 A schematic diagram of the overall network architecture of a brain tumor segmentation method of Comparative Example 4;

[0098] Figure 8 The random uncertainty image outputted by Example 1;

[0099] Fig. 9 The epistemic uncertainty image outputted by Example 1;

[0100] Fig.10 is the entropy uncertainty image outputted by Example 1;

[0101] Fig.11 This is the mutual information uncertainty image output by Example 1. DETAILED DESCRIPTION

[0102] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0103] Example 1

[0104] A brain tumor segmentation method based on uncertainty estimation, such as Figures 1 to 3 As shown, the specific steps are as follows:

[0105] Step 1: Build an image segmentation model;

[0106] The image segmentation model includes 4 encoder modules, 1 decoder module, 1 Monte Carlo simulation module and 1 image processing module;

[0107] The four encoder modules downsample the T1 image, T1c image, T2 image, and Flair image respectively;

[0108] A decoder module simultaneously upsamples the downsampled T1 image, T1c image, T2 image, and Flair image to obtain image X;

[0109] The Monte Carlo simulation module repeatedly calls four encoder modules and one decoder module to perform Monte Carlo sampling image X, =10-50;

[0110] Image processing module The image X is processed to obtain a mean image, a random uncertainty image, a cognitive uncertainty image, an entropy uncertainty image, and a mutual information uncertainty image; the mean image is a predicted segmentation image, and the predicted segmentation image is a brain tumor MRI image with three segmentation regions obtained by prediction, and the three segmentation regions are a complete tumor region, a tumor core region, and an enhancement tumor region;

[0111] The structures of the four encoder modules are the same, which include a down-sampled initial block, two down-sampled convolutional normalized activation modules b, and two convolutional normalized activation discard blocks connected in sequence;

[0112] The downsampling initial block consists of a standard convolution normalization activation module and a downsampling convolution normalization activation module a connected in sequence; the standard convolution normalization activation module consists of a convolution layer, an instance normalization layer, and a LeakyReLU activation function layer connected in sequence;

[0113] The downsampling convolution normalization activation module a and the downsampling convolution normalization activation module b are composed of a convolution layer with a step size of 2, an instance normalization layer, and a LeakyReLU activation function layer connected in sequence;

[0114] Both convolutional normalized activation dropout blocks consist of a sequentially connected downsampled convolutional normalized activation module a and a dropout layer, and the dropout rate of the dropout layer is 0.3;

[0115] The output of the last convolution normalization activation drop block is the output of the encoder module;

[0116] The decoder module includes a multimodal fusion module a, a multimodal fusion module b, a first upsampling fusion convolution discard block, a second upsampling fusion convolution discard block, a first upsampling fusion convolution block, a second upsampling fusion convolution block, a third upsampling fusion convolution block, a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, a first upsampling module b, a second upsampling module b, a third upsampling module b, a fourth upsampling module b, a first element-by-element addition module, a second element-by-element addition module, a third element-by-element addition module, and a fourth element-by-element addition module;

[0117] The multimodal fusion module b, the first upsampling fusion convolution discard block, the second upsampling fusion convolution discard block, the first upsampling fusion convolution block, the second upsampling fusion convolution block, the third upsampling fusion convolution block, the fifth convolution block, and the fourth element-by-element addition module are connected in sequence;

[0118] The first upsampling fusion convolution discarding block, the first convolution block, the first upsampling module b, the first element-by-element addition module, the second upsampling module b, the second element-by-element addition module, the third upsampling module b, the third element-by-element addition module, the fourth upsampling module b, and the fourth element-by-element addition module are connected in sequence;

[0119] The second upsampling fusion convolution discard block, the second convolution block, and the first element-by-element addition module are connected in sequence;

[0120] The first upsampling fusion convolution block, the third convolution block, and the second element-by-element addition module are connected in sequence;

[0121] The second upsampling fusion convolution block, the fourth convolution block, and the third element-by-element addition module are connected in sequence;

[0122] The first upsampling fusion convolution discard block and the second upsampling fusion convolution discard block are composed of an upsampling module a, a multimodal fusion module a, a standard convolution normalization activation module, and a discard layer connected in sequence, and the discard rate of the discard layer is 0.3;

[0123] The first upsampling fusion convolution block, the second upsampling fusion convolution block and the third upsampling fusion convolution block are all composed of an upsampling module a, a multimodal fusion module a, and a standard convolution normalization activation module connected in sequence;

[0124] The output of the fourth element-by-element addition module is the output of the decoder;

[0125] All the above-mentioned sequential connections refer to sequential connections from front to back according to the data flow direction;

[0126] The workflows of the multimodal fusion module a and the multimodal fusion module b are as follows:

[0127] (i) Concatenate T2 and T1 images to form the Student modality , connect the Flair image and T1c image to form the teacher modality ;

[0128] (ii) Apply a convolutional block operation of size 3×3×3, Operation to obtain student attention weight ;

[0129] Sequentially Apply a convolutional block operation of size 5×5×5, Operation to obtain student attention weight ;

[0130] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the student weighted feature representation ;

[0131] Sequentially Apply a convolutional block operation of size 3×3×3, Operation to get the teacher's attention weight ;

[0132] Sequentially Apply a convolutional block operation of size 5×5×5, Operation to get the teacher's attention weight ;

[0133] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the teacher weighted feature representation ;

[0134] Will and Concatenate to form spatial feature representation ;

[0135] Respectively and After applying the average pooling operation and concatenating the results, the multi-layer perceptron operation and Operation to get attention weight ;

[0136] Respectively and After applying the max pooling operation and concatenating the results, the multi-layer perceptron operation and Operation to get attention weight ;

[0137] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the student weighted feature representation ;

[0138] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the teacher weighted feature representation ;

[0139] Will and Concatenate to form channel attention features ;

[0140] (iii) and Concatenate to form fused feature representation ;

[0141] Execute the image obtained from T2 image, T1 image, Flair image and T1c image The module of the operation is recorded as the spatial attention fusion module (SAFM); the execution is obtained by T2 image, T1 image, Flair image and T1c image The module of the operation is denoted as channel attention fusion module (CAFM);

[0142] Step 2: Determine the loss function of the image segmentation model;

[0143] Loss function for image segmentation model The calculation formula is as follows:

[0144] ;

[0145] ;

[0146] ;

[0147] ;

[0148] ;

[0149] In the formula, = =1, represents the number of voxels in the image, Representing voxels The predicted value of Representing voxels The true value of For 10 -5 , Indicates Monte Carlo sampling of voxels The predicted value of

[0150] The loss function of this embodiment is the uncertainty loss function (UL);

[0151] Step 3: training the image segmentation model;

[0152] (a) 285 brain tumor cases were collected (derived from the dataset of the Brain Tumor Segmentation Challenge (BraTS2018), which is widely used in brain tumor segmentation tasks). Each brain tumor case has T1 images, T1c images, T2 images, and Flair images;

[0153] (b) Preprocessing the T1 image, T1c image, T2 image and Flair image (i.e., each image is subjected to grayscale normalization, pixel adjustment and tensorization in sequence, and the size of the image before preprocessing is 240×240×155 pixels, and the grayscale normalization process is to adjust the image grayscale value to a normal distribution with a mean of 0 and a standard deviation of 1, and the pixel adjustment is to adjust the image to 128×128×128 pixels); obtaining a true segmentation image of each brain tumor case, the true segmentation image being the brain tumor MRI image with the three segmentation regions obtained by manual annotation;

[0154] (c) The training set and test set were constructed using T1 images, T1c images, T2 images, Flair images, and true segmentation images corresponding to 285 brain tumor cases. The training set corresponded to 228 brain tumor cases, and the test set corresponded to 57 brain tumor cases.

[0155] (d) Using the training set to train the image segmentation model, during training, the T1 image, T1c image, T2 image, and Flair image are used as the input of the image segmentation model, and the real segmentation image is used as the theoretical output of the image segmentation model. The weight parameters of the image segmentation model are continuously adjusted until the image segmentation model converges;

[0156] (e) Use the test set to test the trained image segmentation model;

[0157] The dice similarity coefficient (DSC) and Hausdorff distance (HD) are used to characterize the segmentation accuracy of the trained image segmentation model. The calculation formula is as follows:

[0158] ;

[0159] Where P and G represent the predicted region Q (complete tumor region WT, tumor core region TC or enhancement tumor region ET) and the true region Q, respectively. They represent the size of the intersection of the predicted area Q and the real area Q, and Represent the total size of the predicted area Q and the true area Q respectively;

[0160] ;

[0161] Where X and Y represent the predicted region Q boundary and the real region Q boundary respectively. is the Euclidean distance, is the maximum value of the minimum distance from each point on the boundary of the predicted region Q to the boundary of the true region Q, is the maximum value of the minimum distance from each point on the boundary of the true region Q to the boundary of the predicted region Q;

[0162] The test results are shown in Table 1;

[0163] Table 1

[0164]

[0165] Step 4: The T1 image, T1c image, T2 image and Flair image of the same brain tumor are preprocessed (grayscale normalization, pixel adjustment, tensor quantization in sequence), and the preprocessed images are input into the trained image segmentation model, which outputs the mean image, random uncertainty image (such as Figure 8 ), epistemic uncertainty images (such as Fig. 9 as shown), entropy uncertainty image (as shown Fig.10 ) and the mutual information uncertainty image (as shown in Fig.11 as shown).

[0166] The traditional teacher-student mechanism achieves model compression by transferring the knowledge of a large model (teacher model) to a smaller model (student model) through knowledge distillation, so that the small model can achieve higher performance in a resource-constrained environment to solve the balance between model complexity and reasoning efficiency. The present invention innovatively applies the teacher-student mechanism to multimodal feature fusion, designs SAFM and CAFM, and integrates the feature information of the teacher and student modalities respectively, thereby improving the performance of the model in tumor segmentation tasks. This fusion strategy not only enhances the model's ability to capture complex structures, but also significantly improves the accuracy and robustness of segmentation. This fusion strategy overcomes the defects of traditional multimodal fusion methods, that is, traditional methods usually adopt a direct splicing method, ignoring the differences and importance of different modalities in the feature space, resulting in the loss of feature validity during the fusion process, and the inability to fully utilize the complementary information between multimodalities, thereby affecting the segmentation accuracy.

[0167] Comparative Example 1

[0168] A brain tumor segmentation method, such as Figure 4 As shown, it is basically the same as Example 1, except that:

[0169] In step 1, the image segmentation model does not include a Monte Carlo simulation module and an image processing module, and the image X output by the decoder module is the predicted segmentation image;

[0170] In step 2, the loss function of the image segmentation model is The calculation formula is as follows:

[0171] ;

[0172] ;

[0173] In the formula, represents the number of voxels in the image, Representing voxels The predicted value of Representing voxels The true value of For 10 -5 .

[0174] The test results are shown in Table 2.

[0175] Table 2

[0176]

[0177] Comparative Example 2

[0178] A brain tumor segmentation method, such as Figure 5 As shown, it is basically the same as Comparative Example 1, except that:

[0179] In step 1, the workflows of the multimodal fusion module a and the multimodal fusion module b are as follows:

[0180] (i) Concatenate T2 and T1 images to form the Student modality , connect the Flair image and T1c image to form the teacher modality ;

[0181] (ii) Apply a convolutional block operation of size 3×3×3, Operation to obtain student attention weight ;

[0182] Sequentially Apply a convolutional block operation of size 5×5×5, Operation to obtain student attention weight ;

[0183] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the student weighted feature representation ;

[0184] Sequentially Apply a convolutional block operation of size 3×3×3, Operation to get the teacher's attention weight ;

[0185] Sequentially Apply a convolutional block operation of size 5×5×5, Operation to get the teacher's attention weight ;

[0186] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the teacher weighted feature representation ;

[0187] Will and Concatenate to form spatial feature representation .

[0188] The test results are shown in Table 3.

[0189] Table 3

[0190]

[0191] Comparative Example 3

[0192] A brain tumor segmentation method, such as Figure 6 As shown, it is basically the same as Comparative Example 1, except that:

[0193] In step 1, the workflows of the multimodal fusion module a and the multimodal fusion module b are as follows:

[0194] (i) Concatenate T2 and T1 images to form the Student modality , connect the Flair image and T1c image to form the teacher modality ;

[0195] (ii) and After applying the average pooling operation and concatenating the results, the multi-layer perceptron operation and Operation to get attention weight ;

[0196] Respectively and After applying the max pooling operation and concatenating the results, the multi-layer perceptron operation and Operation to get attention weight ;

[0197] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the student weighted feature representation ;

[0198] Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the teacher weighted feature representation ;

[0199] Will and Concatenate to form channel attention features .

[0200] The test results are shown in Table 4.

[0201] Table 4

[0202]

[0203] Comparative Example 4

[0204] A brain tumor segmentation method (the model used is the base model (Baseline)), such as Figure 7 As shown, it is basically the same as Comparative Example 1, except that:

[0205] Both the multimodal fusion module a and the multimodal fusion module b are replaced by series modules.

[0206] The test results are shown in Table 5.

[0207] Table 5

[0208]

[0209] From Tables 1 to 5, we can see that on the Baseline, without adding CAFM, SAFM and UL, the average DSC is 82.3% and the average HD is 5.4; on the Baseline, with CAFM, the average DSC is 82.7% and the average HD is 4.8; on the Baseline, with SAFM, the average DSC is 83.2% and the average HD is 4.9; on the Baseline, with CAFM and SAFM, the average DSC is 84.1% and the average HD is 4.5; on the Baseline, with CAFM, SAFM and UL, the average DSC is 84.3% and the average HD is 4.2. By comparison, we can see that the addition of CAFM, SAFM and UL can effectively improve the accuracy of the segmentation model, and the segmentation accuracy and robustness of the model have been significantly improved, which is suitable for brain tumor segmentation tasks with high accuracy requirements.

Claims

1. A brain tumor segmentation method based on uncertainty estimation, characterized in that: The T1 image, T1c image, T2 image and Flair image of the same brain tumor are input into the trained image segmentation model, which outputs the mean image, random uncertainty image, cognitive uncertainty image, entropy uncertainty image and mutual information uncertainty image; The image segmentation model includes 4 encoder modules, 1 decoder module, 1 Monte Carlo simulation module, and 1 image processing module; Both the encoder and decoder modules contain dropout layers; The four encoder modules downsample the T1 image, T1c image, T2 image, and Flair image respectively, and the one decoder module upsamples the downsampled T1 image, T1c image, T2 image, and Flair image to obtain image X; The Monte Carlo simulation module repeatedly calls four encoder modules and one decoder module to perform Monte Carlo sampling image X, =10-50; Image processing module The image X is processed to obtain the mean image, random uncertainty image, cognitive uncertainty image, entropy uncertainty image and mutual information uncertainty image; The mean image is the predicted segmentation image, which is a brain tumor MRI image with three segmentation regions obtained through prediction. The three segmentation regions are the complete tumor region, the tumor core region, and the enhancement tumor region.

2. The brain tumor segmentation method based on uncertainty estimation according to claim 1, characterized in that: The dropout rate of each dropout layer is 0.

3.

3. The brain tumor segmentation method based on uncertainty estimation according to claim 1, characterized in that: The decoder module includes multimodal fusion module a and multimodal fusion module b, and the workflows of both are as follows: (a) Concatenate T2 and T1 images to form the student mode , connect the Flair image and T1c image to form the teacher modality ; (b) Apply a convolutional block operation of size 3×3×3, Operation to obtain student attention weight ; Sequentially Apply a convolutional block operation of size 5×5×5, Operation to obtain student attention weight ; Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the student weighted feature representation ; Sequentially Apply a convolutional block operation of size 3×3×3, Operation to get the teacher's attention weight ; Sequentially Apply a convolutional block operation of size 5×5×5, Operation to get the teacher's attention weight ; Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the teacher weighted feature representation ; Will and Concatenate to form spatial feature representation ; Respectively and After applying the average pooling operation and concatenating the results, the multi-layer perceptron operation and Operation to get attention weight ; Respectively and After applying the max pooling operation and concatenating the results, the multi-layer perceptron operation and Operation to get attention weight ; Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the student weighted feature representation ; Will and Perform element-by-element multiplication and and Perform element-by-element multiplication and add the two products element-by-element to obtain the teacher weighted feature representation ; Will and Concatenate to form channel attention features ; (c) and Concatenate to form fused feature representation .

4. The brain tumor segmentation method based on uncertainty estimation according to claim 3, characterized in that: The structures of the four encoder modules are the same, which include a down-sampled initial block, two down-sampled convolutional normalized activation modules b, and two convolutional normalized activation discard blocks connected in sequence; The downsampling initial block consists of a standard convolutional normalized activation module and a downsampling convolutional normalized activation module a connected in sequence; Both convolution-normalized activation dropout blocks consist of a downsampled convolution-normalized activation module a and a dropout layer connected in sequence; The output of the last convolution normalization activation drop block is the output of the encoder module; The decoder module also includes a first upsampling fusion convolution discard block, a second upsampling fusion convolution discard block, a first upsampling fusion convolution block, a second upsampling fusion convolution block, a third upsampling fusion convolution block, a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, a first upsampling module b, a second upsampling module b, a third upsampling module b, a fourth upsampling module b, a first element-by-element addition module, a second element-by-element addition module, a third element-by-element addition module, and a fourth element-by-element addition module; The multimodal fusion module b, the first upsampling fusion convolution discard block, the second upsampling fusion convolution discard block, the first upsampling fusion convolution block, the second upsampling fusion convolution block, the third upsampling fusion convolution block, the fifth convolution block, and the fourth element-by-element addition module are connected in sequence; The first upsampling fusion convolution discarding block, the first convolution block, the first upsampling module b, the first element-by-element addition module, the second upsampling module b, the second element-by-element addition module, the third upsampling module b, the third element-by-element addition module, the fourth upsampling module b, and the fourth element-by-element addition module are connected in sequence; The second upsampling fusion convolution discard block, the second convolution block, and the first element-by-element addition module are connected in sequence; The first upsampling fusion convolution block, the third convolution block, and the second element-by-element addition module are connected in sequence; The second upsampling fusion convolution block, the fourth convolution block, and the third element-by-element addition module are connected in sequence; The first upsampling fusion convolution discard block and the second upsampling fusion convolution discard block are both composed of an upsampling module a, a multimodal fusion module a, a standard convolution normalization activation module, and a discard layer connected in sequence; The first upsampling fusion convolution block, the second upsampling fusion convolution block and the third upsampling fusion convolution block are all composed of an upsampling module a, a multimodal fusion module a, and a standard convolution normalization activation module connected in sequence; The output of the fourth element-by-element addition module is the output of the decoder; The standard convolutional normalization activation module consists of a convolutional layer, an instance normalization layer, and a LeakyReLU activation function layer connected in sequence; The downsampling convolution normalization activation module a and the downsampling convolution normalization activation module b are composed of a convolution layer with a step size of 2, an instance normalization layer, and a LeakyReLU activation function layer connected in sequence; All sequential connections refer to connections from front to back according to the data flow direction.

5. The brain tumor segmentation method based on uncertainty estimation according to claim 1, characterized in that: Loss function for image segmentation model The calculation formula is as follows: ； ； ； ； ； In the formula, = =1, represents the number of voxels in the image, Representing voxels The predicted value of Representing voxels The true value of For 10 -5 , Indicates Monte Carlo sampling of voxels The predicted value of .

6. A brain tumor segmentation method based on uncertainty estimation according to any one of claims 1 to 5, characterized in that: The T1 image, T1c image, T2 image and Flair image of the same brain tumor are preprocessed before being input into the trained image segmentation model. The preprocessing includes grayscale normalization, pixel adjustment and tensor quantization.

7. The brain tumor segmentation method based on uncertainty estimation according to claim 6, characterized in that: The training steps of the image segmentation model are as follows: (a) Collection Brain tumor cases, ≥285, each brain tumor case had T1 images, T1c images, T2 images, and Flair images; (b) performing the above-mentioned preprocessing on the T1 image, T1c image, T2 image and Flair image; obtaining a true segmentation image of each brain tumor case, where the true segmentation image is a brain tumor MRI image with the above-mentioned three segmentation regions obtained by manual annotation; (c) Adoption The training and test sets were constructed using T1 images, T1c images, T2 images, Flair images, and true segmentation images corresponding to each brain tumor case. (d) Using the training set to train the image segmentation model, during training, the T1 image, T1c image, T2 image, and Flair image are used as the input of the image segmentation model, and the real segmentation image is used as the theoretical output of the image segmentation model. The weight parameters of the image segmentation model are continuously adjusted until the image segmentation model converges; (e) Use the test set to test the trained image segmentation model.

Images