Integrated MP-Unet Segmentation Method Based on Multimodal MRI

Abstract

The present invention aims to address the problem of low utilization of the multi-modal modalities of MRI (Magnetic Resonance Imaging) in existing methods. An integrated MP-Unet segmentation method based on multimodal MRI is proposed. Its implementation strategy is as follows: normalize and denoise the image; generate a Unet dataset and measure the modality quality by Unet accuracy; use high-quality modalities as the dataset for MP-Unet model training; construct an MP-Unet and perform model training; and combine the trained multiple MP-Unet models with the bagging algorithm for integrated decision segmentation. When performing target segmentation on MRI images, the present invention can combine multiple modal data to greatly improve the segmentation accuracy of MRI images. The bagging algorithm can integrate multiple MP-Unets to further improve the segmentation accuracy of MRI images and effectively reduce incorrect predictions.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and further relates to a multi-modal magnetic resonance imaging (MRI) integrated multi-path U-shaped network (MP-Unet) segmentation method in the field of medical image segmentation. The present invention can be used for segmenting multi-parameter MRI medical images taken by the same nuclear magnetic resonance device, understanding the non-linear relationship between multiple modalities through a neural network, and combining the Bagging algorithm for integrated decision-making to obtain high-precision segmentation results. Background Art

[0002] The segmentation of magnetic resonance imaging (MRI) aims to separate the target from other tissues. Due to the differences in the appearance, shape, and position of the target to be segmented, the segmentation of MRI images becomes very complex. Multi-modal MRI is an advanced imaging technology that provides more image information by using multiple different imaging modes. For example, nuclear magnetic resonance T1W highlights the differences in transverse relaxation of tissues, and T2W highlights the differences in longitudinal relaxation of tissues. When existing MRI segmentation technologies involve multi-modal processing, they usually only use one or two modalities with the clearest contour of the target to be segmented for target segmentation, resulting in insufficient utilization of image information. This is because existing segmentation technologies cannot effectively handle the highly non-linear relationship between each modality, while neural networks can fully understand this relationship and combine the advantages of multi-modal MRI for high-precision target segmentation.

[0003] Traditional segmentation methods mainly include threshold segmentation method, region growing method, and edge segmentation algorithm. These traditional algorithms usually have good segmentation effects and do not require manual annotation, so they are widely used. However, such algorithms are less efficient and are easily affected by noise, resulting in large errors.

[0004] The segmentation method based on deep learning extracts features in medical images through a large number of linear and non-linear combinations, which is why deep learning can handle the highly non-linear relationship between multi-modal MRI. The deep learning method has a very good segmentation effect and strong robustness by learning the segmentation pattern from a large amount of data. However, since the deep learning model is prone to overfitting, the training of the model requires a large amount of data as support, which is the biggest shortcoming of the deep learning method. Nevertheless, the deep learning method has gradually become the mainstream solution in the field of medical image segmentation.

[0005] Northwestern Polytechnical University disclosed a dual-modal brain tumor MRI segmentation method based on a clustering fusion algorithm in its patent document "A Dual-Modal Brain Tumor MRI Segmentation Method Based on a Clustering Fusion Algorithm" (application number 201811414799.3, publication number CN 109685767 A). The main steps of this method are as follows: First, input different modal images of tumor patients, including T2-weighted and Flair images, normalize the pixel values of the input images, and extract pixel points and features; Second, integrate three distance K-means clustering algorithms and output the clustering results; Third, use the extracted largest connected region to output the final result. Although this method integrates two-modal MRI data. However, the deficiencies of this method are still that it does not make full use of the advantages of multi-modal MRI, lacks comprehensive imaging information, resulting in low segmentation accuracy. Since the clustering algorithm only uses the distance formula to establish connections between modalities, it cannot effectively fuse features across modalities. Finally, this method extracts the largest connected region as the final prediction result, and the eliminated free regions may have many incorrect predictions, resulting in very limited improvement in accuracy.

[0006] Zhejiang University of Technology disclosed a brain tumor image segmentation method combining multi-modal information in its patent document "A Brain Tumor Image Segmentation Method Combining Multi-Modal Information" (application number 202110652321.X, publication number CN 113496493 A). The implementation steps of this method are as follows: First, divide the data into image patches; Second, construct a prior network to extract multi-modal features; Third, train a multi-modal feature fusion network, inject the extracted multi-modal information into the attention module, and obtain the result after upsampling. The deficiencies of this method are that the constructed segmentation model processes all modal data at once. In a medical dataset with a very small sample size, the training of this model is extremely prone to overfitting. There are differences in the quality of different modalities. This method does not repeat or enhance the utilization of those modalities with excellent quality, and the utilization rate of high-quality modalities is low. Moreover, this method has a single application scenario and cannot be applied in the case of more modalities. Summary of the Invention

[0007] The object of the present invention is to address the deficiencies of the above-mentioned prior art and propose an integrated multi-path Unet segmentation method based on multi-modal MRI sequences to solve the problems of single application scenario and inapplicability in the case of more modalities, as well as the problem of ineffective fusion of features across modalities and the problem of incorrect predictions in free regions.

[0008] The idea to achieve the object of the present invention is that the present invention uses a hierarchical fusion strategy and constructs an MP-Unet model in a multi-encoder manner. After importing the pre-trained parameters into the backbone network, the network is unfrozen for training, enabling the model to simultaneously process three-modal MRI data. Different modalities perform feature extraction separately in three backbone networks. Importing pre-trained parameters for model training can ensure stable feature extraction ability and simultaneously learn the relationships between different modalities. Features are merged in channels at the intersection of the encoder and the decoder, and channel merging also occurs during the downsampling of the modal feature maps and then jumps to the corresponding encoding path, enabling the model to fully understand the non-linear relationships of different modalities, improving the utilization rate of the MRI modality, and thus obtaining consistent and robust segmentation results, eliminating the ambiguity caused by cross-modal segmentation, thereby solving the problem of maximizing the elimination of modal redundancy in the case of multi-modal MRI. The present invention uses a hierarchical fusion strategy and constructs an MP-Unet model in a multi-encoder manner. After importing the pre-trained parameters into the backbone network, the network is unfrozen for training. Multiple trained MP-Unet models are integrated, and the bagging (Bootstrap aggregating) algorithm is used for majority voting decision-making, which further improves the model accuracy. The training methods of the base models in bagging are the same, and the entire learning process does not perform artificial sub-problem division, and the entire training process is relatively simple. This greatly reduces the incorrect predictions in the segmentation results.

[0009] The specific steps of the method of the present invention are as follows:

[0010] Step 1, preprocess the images:

[0011] Step 1.1, perform grayscale value normalization processing on the images of each MRI modality:

[0012] Step 1.2, perform threshold denoising processing on the images of three MRI modalities, namely T1Mapping, T2Mapping, and STIR:

[0013] Step 2, generate a dataset for Unet training:

[0014] Generate a sample set of each modality from the preprocessed images of each modality, and divide the samples in each sample set into a training set for training the Unet model and a validation set for detecting the accuracy of the Unet model of this modality at a ratio of 7:3;

[0015] Step 3, train the Unet model of each modality:

[0016] Step 3.1, set the training batch size parameter to 1 and the training period to 300 epochs;

[0017] Step 3.2, Input the training set of each modality into the Unet model respectively, calculate the loss value between the prediction result and the annotation map, and use the Adam algorithm to optimize the loss value for 300 epochs to obtain the trained Unet model for each modality;

[0018] Step 4, Select high-quality modalities for the MP-Unet model:

[0019] Input the validation set into the trained Unet model of each modality, and judge the quality of the MRI image modality by comparing the Dice coefficient accuracy of Unet on the validation set. The higher the accuracy, the better the modality quality. The modality quality is ranked as M1, M2,...., M n , where M i represents the MRI modality with the i-th ranked modality quality. Select the first k modalities whose difference in Unet accuracy between all the k-th and (k + 1)-th ranked modalities exceeds 5% as high-quality modalities;

[0020] Step 5, Generate the MP-Unet model dataset:

[0021] Step 5.1, Combine the k high-quality modalities into k - 2 groups of datasets, and each group of datasets contains three high-quality modalities;

[0022] Step 5.2, Perform augmentation processing of flipping, rotating, cropping, deforming, and scaling on each group of datasets in turn, and split each group of augmented datasets into a training dataset for the MP-Unet model and a validation set for performance verification at a ratio of 7:3;

[0023] Step 6, Construct the MP-Unet model:

[0024] Build an MP-Unet model composed of three encoders and one decoder. All three encoders of the MP-Unet use the VGG16 network;

[0025] Before each downsampling, the feature maps obtained in the three encoding paths are copied and concatenated, and the resulting multi-channel feature maps are used as the input of the decoder. The multi-channel feature maps from the multi-path concatenation are concatenated with the feature maps of the same resolution in the decoding path;

[0026] After the final output feature map of the MP-Unet decoder is calculated by the softmax function, the sum of the Dice loss function and the cross-entropy loss function is used as the MP-Unet loss function;

[0027] The encoder contains five downsamplings. Before each downsampling, two convolutional operations are performed, the size of each convolutional kernel is set to 3*3, and the ReLU function is used as the activation layer after each convolutional operation. The downsampling is implemented through the max-pooling layer;

[0028] The decoder includes five times of upsampling. Before each upsampling, two convolutional operations are performed. The size of each convolutional kernel is set to 3*3. After each convolutional operation, the ReLU function is used as the activation layer. The upsampling is implemented using the nearest interpolation algorithm. The softmax function is used for activation in the last layer of the model;

[0029] Step 7, training the MP-Unet model:

[0030] Step 7.1, form a training batch by every 2 samples in the training set;

[0031] Step 7.2, initialize the backbone part of the MP-Unet model with the parameters of the VGG16 model trained on the ImageNet dataset;

[0032] Step 7.3, perform forward propagation on the training batch input to the MP-Unet segmentation model, and output the segmented image;

[0033] Step 7.4, use the MP-Unet loss function to calculate the loss value between the image segmented by the MP-Unet model and the corresponding annotated image;

[0034] Step 7.5, use the Adam algorithm to optimize the MP-Unet loss function, and repeat steps 7.3 to 7.4 for 300 epochs to obtain the optimal MP-Unet model weights;

[0035] Step 7.6, perform the same operations as steps 7.1 to 7.5 on the k-2 groups of datasets in step 5.1 to obtain k-2 trained MP-Unet models;

[0036] Step 8, use the Bagging algorithm to perform decision segmentation on all MP-Unet models:

[0037] Step 8.1, input the sample to be segmented into the trained MP-Unet model corresponding to the modality;

[0038] Step 8.2, all k-2 MP-Unets execute step 8.1 to generate k-2 prediction results, which are Seg 1 , Seg 2 ,..., Seg k-2 ;

[0039] Step 8.3, the values of all prediction results at the pixel position (i, j) are respectively Seg i,jThe possible values are 0 and 1, representing the foreground and background respectively. If the number of occurrences of the value 1 at the position (i, j) is more than that of the value 0, then 1 is selected as the final prediction result at this position; otherwise, 0 is selected as the final prediction result at this position. By performing the above operations on all pixel positions, the final integrated decision segmentation result is obtained.

[0040] Compared with the existing technologies, the present invention has the following advantages:

[0041] First, the present invention utilizes a hierarchical fusion strategy and constructs an MP-Unet model in a multi-encoder manner. After importing the pre-trained parameters into the backbone network, it is unfrozen for training, enabling the model to simultaneously process three-modal MRI data, avoiding the risk of overfitting in the training of the model, overcoming the defect of the existing technology of maximizing the elimination of modal redundancy in the case of multi-modal MRI, and enabling the present invention to obtain consistent and highly robust segmentation results in multiple modalities while improving the utilization rate of MRI modalities.

[0042] Second, the present invention integrates multiple trained MP-Unet models and uses the bagging algorithm for maximum voting decision-making, overcoming the deficiency of the existing multi-modal segmentation technology in the low utilization rate of high-quality modalities, enabling the present invention to improve the contour clarity of the MRI image segmentation target and greatly eliminating the incorrect predictions in the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is the flowchart of the present invention;

[0044] Figure 2 is a schematic diagram of the multi-modal MRI axial image of the same subject in the multi-modal data collection of the present invention;

[0045] Figure 3 is the comparison chart before and after the noise reduction processing of the present invention;

[0046] Figure 4 is the schematic diagram of the MP-Unet network structure of the present invention;

[0047] Figure 5 is the schematic diagram of the effect after the integrated decision of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0048] The present invention will be further described below in conjunction with the drawings and embodiments.

[0049] Refer to Figure 1 for a further detailed description of the implementation steps of the embodiments of the present invention.

[0050] Step 1, data preprocessing.

[0051] Step 1.1, perform gray value normalization on each modal data.

[0052] Figure 2 This is a schematic diagram of multi-modal MRI axial images of the same subject in collecting multi-modal data for the present invention, including 9 modalities. Among them, Figure 2 (a) to Figure 2 (i) represent nine MRI modalities of the subject, Figure 2 (j) shows the labeled data shared by each modality, Figure 2 (a) is T1-weighted T1 (T1W), Figure 2 (b) is T2-weighted (T2W), Figure 2 (c) is T1 fluid-attenuated inversion recovery (T1WFlair), Figure 2 (d) is T2 fluid-attenuated inversion recovery (T2WFlair), Figure 2 (e) is short tau inversion recovery (STIR), Figure 2 (f) is proton density weighted (PDW), Figure 2 (g) is proton density mapping (PDMapping), Figure 2 (h) is T1-weighted mapping (T1Mapping), Figure 2 (i) is T2-weighted mapping (T2Mapping).

[0053] Since multiple MRI modality scans taken from the same subject only differ in the gray values of the images, and the positions of the objects to be segmented in each modality are the same, it is necessary to manually annotate the dataset by selecting the T2WFlair modality with the clearest contour of the object to be segmented. This means that all modality data share one annotated data, Figure 2 (j) shows the manual annotation of the object to be segmented by an expert in the embodiment of the present invention.

[0054] Before training the model, the following formula is used to normalize each pixel value in the image, which can reduce the loss of the neural network and effectively accelerate the convergence speed of the model:

[0055]

[0056] where, Y k represents the normalized pixel value of the k-th pixel in the input image, X k represents the pixel value of the k-th pixel in the input image, X min represents the minimum pixel value in the input image, X max represents the maximum pixel value in the input image.

[0057] Step 1.2, perform threshold denoising on the T1Mapping, T2Mapping, and STIR modality data.

[0058] Among the nine modalities of data in the embodiments of the present invention, only the data of three modalities, namely T1Mapping, T2Mapping, and STIR, will generate severe salt-and-pepper noise on the image during generation. Therefore, after normalizing the images of these three modalities, it is necessary to scale the gray values of their images in the interval of [0, 1]. Since the gray values of the noise are much higher than those of other regions, the gray values of a small amount of noise will be scaled to near 1, while the gray values of the normal regions will be compressed to near 0, and the contrast between the region to be segmented and the surrounding tissues will be significantly reduced, which is very unfavorable for the model to segment the target region.

[0059] The embodiments of the present invention generate three gray histograms for T1Mapping, T2Mapping, and STIR modalities respectively. The gray histogram reflects the relationship between the frequency of pixels with each gray level appearing in an image and the gray level. Each gray histogram uses the gray level as the abscissa and the frequency as the ordinate. The ordinate frequency of the gray histogram of the normal modality usually shows a distribution that first rises and then falls. The modality with noise will suddenly increase after the frequency drops, and the noise threshold is determined as the abscissa gray level before the sudden increase. Since the salt-and-pepper noise in the embodiments of the present invention almost always appears in the periphery of the target (i.e., the background part). The gray values in the image greater than this threshold are set to 0 (the same as the background).

[0060] Figure 3 Shows the images before and after denoising in the embodiments of the present invention, where Figure 3 (a) to Figure 3 (c) represent the images without denoising, Figure 3 (d) to Figure 3 (f) represent the images after denoising.

[0061] Step 2, generate a dataset for Unet training.

[0062] In the embodiments of the present invention, since each subject contains nine modalities of data, and each modality only differs in the image gray value, and the size and shape of the target to be segmented are the same, therefore, the present invention generates a dataset for each modality, and each dataset is divided into a training set and a validation set in a ratio of 7:3. The training set is used for the training of the Unet model, and the validation set is used to detect the accuracy of the Unet model. The accuracy of the validation set of a modality dataset represents the quality of the modality.

[0063] The Dice coefficient is used to measure the Unet accuracy:

[0064]

[0065] Where X represents the predicted result map of the Unet model, and Y represents the annotation map.

[0066] Step 3: Train the Unet model on each modality dataset.

[0067] The batch size for training each Unet model is set to 1, and the number of training epochs is set to 300. After the model inputs the data, the prediction results are calculated with the annotation map for loss, and the Adam algorithm is used for loss optimization.

[0068] Step 4: Select high-quality modalities for the MP-Unet model.

[0069] Input the validation set into the trained Unet model of each modality. By comparing the Dice coefficient accuracy of Unet on the validation set, judge the quality of the MRI image modality. The higher the accuracy, the better the modality quality. The modality quality ranking is M1, M2,...., M n where M i represents the MRI modality with the i-th ranked modality quality. Select the first k modalities whose difference in Unet accuracy between the k-th and (k + 1)-th ranked modalities exceeds 5% as high-quality modalities.

[0070] In the embodiment of the present invention, the accuracies of Unet on each modality validation set are ranked from high to low as follows: T1W (70.19%), T1WFlair (69.56%), T2WFlair (68.43%), T2W (68.33%), T1Mapping (65.79%), PDW (58.45%), PDMapping (55.39%), T2Mapping (54.40%), STIR (52.66%).

[0071] It is found through the embodiment of the present invention that due to the too low accuracies of the last four modalities, the performance of the generated MP-Unet model will also be very poor, seriously dragging down the accuracy of the integrated prediction in the last step. Therefore, the first five high-quality modalities are selected in the embodiment of the present invention for training the MP-Unet.

[0072] Step 5: Generate the MP-Unet model dataset.

[0073] Step 5.1: Combine the k high-quality modalities into k - 2 groups of datasets, with each dataset containing three high-quality modalities. In the embodiment of the present invention, five modalities are used to generate three datasets. The first dataset contains T1W, T1WFlair, and T2WFlair; the second dataset contains T1W, T1WFlair, and T2W; T1W, T1WFlair, and T1Mapping.

[0074] Step 5.2, perform augmentation processing of flipping, rotation, cropping, deformation, and scaling on each group of datasets in turn, and split each group of augmented datasets into a training dataset for the MP-Unet model and a validation set for performance verification at a ratio of 7:3;

[0075] Step 6, construct the MP-Unet model.

[0076] Figure 4 It is a schematic diagram of the MP-Unet network structure constructed by the present invention. This MP-Unet network structure has multiple backbone encoders and a decoder. Its architecture can simultaneously accommodate the input of multiple modality data, and uses the VGG16 classifier network as the encoder of the MP-Unet. Before each downsampling, the feature maps obtained in the three encoding paths are copied and concatenated. The multi-channel feature maps from the multi-path concatenation are concatenated with the feature maps of the same resolution in the decoding path to recover the image information lost due to downsampling in the encoder.

[0077] The encoder includes five downsamplings. Before each downsampling, two convolutional operations are performed, the size of each convolutional kernel is set to 3*3, and the ReLU function is used as the activation layer after each convolutional operation. The downsampling is implemented through the max pooling layer.

[0078] The decoder includes five upsamplings. Before each upsampling, two convolutional operations are performed, the size of each convolutional kernel is set to 3*3, and the ReLU function is used as the activation layer after each convolutional operation. The upsampling is implemented using the nearest interpolation algorithm. The softmax function is used for activation in the last layer of the model.

[0079] After the final output feature map of the network is calculated by the softmax function, the sum of the Dice loss function and the cross-entropy loss function is used as the MP-Unet loss function.

[0080] The MP-Unet loss function is as follows:

[0081]

[0082] where L represents the loss value, X is the predicted value of the model, Y is the expert annotation map, N represents the number of pixels in the prediction result, y c represents the label value (0 or 1), and p c represents the predicted probability after the softmax function.

[0083] Step 7, train multiple MP-Unet models.

[0084] The backbone part of the model is initialized with the VGG16 model parameters trained on the ImageNet dataset to ensure the stable convergence of the model and avoid overfitting.

[0085] During the training process, the model inputs three different modalities of MRI images of the same subject respectively. After forward propagation, the predicted results are obtained and the loss is calculated with the labeled data. The Adam algorithm optimizer is used to optimize the loss, and the learning rate in the optimizer adopts the adjustment strategy of the following formula:

[0086]

[0087] where epoch i represents the training of the i-th cycle, new_lr represents the new learning rate obtained after each step_size training cycle, step_size is initialized to 1, initial_lr represents the initial learning rate, γ represents the update factor with an initial value of 0.9, and new_lr will generate a new value in each round of training, and the learning rate changes dynamically.

[0088] The multiple groups of datasets generated in step 5 are used to train multiple MP-Unet models. The training cycle of each group of models is 300 epochs, and the batch size is set to 1.

[0089] In step 8, the Bagging algorithm is executed for integration.

[0090] In step 8.1, the sample to be segmented is input into the trained MP-Unet model of the corresponding modality.

[0091] In step 8.2, k - 2 MP-Unets all execute step 8.1 to generate k - 2 predicted results, which are Seg 1 , Seg 2 ,..., Seg k-2 .

[0092] In step 8.3, the values of all predicted results at the pixel position (i, j) are The possible values of Seg i,j are 0 and 1, representing the foreground and background respectively. If the number of occurrences of the value 1 at the position (i, j) is more than that of the value 0, then 1 is selected as the final predicted result at this position. Otherwise, 0 is selected as the final predicted result at this position. By performing the above operations on all pixel positions, the final integrated decision segmentation result is obtained, Figure 5 is the effect of the integrated decision, where Figure 5 (a) is the expert annotation map, Figure 5 (b) is the result map after segmentation by a single network, and Figure 5 (c) is the result map after integrated decision.

[0093] The essence of semantic segmentation is a pixel-level classification algorithm. At the last layer of the segmentation network, the softmax or sigmoid operation classifies each pixel as foreground or background. Similar to the traditional Bagging algorithm, multiple MP-Unet perform further voting on each classified pixel, and the result with the most votes is used as the prediction result, thus obtaining the final result of integrated segmentation. At the same time, it can also eliminate free incorrect predictions. The integrated segmentation model can remove a large number of incorrect detections from the single-modal segmentation result, further improving the segmentation accuracy.

Claims

1. An integrated MP-Unet segmentation method based on multi-modal MRI, characterized in that, Construct the MP-Unet model and use the Bagging algorithm to implement the integrated decision-making segmentation of all MP-Unet models; the steps of this segmentation method are as follows: Step 1, preprocess the images: Step 1.1, perform gray value normalization on the images of each MRI modality: Step 1.2, perform threshold denoising on the images of three MRI modalities, namely T1Mapping, T2Mapping, and STIR: Step 2, generate a dataset for Unet training: Generate a sample set for each modality from the preprocessed images of each modality, and divide the samples in each sample set into a training set for training the Unet model and a validation set for detecting the accuracy of the Unet model at a ratio of 7:3 for each modality; Step 3, train the Unet model for each modality: Step 3.1, set the training batch size parameter to 1 and the training period to 300 epochs; Step 3.2, input the training set of each modality into the Unet model respectively, calculate the loss value between the prediction result and the annotation map, and use the Adam algorithm to optimize the loss value for 300 epochs to obtain the trained Unet model for each modality; Step 4, select high-quality modalities for the MP-Unet model: The validation set is input into the trained Unet model for each modality. By comparing the Dice coefficient accuracy of Unet on the validation set, the quality of the MRI image modality is judged. The higher the accuracy, the better the modality quality. The modality quality ranking is M1, M2,...., M n , where M i represents the MRI modality with the i-th ranked modality quality. The first k modalities with the difference in Unet accuracy between all the k-th and (k + 1)-th ranked modalities exceeding 5% are selected as high-quality modalities; Step 5, generate the MP-Unet model dataset: Step 5.1, form k-2 groups of datasets from k high-quality modalities, and each group of datasets contains three high-quality modalities; Step 5.2, perform enhancement processing of flipping, rotating, cropping, deforming, and scaling on each group of datasets in turn, and divide each group of enhanced datasets into a training dataset for the MP-Unet model and a validation set for performance verification at a ratio of 7:3; Step 6, construct the MP-Unet model: Build an MP-Unet model composed of three encoders and one decoder. All three encoders of the MP-Unet use the VGG16 network; Before each downsampling, the feature maps obtained in the three encoding paths are copied and concatenated, and the resulting multi-channel feature maps are used as the input of the decoder. The multi-channel feature maps from the multi-path concatenation are concatenated with the feature maps of the same resolution in the decoding path; After the final output feature map of the MP-Unet decoder is calculated by the softmax function, the sum of the Dice loss function and the cross-entropy loss function is used as the MP-Unet loss function; The encoder contains five downsamplings. Before each downsampling, two convolutional operations are performed, the size of each convolutional kernel is set to 3*3, and the ReLU function is used as the activation layer after each convolutional operation. The downsampling is implemented through the max pooling layer; The decoder contains five upsamplings. Before each upsampling, two convolutional operations are performed, the size of each convolutional kernel is set to 3*3, and the ReLU function is used as the activation layer after each convolutional operation. The upsampling is implemented using the nearest interpolation algorithm, and the softmax function is used for activation in the last layer of the model; Step 7, train the MP-Unet model: Step 7.1, form a training batch by every 2 samples in the training set; Step 7.2, the backbone part of the MP-Unet model is initialized with the parameters of the VGG16 model trained on the ImageNet dataset; Step 7.3, perform forward propagation on the training batches input into the MP-Unet segmentation model, and output the segmented images; Step 7.4, use the MP-Unet loss function to calculate the loss value between the images segmented by the MP-Unet model and the corresponding annotated images; Step 7.5, use the Adam algorithm to optimize the MP-Unet loss function, and repeat Steps 7.3 to 7.4 for a total of 300 epochs to obtain the optimal MP-Unet model weights; Step 7.6, for the k-2 groups of datasets in Step 5.1, perform the same operations as in Steps 7.1 to 7.5 to obtain k-2 trained MP-Unet models; Step 8, use the Bagging algorithm to perform decision segmentation on all MP-Unet models: Step 8.​​1, input the sample to be segmented into the trained MP-Unet model of the corresponding modality; Step 8.2, k - 2 MP-Unets all execute Step 8.1 to generate k - 2 prediction results, namely Seg 1 , Seg 2 ,..., Seg k-2 ; Step 8.3, the values of all prediction results at the pixel position (i, j) are respectively Seg i,j The possible values of are 0 and 1, representing the foreground and the background respectively. If the number of occurrences of the value 1 at the position (i, j) is more than that of the value 0, then 1 is selected as the final prediction result at this position; otherwise, 0 is selected as the final prediction result at this position. By performing the above operations on all pixel positions, the final integrated decision segmentation result is obtained.

2. The integrated MP-Unet segmentation method based on multi-modal MRI according to claim 1, wherein The gray value normalization described in Step 1.1 is achieved by the following formula: Among them, Y k represents the normalized pixel value of the k-th pixel in the input image, and X k represents the pixel value of the k-th pixel in the input image, and X min represents the minimum pixel value in the input image, and X max represents the maximum pixel value in the input image.

3. The integrated MP-Unet segmentation method based on multi-modal MRI according to claim 1, wherein The steps of the threshold denoising process described in Step 1.2 are as follows: First step, generate gray histograms corresponding to the T1Mapping, T2Mapping, and STIR modalities respectively. Each gray histogram uses the gray level as the horizontal coordinate and the frequency as the vertical coordinate; Second step, after the frequency of each modality gray histogram drops, the gray level of the horizontal coordinate corresponding to the sudden increase in the vertical coordinate frequency is used as the noise threshold of that modality; Second step, set the gray value of the elements in the image greater than this threshold to 0, which is the same as the background.

4. The integrated MP-Unet segmentation method based on multimodal MRI according to claim 1, characterized in that, The Dice coefficient accuracy described in Step 1.2 is obtained by the following formula: Among them, X represents the predicted result map of the Unet model, and Y represents the annotated map.

5. The integrated MP-Unet segmentation method based on multimodal MRI according to claim 4, wherein The MP-Unet loss function described in Step 6 is as follows: Among them, L represents the loss value, X represents the prediction result map of the MP-Unet model, Y represents the annotation map, N represents the number of pixels in the prediction result, y c represents the label value, and p c represents the output value after passing through the softmax function.

6. The integrated MP-Unet segmentation method based on multimodal MRI according to claim 1, wherein The training process in Step 7 refers to inputting the three different modality MRI images of the same subject into the MP-Unet model respectively, using the Adam algorithm optimizer, and calculating the loss value between the prediction result and the annotated data obtained after forward propagation; the adjustment strategy of the learning rate in the optimizer is as follows: Among them, epoch i represents the training of the i-th cycle. new_lr represents the new learning rate obtained after each step_size training cycle. step_size is initialized to 1. initial_lr represents the initial learning rate. γ represents the update factor with an initial value of 0.

9. A new value of new_lr will be generated in each round of training, and the learning rate changes dynamically.

Images