MRI (Magnetic Resonance Imaging) image segmentation method with adaptive mode

Abstract

The invention belongs to the technical field of medical image segmentation, and particularly relates to a mode-adaptive MRI image segmentation method. According to the method, features of different MRI modes are extracted independently, and correlation among the features of the different modes is modeled through a cross attention mechanism; meanwhile, the modeled modal correlation feature representation is enhanced through a multi-head self-attention mechanism, so that useful feature information is highlighted; besides, a mode-specific decoder is adopted to respectively obtain segmentation results of different modes, so that the encoder of each mode can learn identification features of a target area according to the segmentation loss in the feature extraction process, and the problem of unbalanced training of the network when the modes are incomplete is avoided. According to the method provided by the invention, modal self-adaption of the network can be realized, so that effective image segmentation can be carried out under any MRI modal combination, the problem that the image segmentation precision is reduced under the condition of incomplete modality is solved, and image synthesis is not needed.

Description

Technical Field

[0001] The invention relates to a modality-adaptive MRI image segmentation method, belonging to the technical field of medical image segmentation. Background Art

[0002] Magnetic resonance imaging (MRI) is non-invasive, high-resolution, and non-invasive. It can provide detailed information about the structure and function of human tissues and is often used to diagnose various diseases and lesions. In recent years, with the emergence of convolutional neural networks (CNN) and the rapid development of computer hardware, deep learning has been widely used in MRI image segmentation tasks. MRI image segmentation based on deep learning can accurately and quickly segment the lesion area, overcoming the shortcomings of strong subjectivity of manual delineation, high missed diagnosis and misdiagnosis rates, poor repeatability, and low efficiency. It can effectively assist doctors in diagnosing diseases and has become an important means of AI medical treatment in recent years. In clinical practice, by adjusting the scanning parameters and pulse sequences, MRI images of different modalities can be obtained, which can provide comprehensive and detailed information. Therefore, multimodal MRI image segmentation methods are the development trend of medical image segmentation. At present, MRI scan images include four modalities: Flair, T1ce, T1, and T2.

[0003] However, in clinical practice, it is not always possible to obtain complete MRI modalities. When some modalities do not exist, the segmentation effect will be greatly affected. In response to this problem, many researchers have proposed corresponding solutions, which can be mainly divided into three categories. The first category is the method based on image synthesis, which first generates the missing modality and then performs the subsequent segmentation task. This method is more complex, and the accuracy of modality generation has a greater impact on the segmentation result; the second category is the method based on knowledge distillation, which uses full modality combination and incomplete modality combination to train the network, where the network under the full modality combination is used to guide the network training under the incomplete modality combination, but it is very difficult to find a suitable guidance method; the third category is the method based on potential feature space analysis, which obtains the potential correlation features between different modalities, fuses them in a certain way, and then segments the target area according to the fused features. This method is less complex, and how to effectively restore the information of the missing modality is its main difficulty. These methods can reduce the impact of the missing modality on the network segmentation performance to a certain extent, but there are still many shortcomings and need to be further improved.

[0004] Therefore, providing an MRI image segmentation method that can effectively cope with different modality combinations is a problem that those skilled in the art still need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a modality-adaptive MRI image segmentation method for coping with the MRI image segmentation problem under different modality combinations, so that image segmentation can be effectively performed under various MRI modality combinations.

[0006] To achieve the above object, the solution of the present invention includes:

[0007] A modality-adaptive MRI image segmentation method of the present invention uses a trained segmentation network to segment a multi-modal MRI image and output fusion prediction results of different modalities; the segmentation network to be trained includes a feature extraction part independent of different MRI modalities, a correlation modeling part between different modalities for fusing modality features of different modalities obtained by the feature extraction part, a correlation feature enhancement part for performing correlation feature enhancement on the modality correlation feature representation output by the correlation modeling part, a fused modality target region segmentation part for performing target region segmentation on the fused modality after the modality correlation feature enhancement, and an independent modality target region segmentation part for performing feature recovery on the modality features of different modalities obtained by the feature extraction part to obtain independent segmentation results of different modalities; the independent modality target region segmentation part is used to assist in training to prevent unbalanced training of the model under the missing modality.

[0008] Furthermore, the feature extraction part is composed of CNN; the correlation modeling part is implemented by cross attention, the correlation feature enhancement part is implemented by a multi-head self-attention mechanism, and the fusion modality target area segmentation part and the independent modality target area segmentation part are both composed of CNN.

[0009] Furthermore, the training of the segmentation network includes: inputting the 3D multimodal MRI data set into the segmentation network to be trained, performing layer-by-layer feature extraction and feature recovery, iteratively training the segmentation network, and obtaining a network model with optimal weight parameters as a trained segmentation network.

[0010] Furthermore, during the training process, the feature extraction part independently extracts features of different modalities, and then marks the extracted modal features as missing to simulate various modal combinations, and fuses all modal features together through the correlation modeling part to obtain a modal correlation feature representation, which is then input into the fused modality target region segmentation part after passing through the correlation feature enhancement part to obtain the final fused modality segmentation result; the independent modality target region segmentation part performs feature recovery on the modal features that are not marked as missing, and obtains independent independent modality segmentation results of different modalities; for the fused modality segmentation result and the independent modality segmentation result, the difference between the ground truth and the ground truth is calculated and back-propagated to force the encoder of each modality to learn the identification features of the target area, and the encoder includes a feature extraction part, a correlation modeling part and a correlation feature enhancement part.

[0011] Furthermore, the difference between the ground truth and the segmentation loss is measured using Dice loss and weighted cross entropy loss, and the optimal weight parameters are gradually obtained in the process of minimizing the segmentation loss.

[0012] Furthermore, in each iteration, the segmentation loss of the training set and the validation set is calculated to determine whether overfitting occurs during the network training process. If overfitting occurs, the network training is stopped and the optimal parameters are saved. The overfitting judgment condition is: the segmentation loss calculated by the training set decreases, and the segmentation loss calculated by the validation set increases.

[0013] Furthermore, the 3D multimodal MRI dataset is a preprocessed dataset, and the preprocessing includes resampling, removing background areas, random cropping, normalization, and data enhancement.

[0014] Furthermore, the images in the 3D multimodal MRI dataset include Flair, T1ce, T1, and T2 modalities.

[0015] Furthermore, during the training process, the Adam optimizer is used to optimize the network, and the learning rate decay method is:

[0016]

[0017] Among them, max_epoch is the iteration period, and epoch is the number of cycles of the current iteration.

[0018] Furthermore, the training of the segmentation network also includes a network testing process. During the network testing process, the segmentation result is obtained by combining the optimal parameters retained by the network, and the overlap coefficient and Hausdorff distance are used to measure the similarity between the segmentation result of the network and the ground truth to evaluate the segmentation performance of the network.

[0019] The beneficial effects of the present invention are:

[0020] The present invention designs a new modality-adaptive deep learning network, which learns the correlation between different modalities under all modality combinations. Therefore, when some modalities do not exist, the missing modal information can be restored to a certain extent according to the correlation dependency established by the model, thereby achieving modality adaptation to ensure the accuracy of the segmentation results under various modality combinations.

[0021] The present invention further models the correlation between modalities through cross-attention, so that the missing modality can obtain the corresponding feature representation according to the correlation, and obtain the missing modal features to a certain extent, which can effectively prevent the reduction of segmentation accuracy when the modality is missing, and the complexity is not high. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a modality-adaptive MRI image segmentation method of the present invention;

[0023] Figure 2 It is a schematic diagram of a segmentation network model of a modality-adaptive MRI image segmentation method of the present invention;

[0024] Figure 3 It is a schematic diagram of a modality correlation learning module of a modality-adaptive MRI image segmentation method of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail in a clear and complete manner in conjunction with the accompanying drawings and embodiments.

[0026] Modality-adaptive MRI image segmentation method embodiment:

[0027] This embodiment discloses a modality-adaptive MRI image segmentation method, such as Figure 1 As shown, the specific steps include:

[0028] Step (1): Obtain a 3D multimodal MRI dataset and preprocess it;

[0029] In this embodiment, the data preprocessing operation includes: resampling the original image to 1mm 3 The background is removed, the image is normalized and randomly cropped to 80×80×80 pixels, and data enhancement is performed using intensity shift, mirror flip, and random rotation operations.

[0030] The series of preprocessing operations in this embodiment can improve the quality and analyzability of image data, and help the subsequent segmentation work to proceed smoothly.

[0031] Step (2): Divide the preprocessed 3D multimodal MRI dataset into a training set, a validation set, and a test set;

[0032] In this embodiment, samples are randomly selected from the preprocessed data set in a ratio of approximately 6:1:2 as a training set, a validation set, and a test set, respectively, to facilitate optimal training and accurate evaluation of the network.

[0033] Step (3): input the training set and validation set samples into the segmentation network, perform layer-by-layer feature extraction and feature recovery, and obtain the segmentation result and verification result predicted by the network; the segmentation network includes different parts;

[0034] like Figure 2 As shown in FIG. 1 , in this embodiment, the segmentation network adopts an encoding-decoding structure, and the network input is images of four modalities (Input Images), including Flair, T1ce, T1, and T2 modalities. The encoder and decoder have four layers (E1, E2, E3, E4 and D1, D2, D3, D4), wherein the encoder part includes independent feature extraction parts for different MRI modalities and also includes a correlation learning part; the feature extraction part is composed of CNN, including feature extraction layers of different depths of E1, E2, E3, and E4, and the correlation learning part (Correlation Learning Module) is as shown in FIG. Figure 3 As shown in FIG. 1 , the module includes a correlation modeling module and a correlation feature enhancement module between different modalities. The correlation modeling module is implemented by multi-head cross attention, and the correlation feature enhancement module is implemented by multi-head self-attention. The output of the E3 layer is divided into two paths, one path entering the correlation learning part after the mask operation and the other path entering E4. Correspondingly, the decoder is divided into a decoder for fusion modality target region segmentation and a decoder for independent modality target region segmentation. Both of them include four layers, D4, D3, D2, and D1, and are composed of CNN. The decoder for fusion modality target region segmentation receives the modality correlation feature representation obtained by the correlation learning part and outputs the fusion prediction result (Fused Predict Value). The decoder for independent modality target region segmentation plays the role of auxiliary training, the purpose of which is to prevent the unbalanced training of the model under the missing modality. It receives the encoding features obtained by the E4 layer and outputs the segmentation result of the independent modality (Independent Predict Value).

[0035] The implementation of the correlation modeling part is as follows:

[0036]

[0037] F corr =FFN(LN(F′))+F′

[0038] Among them, n and i represent the current mode, q n , k i 、v i represents the linear transformation value of the corresponding mode, N represents the mode set, specifically {Flair, T1ce, T1, T2}, n,i∈N, represents multiplication, represents the correlation feature representation of mode n, C represents feature concatenation, F represents the concatenated features of the four modes before correlation modeling, F′ is an intermediate variable, FFN and LN represent the feedforward layer and layer normalization respectively, and F corr Represents the modeled modal correlation feature representation.

[0039] The implementation of the correlation feature enhancement part is as follows:

[0040] F″=MHA(LN(F corr ))+F corr

[0041] F en =FFN(LN(F″))+F″

[0042] Among them, MHA stands for multi-head self-attention, F″ stands for intermediate variable, and F en It is the enhanced modal correlation feature representation.

[0043] Further, in this embodiment, for the preprocessed input image, such as Figure 2As shown in the figure, the features of different modalities are first extracted independently, and then they are marked as missing to simulate various modal combinations. All modal features (including missing modal features) are fused together through the correlation modeling part to obtain the modal correlation feature representation, which is then input into the decoder through the correlation feature enhancement part to obtain the final segmentation result (Fused Predict Value). The independent modal target area segmentation part decodes the modal features that are not marked as missing (feature recovery) respectively to obtain independent segmentation results of different modalities (Independent Predict Value). The difference between the predicted segmentation result and the ground truth (Ground Truth) is calculated and back-propagated to force the encoder of each modality to learn the identification features of the target area. The encoder includes a feature extraction part, a correlation modeling part and a correlation feature enhancement part.

[0044] The advantages of this embodiment are: the "U"-shaped network with an encoder-decoder structure has a larger receptive field, can effectively obtain medical image features, and obtain good segmentation results; the use of cross attention can learn the correlation between different modalities, and the use of a correlation feature enhancement module based on self-attention can further strengthen the correlation between different modalities and reduce the impact of missing modalities on correlation learning; the independent modality target area segmentation part can prevent unbalanced training of the network when the modality is missing.

[0045] Step (4): Iteratively train the segmentation network to obtain a network model with optimal weight parameters;

[0046] In this embodiment, Adam is used for optimization during network training, and the weight decay is 1×10 -5 , the initial learning rate is 2×10 -4 , and decays according to the following formula:

[0047]

[0048] Among them, epoch is the number of cycles of the current iteration, the iteration cycle max_epoch is 1200, and the batch size is set to 1.

[0049] Furthermore, if Figure 1 , 2As shown in the figure, during the network training process, the segmentation loss and the verification loss are calculated to monitor whether the network training is overfitting. If overfitting occurs or the maximum number of cycles is reached, the training is terminated. Otherwise, the network training continues, and the network parameters are updated by the calculated training loss and, including the segmentation loss of each layer of the network, the segmentation loss of the fused modality target area, and the segmentation loss of the independent modality target area. The segmentation loss of each layer of the network calculates the difference between the segmentation result of each layer of the network and the ground truth, the segmentation loss of the fused modality target area calculates the difference between the final segmentation result and the ground truth, and the segmentation loss of the independent modality target area calculates the segmentation loss and of each modality. The loss function used is a combination of Dice loss and weighted cross entropy loss, which is specifically defined as follows:

[0050]

[0051] L pre =L DL (p,g)+L WCE (p,g)

[0052]

[0053] L=L lay +L pre +L ind

[0054] Among them, i represents the i-th layer of the network, p i represents the segmentation result of the i-th layer, g represents the ground truth, p represents the final segmentation result, m represents the modality, M is the modality set, specifically {Flair, T1ce, T1, T2}, p m represents the segmentation result of mode m, L lay represents the segmentation loss of each layer of the network, L pre represents the final segmentation loss of the network, L ind represents the segmentation loss of each modality, and L is the sum of the losses during training;

[0055] The specific definition of the Dice loss is as follows:

[0056]

[0057] where p r and g r denote the predicted value and ground truth value of region r, ∩ represents the coverage ratio between them, L1 represents L1 regularization, R num Indicates the number of regions;

[0058] The specific definition of the weighted cross entropy loss is as follows:

[0059]

[0060] Where H, W, and Z represent the length, width, and height of the 3D volume, respectively. r represents the weight parameter of region r, which is defined as:

[0061]

[0062] The advantages of this embodiment are: calculating the segmentation loss at each layer of the network can better monitor the segmentation effect of the network; calculating the segmentation loss for each modality separately and back-propagating it can prevent the problem of unbalanced training of the network when the modality is incomplete; using Dice loss and weighted cross entropy loss can well guide network training and alleviate the problem of voxel category imbalance to a certain extent.

[0063] Step (5): Input the test set into the network model with the optimal weight parameters, obtain the segmentation results and evaluate them.

[0064] In this embodiment, if Figure 1 As shown, the test process first obtains the segmentation result by combining the optimal parameters retained by the network, and then uses the degree of coincidence coefficient (DSC) and Hausdorff distance (HD) to evaluate the test results of the network model;

[0065] The coincidence coefficient is defined as follows:

[0066]

[0067] Among them, y and g represent the segmentation result and the ground truth, respectively. The maximum value of DSC(y,g) is 1. A larger score indicates a higher similarity between the segmentation result and the ground truth.

[0068] The expression of the Hausdorff distance is as follows:

[0069]

[0070] in Represents y k The elements in y′ and g′ represent the elements in the segmentation result y and the ground truth g, d represents the Euclidean distance, and the smaller HD(y,g) is, the closer the two are, that is, the more accurate the segmentation result is.

[0071] This embodiment can evaluate the accuracy of the obtained segmentation results, and provide data support for the evaluation of the invented segmentation network.

[0072] The modality-adaptive MRI image segmentation method of this embodiment retains the information of the missing modality by modeling the correlation between different MRI modalities, can cope with image segmentation under various modality combinations, realize modality adaptation, and ensure the effectiveness of MRI image segmentation under different modality combinations; before training the network, the image data quality and analyzability are improved by resampling, background removal, normalization and data enhancement operations, and the optimal training and accurate evaluation of the network are helped by dividing the data set; during the network training process, the designed modality correlation learning module effectively models the correlation between different modalities through cross-attention, the modality correlation enhancement module strengthens the useful correlation feature representation, the modality-specific decoder obtains the segmentation results of different modalities and calculates the segmentation loss, which can alleviate the unbalanced training in the case of incomplete modalities, and by using Adam and the learning rate decay strategy, the efficiency of network training can be improved, and the overfitting of the network can be effectively monitored by verifying the network with the validation set; when testing the network, the segmentation performance of the network can be effectively evaluated by using the overlap coefficient and the Hausdorff distance.

[0073] The method of the present invention extracts the features of different MRI modalities independently, and models the correlation between the features of different modalities through a cross-attention mechanism; at the same time, the modeled modality correlation feature representation is strengthened through a multi-head self-attention mechanism to highlight useful feature information; in addition, a modality-specific decoder is used to obtain the segmentation results of different modalities respectively, so that the encoder of each modality can learn the identification features of the target area according to its segmentation loss during the feature extraction process, avoiding the problem of unbalanced training of the network when the modality is incomplete. The method proposed by the present invention enables the network to achieve modality adaptation, so that effective image segmentation can be performed under any MRI modality combination, solving the problem of decreased image segmentation accuracy in the case of incomplete modalities, and no image synthesis is required.

Claims

1. A modality-adaptive MRI image segmentation method, characterized in that: The multimodal MRI image is segmented using a trained segmentation network to output fusion prediction results of different modalities; the segmentation network to be trained includes a feature extraction part independent of different MRI modalities, a correlation modeling part between different modalities for fusing modal features of different modalities obtained by the feature extraction part, a correlation feature enhancement part for performing correlation feature enhancement on the modal correlation feature representation output by the correlation modeling part, a fused modality target region segmentation part for performing target region segmentation on the fused modality after the modality correlation feature enhancement, and an independent modality target region segmentation part for performing feature recovery on the modal features of different modalities obtained by the feature extraction part to obtain independent segmentation results of different modalities; The independent modality target region segmentation portion is used to assist training to prevent unbalanced training of the model in the missing modality.

2. The modality-adaptive MRI image segmentation method according to claim 1, characterized in that: The feature extraction part is composed of CNN; the correlation modeling part is realized by cross attention, the correlation feature enhancement part is realized by multi-head self-attention mechanism, and the fusion modality target area segmentation part and the independent modality target area segmentation part are both composed of CNN.

3. The modality-adaptive MRI image segmentation method according to claim 1, characterized in that: The training of the segmentation network includes: inputting the 3D multimodal MRI data set into the segmentation network to be trained, performing layer-by-layer feature extraction and feature recovery, iteratively training the segmentation network, and obtaining a network model with optimal weight parameters as a trained segmentation network.

4. The modality-adaptive MRI image segmentation method according to claim 3, characterized in that: During the training process, the feature extraction part independently extracts features of different modalities, and then marks the extracted modal features as missing to simulate various modal combinations. All modal features are fused together through the correlation modeling part to obtain a modal correlation feature representation, which is then input into the fused modality target region segmentation part after passing through the correlation feature enhancement part to obtain the final fused modality segmentation result; the independent modality target region segmentation part performs feature recovery on the modal features that are not marked as missing respectively to obtain independent independent modality segmentation results of different modalities; for the fused modality segmentation result and the independent modality segmentation result, the difference between the ground truth and the ground truth is calculated and back-propagated to force the encoder of each modality to learn the identification features of the target area, and the encoder includes a feature extraction part, a correlation modeling part and a correlation feature enhancement part.

5. The modality-adaptive MRI image segmentation method according to claim 4, characterized in that: The difference between the ground truth and the segmentation loss is measured using Dice loss and weighted cross entropy loss, and the optimal weight parameters are gradually obtained in the process of minimizing the segmentation loss.

6. The modality-adaptive MRI image segmentation method according to claim 5, characterized in that: In each iteration, the segmentation loss of the training set and the validation set is calculated to determine whether overfitting occurs during the network training process. If overfitting occurs, the network training is stopped and the optimal parameters are saved. The overfitting judgment condition is: the segmentation loss calculated by the training set decreases, and the segmentation loss calculated by the validation set increases.

7. The modality-adaptive MRI image segmentation method according to claim 3, characterized in that: The 3D multimodal MRI dataset is a preprocessed dataset, wherein the preprocessing includes resampling, background area removal, random cropping, normalization, and data enhancement.

8. The modality-adaptive MRI image segmentation method according to claim 3, characterized in that: The images in the 3D multimodal MRI dataset include Flair, T1ce, T1, and T2 modalities.

9. The modality-adaptive MRI image segmentation method according to claim 3, characterized in that: During the training process, the Adam optimizer is used to optimize the network, and the learning rate lr is attenuated as follows: Among them, max_epoch is the iteration period, and epoch is the number of cycles of the current iteration.

10. The modality-adaptive MRI image segmentation method according to claim 3, characterized in that: The training of the segmentation network also includes a network testing process. During the network testing process, the segmentation result is obtained by combining the optimal parameters retained by the network, and the coincidence coefficient and Hausdorff distance are used to measure the similarity between the segmentation result of the network and the ground truth to evaluate the segmentation performance of the network.

Images