A multi-lesion segmentation method for multi-modal MRI

Abstract

The present invention discloses a multi-modal MRI multi-lesion segmentation method, which realizes the purpose of simultaneous and accurate segmentation of multi-modal MRI multi-lesions. The method mainly includes four stages, a multi-modal image feature extraction stage, a multi-modal feature fusion stage, a multi-modal feature screening stage, and a multi-category segmentation stage. In the multi-modal image feature extraction stage, image features of different modalities are extracted separately. In the multi-modal feature fusion stage, spatial attention and modal attention mechanisms are used to deeply fuse the extracted image features of different modalities in two dimensions. In the multi-modal feature screening stage, a modality-sensitive gating mechanism is used to select multi-modal features, screen more effective features, and improve feature expression capabilities. In the multi-category segmentation stage, the multi-modal fusion features of different stages are subjected to upsampling operations at different stages through a decoder to restore the image size, and the screened multi-modal features are fused, and finally predicted through the output branch to obtain the segmentation result.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a multi-modal MRI multi-lesion segmentation method. Background Art

[0002] Multi-lesion segmentation study of multimodal MRI: Brain tumors are one of the most common types of cancer in the world, among which gliomas are the most common malignant brain tumors with different invasiveness, and meningiomas are the most common primary intracranial tumors in adults. Specifically, for gliomas, commonly used MRI sequences include T1-weighted (T1), enhanced T1-weighted (T1Gd), T2-weighted (T2), and T2 fluid-attenuated inversion recovery (T2-FLAIR) images; they play different roles in distinguishing tumors, peritumoral edema, and tumor cores. Clinically, meningiomas have different characteristic manifestations on contrast-enhanced T1-weighted (T1Gd) and contrast-enhanced T2-FLAIR (FLAIR-C for short) MRI images.

[0003] Clinically, doctors judge the severity of the disease by outlining the location of different brain lesions, and then adopt different treatment methods; however, the analysis, diagnosis and outlining of brain lesions is extremely time-consuming and laborious, so it is particularly important to automatically segment the brain lesion area and analyze the volume through computers. For other organs, it is also important to automatically judge and segment lesions through computers.

[0004] Research on medical image segmentation technology: Medical image segmentation technology has shown great potential in the field of medical images. In recent years, various excellent medical image algorithms have emerged one after another. The core lies in how to build a better network structure to extract and fuse multimodal medical image features to achieve better segmentation effects. The more popular method is the "U"-shaped network structure, that is, the 3D medical image is first gradually downsampled by the convolutional neural network encoder to extract image features of different scales, and then the extracted image features are upsampled by the convolutional neural network decoder to restore the image size, and finally the segmentation result is obtained through the output branch. In recent years, the Transformer structure has gradually become popular in the field of natural language processing due to its excellent global feature modeling ability, but it is currently less used in the field of medical image segmentation. The more mature work includes using convolutional neural networks as encoders and using Transformer structures to model global features; Transformer structures are used as encoders to directly extract global information of multimodal images, and both have achieved relatively ideal results. Summary of the invention

[0005] In order to solve the problems in the prior art, the present invention provides a multi-modal MRI multi-lesion segmentation method to solve the problem that the prior art cannot well fuse multi-modal information and use the supplementary information between multi-modalities. The present invention extracts different modal features through a multi-encoder to avoid the mixing of modal images at the input end, fuses multi-modal features through two dimensions of space and modality, and screens multi-modal features through a modality-sensitive gating network, thereby enhancing modality-sensitive features and improving the expressiveness of features.

[0006] Technical solution of the present invention:

[0007] A multi-modal MRI multi-lesion segmentation method, fusion and screening of multi-modal features, comprising the following steps:

[0008] Step 1: Multimodal MRI extraction: multi-stage feature extraction of brain MRI images of different modalities is performed through a deep neural network with the same structure but independent parameters;

[0009] Step 2: feature fusion: the multimodal features obtained in step 1 are fused in two dimensions: spatial dimension and modal dimension. In the spatial dimension, a multi-directional self-attention mechanism is used to obtain spatial relationships to fuse spatial features and improve computational efficiency. In the modal direction, a cross-attention mechanism is used to extract modal relationships and fuse multimodal features.

[0010] Step 3: Feature screening: Use the fusion features obtained in step 2 to construct a modality-sensitive feature selection gate structure, screen the multimodal features at different stages of the upper layer, select more effective features to input into the decoder, and improve the feature expression capability;

[0011] Step 4: multi-category segmentation; use the decoder to perform upsampling operations on the deep fusion features at different stages to restore the image size, and combine it with the multimodal enhanced features after the feature screening module to improve the feature encoding capability. Finally, the fusion features obtained by the decoder are predicted through the output branch to obtain the segmentation result.

[0012] In the step 1 of multimodal MRI extraction, the commonly used Poolformer feature extraction network is improved, and a global information module (GlobalPoolformer) is designed to incorporate global information into the MRI encoding process, perform multiple feature extraction and downsampling operations on brain MRI of different modalities, and extract deep features of different scales.

[0013] For the feature fusion in step 2, we designed the NMaFA (Nested Modality-aware Feature Aggregation) fusion module, which uses the attention mechanism of spatial dimension and modality dimension to fuse the multimodal features extracted by the encoder in a nested manner to obtain the fused feature , the specific methods include:

[0014] (1) In the spatial dimension, the Transformer structure (T_tsa) with the self-attention mechanism is used for feature fusion. The features of multiple modalities extracted by the encoder are concatenated in the channel dimension and encoded into multiple block features through the block embedding layer. The attention between each feature block is calculated in the spatial dimension through the self-attention mechanism, and weighted fusion is performed to obtain the fused features in the spatial dimension.

[0015] (2) In the modality dimension, the Transformer structure (T_cma) with cross modality-aware attention is used for feature fusion. The features of multiple modalities extracted by the encoder are encoded into block features of multiple modalities through the block embedding layer. The fused features obtained in (1) are used to perform cross-attention operations with the encoded block features of different modalities. The attention between the fused feature block and the blocks of different modalities is calculated in the modality dimension, and weighted fusion is performed to obtain the fused features in the modality dimension.

[0016] In the feature selection of step 3, a MSG feature selection module is designed to use high-level fusion features to screen low-level multimodal features of different scales, select more effective features to input into the decoder, and improve the reuse effect of features. The specific methods include:

[0017] (1) Fusion features Dimensionality reduction to the number of modalities through convolutional layers with learnable parameters;

[0018] (2) A modality-sensitive feature selection gate is constructed through an activation function, and multimodal features are selected proportionally for output. Beneficial Effects

[0019] Compared with the prior art, the multimodal MRI multi-lesion segmentation method proposed in the present invention has the following advantages:

[0020] 1) An effective encoder GlobalPoolformer is proposed, which incorporates global information and can more effectively extract features of images of different modalities.

[0021] 2) A feature fusion module based on multi-dimensional attention is proposed, which can more effectively fuse the deep image features extracted by the encoder.

[0022] 3) A modality-sensitive gated filtering module is proposed, which can select features of different modalities and improve the expressiveness of features.

[0023] 4) The experimental results verify the effectiveness of the present invention. By comparing with the currently popular multimodal segmentation methods, the effectiveness of the proposed method for multimodal MRI multi-lesion segmentation is verified. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a model flow chart of a multi-modal MRI multi-lesion segmentation method of the present invention; wherein GlobalPoolformerEncoder represents an improved encoder that incorporates global information to extract features of images of different modalities; NMaFA represents the proposed multi-modal feature fusion module; T_tsa represents the attention mechanism in the spatial direction, and T_cma represents the attention mechanism in the modality direction; MSG represents a modality-sensitive gating module; Decoder represents a decoder that upsamples and fuses image features at different stages and finally decodes them into a segmentation result;

[0025] Figure 2 Multimodal fusion flow chart of multimodal MRI multi-lesion segmentation task;

[0026] Figure 3 Multimodal fusion module (spatial dimension attention and modality dimension attention) for multimodal MRI multi-lesion segmentation task. DETAILED DESCRIPTION

[0027] The technical solution of the present invention is further described in conjunction with the accompanying drawings.

[0028] The multi-modal MRI multi-lesion segmentation method of the present invention, the fusion and screening of multi-modal features, comprises the following steps:

[0029] 1. Multimodal brain MRI feature extraction

[0030] Different modalities of brain MRI contain significant information of different lesions. The present invention can fuse different numbers of modal features for different brain tumor data. For example, BraTS glioma data contains four modalities: T1-weighted (T1), post-contrast T1-weighted (T1Gd), T2-weighted (T2) and T2Fluid Attenuation Inversion Recovery (T2-FLAIR); Huanhu Hospital meningioma data contains two modalities: post-constrast T1-weighted (T1Gd) and contrast-enhanced T2-FLAIR (abbreviated as FLAIR-C); both data are 3D data. At the same time, the present invention is also applicable to other organs.

[0031] The extraction process of multimodal MRI is as follows:

[0032] (1) 3D-MRI (D*W*H) of different modalities are gradually downsampled and feature extracted through N (N is the number of modalities) neural network models (GlobalPoolformer Encoder) with the same structure but independent parameters to obtain N Feature map of size;

[0033] (2) The image features of different scales obtained in the downsampling stage are saved and passed to the decoder for feature reuse.

[0034] (II) Feature Fusion

[0035] The NMaFA (Nested Modality-aware Feature Aggregation) fusion module is used to fuse the deep image features obtained in (I) to obtain fused features. The fused features are input into the decoder for decoding, and the modality-sensitive gating module is used to select multi-modal features of different scales. The specific process is as follows:

[0036] (1) The multimodal features obtained in (I) are fused in the spatial dimension and the modal dimension. The spatial dimension uses the self-attention mechanism to extract the spatial relationship of the multimodal features. In order to improve the computational efficiency of the model, the global attention is decomposed into three sub-self-attention modules: horizontal, vertical, and window, to fuse the spatial features. The horizontal attention module only focuses on the attention relationship within each slice in the 3D-MRI. In order to improve the global modeling capability, the vertical attention module focuses on the attention relationship between slices in the 3D-MRI. The window attention module divides the 3D-MRI into different windows and focuses on the attention relationship within the window. Using three sub-attention modules, the global attention relationship is approximately modeled.

[0037] The formula of the multi-directional self-attention mechanism is as follows:

[0038]

[0039] in , , They represent vertical, horizontal, and window attention modules respectively. It is a multimodal feature.

[0040] (2) In the modal dimension, the cross-attention mechanism is used to extract the feature relationship of different modalities and fuse the multimodal features in a fine-grained manner. First, the query vector (query), key vector (key), and value vector (value) required by the cross-attention mechanism are calculated using the fused features obtained in (1) and the multimodal features. The calculation formula is as follows:

[0041]

[0042] in is the feature fusion result of the direction dimension in (1), is a multimodal feature; the two vectors are respectively related to the corresponding learnable parameter matrix Do matrix operations to get the required calculation attention vector.

[0043] The formula of the modal cross-attention mechanism is as follows:

[0044]

[0045] in It is the result of multi-directional self-attention fusion; It is a multimodal feature; They are query vector, key vector, and value vector, respectively, which are used to calculate the cross-attention fusion results. is the attention weight calculation function, is the size of the feature, and the fusion feature is finally obtained by fusion of spatial self-attention feature and modal cross-attention feature. , the size is .

[0046] (III) Feature screening

[0047] The fusion features obtained from (ii) Contains rich information of different modal images. Traditional methods usually input all the features of different modalities from the encoder to the decoder, ignoring the importance relationship of the features. This invention proposes a gated screening module based on fusion features, which improves the reuse effect of modal features by selecting features of different scales in different modalities. The feature screening process is as follows:

[0048] 1) Construct gating mechanisms MSG of different scales by fusing features, reduce the feature dimension through the fully connected layer network to the size of the modal number, and then perform upsampling of different times through the upsampling network to adapt to features of different scales. The formula is as follows:

[0049]

[0050] in For different times of upsampling operations, It is a fully connected layer network; is the sigmoid activation function; is the fusion feature after NMaFA module;

[0051] (2) Use the gating mechanism to select multimodal features of different scales, assign different weights to the features of each pixel in each modality, select important features, and filter out unimportant features. The formula is as follows:

[0052]

[0053] in is the feature gating matrix obtained from the deep representation in (1), which indicates the importance of different modalities. It is a multi-modal feature of different scales. Indicates scale, Indicates modal.

[0054] (IV) Multimodal Segmentation Task

[0055] Construct the output branch, use the multimodal fusion features decoded by the decoder as input, output the segmentation result, use the cross entropy function as the segmentation loss function, calculate the loss, optimize the network parameters, and train the model. We trained the model for 300 epochs with a learning rate of 1e-4. In order to improve the robustness of the model, we used a variety of data enhancement methods, such as random rotation, random mirror flipping, random Gaussian noise, random contrast enhancement, random brightness enhancement, random scale enhancement, gamma transformation, etc.

[0056] In order to improve the reliability of the test results, we conducted comparative experiments on the public multimodal medical image segmentation dataset (BraTS2020) and the internal multimodal medical image segmentation dataset (MeniSeg). The BraTS2020 dataset is a glioma dataset. Each data contains 4 modalities, and a total of 364 labeled 3D brain MRIs. We use 315 randomly selected cases as training sets, 15 cases as validation sets, and 34 cases as test sets; the MeniSeg dataset is a meningioma dataset. Each data contains 2 modalities, and a total of 110 labeled 3D brain MRIs. We use a two-fold cross-validation method for experiments. The experimental results show that our proposed method has obvious advantages over other comparative methods.

[0057] The results of the present invention and the comparison results with other multimodal segmentation methods are shown in Tables 1 and 2.

[0058] Table 1 Comparison results of the present invention and other multimodal segmentation methods (BraTS2020 dataset)

[0059]

[0060] Table 2 Comparison results of the present invention and other multimodal segmentation methods (MeniSeg dataset)

[0061]

Claims

1. A multi-modal MRI multi-lesion segmentation method, characterized in that The fusion and screening of multimodal features includes the following steps: Step 1: Multimodal MRI extraction: multi-stage feature extraction of brain MRI images of different modalities is performed through a deep neural network with the same structure but independent parameters; Step 2: feature fusion: the multimodal features obtained in step 1 are fused in two dimensions: spatial dimension and modal dimension. In the spatial dimension, a multi-directional self-attention mechanism is used to obtain spatial relationships to fuse spatial features and improve computational efficiency. In the modal dimension, a cross-attention mechanism is used to extract modal relationships and fuse multimodal features. Step 3: Feature screening: Use the fusion features obtained in step 2 to construct a modality-sensitive feature selection gate structure, screen the multimodal features at different stages of the upper layer, select more effective features to input into the decoder, and improve the feature expression capability; Step 4: multi-category segmentation: use the decoder to perform upsampling operations on the deep fusion features at different stages to restore the image size, and combine the multi-modal enhanced features after the feature screening module to improve the feature encoding capability. Finally, the fusion features obtained by the decoder are predicted through the output branch to obtain the segmentation result. In the step 1, multimodal MRI extraction, the neural network encoder GlobalPoolformer is improved, global information is incorporated into the MRI encoding process, and multiple feature extraction and downsampling operations are performed on brain MRIs of different modalities to extract deep features of different scales; The feature fusion in step 2 is to fuse the multimodal features extracted by the encoder through the attention mechanism of spatial dimension and modal dimension to obtain the fused feature , the specific methods include: (1) In the spatial dimension, the features of multiple modalities extracted by the encoder are concatenated in the channel dimension and encoded into multiple block features through the block embedding layer. The self-attention mechanism is used to calculate the attention between each feature block in the spatial dimension, and weighted fusion is performed to obtain the fused features in the spatial dimension. (2) In the modal dimension, the features of multiple modalities extracted by the encoder are respectively encoded into block features of multiple modalities through the block embedding layer. The fused features obtained in (1) are used to perform cross-attention operations with the block features of different modalities after encoding. The attention between the fused feature block and the blocks of different modalities is calculated in the modal dimension, and weighted fusion is performed to obtain the fused features of the modal dimension.

2. The multimodal MRI multi-lesion segmentation method according to claim 1, characterized in that: The feature screening in step 3 uses high-level fusion features to screen low-level multimodal features of different scales, selects more effective features to input into the decoder, and improves the reuse effect of features. The specific method includes: (1) Fusion features Dimensionality reduction to the number of modalities through convolutional layers with learnable parameters; (2) A modality-sensitive feature selection gate is constructed through an activation function, and multimodal features are selected proportionally for output.

Images