Multi-mode MRI brain tumor image segmentation method

Abstract

The invention discloses a multi-modal MRI (Magnetic Resonance Imaging) brain tumor image segmentation method. The method comprises the following steps: data preprocessing and feature extraction: collecting and processing image data from different MRI sequences; adaptive feature selection and alignment: dynamically selecting the most representative modal features for alignment according to the information of the brain tumor; multi-scale feature fusion enhancement: introducing an extra network layer to perform feature fusion, the network layer being responsible for learning a weight distribution strategy, dynamically adjusting a fusion weight and strategy, inputting a fused feature map into a segmentation network, performing preliminary segmentation, and on the basis of the preliminary segmentation, performing multi-scale feature fusion enhancement; introducing an attention mechanism to guide the segmentation network to pay attention to the tumor region; reinforcement learning optimization: adopting a reinforcement learning algorithm to optimize the parameters and the structure of the segmentation network; uncertain region optimization: further optimizing the uncertain region in the segmentation result by adopting a post-processing method; and obtaining a three-dimensional tumor segmentation result: outputting the three-dimensional tumor segmentation result.

Description

Technical Field

[0001] The present invention relates to the technical field of brain tumors, and in particular, to a method for segmenting multi-modal MRI brain tumor images. Background Art

[0002] Brain tumors are abnormal growths of brain cells in the brain and are considered a life-threatening disease. Relevant data shows that brain tumors account for more than 85% of all primary central nervous system tumors globally and approximately 2% - 3% of cancer-related deaths, posing a huge threat to human health. Therefore, early diagnosis and treatment of brain tumors are particularly important. MRI (Magnetic Resonance Imaging) is a non-invasive imaging technique that can clearly display soft tissue lesions and is widely used in the diagnosis and treatment of brain tumor diseases. Using different imaging sequences to obtain MRI images of the same tissue from different angles or in different forms is usually referred to as multi-modal MRI images. Different modalities of MRI images can reflect different information of the tumor region, and multi-modal MRI images are used to accurately segment the lesion region.

[0003] However, the multi-modal MRI image dataset has different resolutions, noise levels, and contrasts, making the segmentation task complex and variable. When using threshold segmentation or region-based segmentation to process complex and variable brain tumor images, it is often unable to capture the fine structure of the tumor, resulting in inaccurate or insufficiently robust segmentation results. Therefore, a method for segmenting multi-modal MRI brain tumor images is proposed. Summary of the Invention

[0004] The purpose of the present invention is to solve the deficiencies existing in the prior art, and a method for segmenting multi-modal MRI brain tumor images is proposed.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions:

[0006] A method for segmenting multi-modal MRI brain tumor images includes the following steps:

[0007] Data preprocessing and feature extraction: Collect and process image data from different MRI sequences (such as T1, T2, FLAIR, etc.), preprocess the MRI image data, including steps such as denoising and enhancing contrast to improve the image quality, and use a convolutional neural network to extract multi-modal and multi-scale feature maps;

[0008] Adaptive feature selection and alignment: According to information such as the type, size, and location of the brain tumor, dynamically select the most representative modal features for alignment. During the feature selection process, dynamically adjust the weights of each modal feature through a dynamic weight generation network according to the characteristics of the input data (such as different attributes of the brain tumor), and use an alignment algorithm to align the feature maps of different modalities;

[0009] Multi-scale Feature Fusion Enhancement: An additional network layer is introduced for feature fusion. This network layer is responsible for learning the weight assignment strategy. The weight assignment strategy dynamically adjusts the fusion weights and strategies according to the internal relationships and interactions between feature maps of different scales, and inputs the fused feature map into the segmentation network for preliminary segmentation. On the basis of the preliminary segmentation, an attention mechanism is introduced to guide the segmentation network to focus on the tumor region;

[0010] Reinforcement Learning Optimization: A reinforcement learning algorithm is used to optimize the parameters and structure of the segmentation network. The reinforcement learning algorithm continuously iterates and adjusts the segmentation strategy to find the best segmentation result;

[0011] Uncertain Region Optimization: For the uncertain regions in the segmentation result, post-processing methods are used for further optimization;

[0012] Three-dimensional Tumor Segmentation Result: Output the three-dimensional tumor segmentation result, which is used for clinical diagnosis and treatment plan formulation.

[0013] The above further includes:

[0014] Further, in data preprocessing and feature extraction, the MRI sequences include T1-weighted images, T2-weighted images, and FLAIR (Fluid-Attenuated Inversion Recovery) sequences. For the images of the T1 sequence, Gaussian filtering is used to remove the noise in the images. For the images of the T2 and FLAIR sequences, histogram equalization is used to enhance the contrast to make the tumor region clearer.

[0015] In one embodiment, in data preprocessing and feature extraction, the preprocessed multi-modal image data is input into a convolutional neural network model, and the feature maps of each modality are extracted through convolutional layers. For the T1 sequence images, the feature maps of gray matter and white matter are extracted. For the T2 sequence images, the feature maps of edema and tumor tissues are extracted. For the FLAIR sequence images, the feature maps of brain tissues after cerebrospinal fluid suppression are extracted.

[0016] Further, the specific steps of the adaptive feature selection and alignment:

[0017] Adjust the weights of the features of each modality: Let the feature vector of the multi-modal MRI image be X = [x1, x2,..., x n , where x i represents the feature vector of the i-th modality, and the output of the dynamic weight generation network is the weight vector W = [w1, w2,..., w n , where w i represents the weight coefficient of the i-th modality, W = f DWGN (X), where f DWGN represents the function of the dynamic weight generation network;

[0018] Feature selection: After obtaining the weight vector W, the features of each modality are weighted and summed according to the weights to obtain the fused feature vector.

[0019] Feature alignment: Using mutual information as the similarity metric, the transformation parameters that maximize the mutual information between the feature maps F1 and F2 of the two modalities are found through an optimization algorithm. The transformation parameters are used to align F2 to the space of F1, thereby obtaining the aligned feature map F'2. The mutual information I(F1,F'2) is expressed as where p(f1,f'2) represents the joint probability distribution of F1 and F'2, and p(f1) and p(f'2) represent the marginal probability distributions of F1 and F'2 respectively. The goal of the optimization algorithm is to find the transformation parameters that maximize I(F1,F'2).

[0020] Furthermore, in multi-scale feature fusion enhancement, the specific steps of the feature fusion are as follows:

[0021] Feature collection: Collect feature maps of different scales.

[0022] Similarity calculation: Calculate the similarity between feature maps of different scales by calculating the cosine similarity between the feature maps. The cosine similarity formula is where F i and F j represent the i-th and j-th feature maps respectively, · represents the inner product operation, and ||·|| represents the norm of the vector.

[0023] Weight calculation: According to the similarity metric, calculate the weight of each feature map through the softmax function. The softmax function formula is where w i represents the weight of the i-th feature map, and reference is a fixed reference feature map or the average feature map of all feature maps.

[0024] Feature fusion: According to the calculated weights, fuse the feature maps of different scales, and use the weighted sum method to fuse the feature maps. The feature fusion formula is where F fused represents the fused feature map.

[0025] Furthermore, in multi-scale feature fusion enhancement, input the fused feature map into the segmentation network for preliminary segmentation, including the following steps:

[0026] Preliminary segmentation: Obtain a feature map that fuses multi-modal and multi-scale information. The feature map is then input into a convolutional neural segmentation network for preliminary tumor segmentation.

[0027] Attention mechanism-guided segmentation: The attention mechanism calculates the weights at each position in the feature map, enabling the segmentation network to focus more on the tumor region. A probability map of the brain tissue is fused to generate an attention mask, which guides the segmentation network to focus on key regions. The formula for generating the attention mask is where F img is the image feature, F prod is the prior probability map, and is the channel concatenation operation.

[0028] Furthermore, in the optimization of reinforcement learning, the specific steps for optimizing the parameters and structure of the segmentation network using the reinforcement learning algorithm are as follows:

[0029] Define the reward function: Define a reward function to evaluate the quality of the segmentation result. The Dice coefficient is used as an indicator of the reward function. The higher the Dice coefficient, the closer the segmentation result is to the ground truth label, and the greater the reward;

[0030] Policy representation: Represent the policy of the segmentation network;

[0031] Sampling policy: Use the Monte Carlo method to sample different policies to explore possible segmentation results in order to find the best segmentation policy;

[0032] Update policy: Evaluate the quality of the sampling policy according to the reward function and use gradient ascent to update the policy parameters. The goal of the update is to maximize the reward, that is, to find the policy that produces the best segmentation result;

[0033] Iteration and convergence: Repeat the sampling policy and the update policy, continuously iterate and adjust the policy parameters until the convergence condition is reached (such as the reward function no longer improves significantly or the preset number of iterations is reached). At this time, it is considered that the best segmentation policy has been found.

[0034] Furthermore, in the optimization of the uncertain region, Monte Carlo Dropout is used to perform multiple inferences to calculate the voxel-level uncertainty, identify the uncertain regions in the segmentation result, and use the CRF model to smooth the segmentation result of the uncertain regions;

[0035] The specific steps for using Monte Carlo Dropout to perform multiple inferences to calculate the voxel-level uncertainty and identify the uncertain regions in the segmentation result are as follows:

[0036] Input image: Use the multi-modal MRI image as the input;

[0037] Model inference: Use a neural network model with a dropout layer to perform multiple inferences. In each inference, a part of the neurons are randomly discarded, thereby obtaining multiple different segmentation results;

[0038] Calculation of Uncertainty: For each voxel, calculate the standard deviation or variance of the segmentation results in multiple inferences as the uncertainty of that voxel.

[0039] The present invention has the following beneficial effects:

[0040] In the present invention, an additional network layer is introduced for feature fusion. According to the internal relationships and interactions between feature maps of different scales, the fusion weights and strategies are dynamically adjusted to capture key information at different scales, thereby enhancing the performance of the segmentation network. The fused feature map is input into the segmentation network for preliminary segmentation. On the basis of the preliminary segmentation, an attention mechanism is introduced to guide the segmentation network to focus on the tumor region, significantly improving the accuracy of segmentation. The parameters and structure of the segmentation network are optimized using an enhanced learning algorithm. Through continuous iteration and adjustment, the best segmentation strategy is found to improve the accuracy and robustness of the segmentation results. Description of the Drawings

[0041] Figure 1 It is a step diagram of a multi-modal MRI brain tumor image segmentation method proposed by the present invention. Detailed Embodiments

[0042] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0043] Please refer to Figure 1 As shown, the present invention is a multi-modal MRI brain tumor image segmentation method, including the following steps:

[0044] Data Preprocessing and Feature Extraction: Collect and process image data from different MRI sequences (such as T1, T2, FLAIR, etc.), preprocess the MRI image data, including steps such as denoising and enhancing contrast to improve the image quality, and use a convolutional neural network to extract multi-modal and multi-scale feature maps;

[0045] Adaptive Feature Selection and Alignment: Dynamically select the most representative modal features for alignment according to information such as the type, size, and location of the brain tumor. During the feature selection process, the weights of each modal feature are dynamically adjusted through a dynamic weight generation network according to the characteristics of the input data (such as different attributes of the brain tumor), and an alignment algorithm is used to align the feature maps of different modalities;

[0046] Multi-scale Feature Fusion Enhancement: An additional network layer is introduced for feature fusion. This network layer is responsible for learning the weight assignment strategy, which dynamically adjusts the fusion weights and strategies according to the internal relationships and interactions between feature maps of different scales. The fused feature map is input into the segmentation network for preliminary segmentation. Based on the preliminary segmentation, an attention mechanism is introduced to guide the segmentation network to focus on the tumor region;

[0047] Reinforcement Learning Optimization: A reinforcement learning algorithm is used to optimize the parameters and structure of the segmentation network. This reinforcement learning algorithm continuously iterates and adjusts the segmentation strategy to find the best segmentation result;

[0048] Uncertain Region Optimization: For the uncertain regions in the segmentation result, a post-processing method is used for further optimization;

[0049] Three-dimensional Tumor Segmentation Result: Output the three-dimensional tumor segmentation result, which is used for clinical diagnosis and treatment plan formulation.

[0050] In one embodiment, in data preprocessing and feature extraction, the MRI sequences include T1-weighted images, T2-weighted images, and FLAIR (Fluid-Attenuated Inversion Recovery) sequences. For the images of the T1 sequence, Gaussian filtering is used to remove the noise in the images. For the images of the T2 and FLAIR sequences, histogram equalization is used to enhance the contrast, making the tumor region clearer.

[0051] In one embodiment, in data preprocessing and feature extraction, the preprocessed multi-modal image data is input into a convolutional neural network model, and feature maps of each modality are extracted through convolutional layers. For the T1 sequence images, feature maps of gray matter and white matter are extracted. For the T2 sequence images, feature maps of edema and tumor tissues are extracted. For the FLAIR sequence images, feature maps of brain tissues after cerebrospinal fluid suppression are extracted.

[0052] In one embodiment, the specific steps of adaptive feature selection and alignment are as follows:

[0053] Adjust the weights of each modality feature: Let the feature vector of the multi-modal MRI image be X = [x1, x2,..., x n , where x i represents the feature vector of the i-th modality, and the output of the dynamic weight generation network is the weight vector W = [w1, w2,..., w n , where w i represents the weight coefficient of the i-th modality, and W = f DWGN (X ) , where f DWGN represents the function of the dynamic weight generation network;

[0054] Feature Selection: After obtaining the weight vector W, the features of each modality are weighted and summed according to the weights to obtain the fused feature vector.

[0055] Feature Alignment: Using mutual information as the similarity metric, the transformation parameters that maximize the mutual information between the feature maps F1 and F2 of two modalities are found through an optimization algorithm. The transformation parameters are used to align F2 to the space of F1, thereby obtaining the aligned feature map F'2. The mutual information I(F1,F'2) is expressed as where p(f1,f'2) represents the joint probability distribution of F1 and F'2, and p(f1) and p(f'2) represent the marginal probability distributions of F1 and F'2 respectively. The goal of the optimization algorithm is to find the transformation parameters that maximize I(F1,F'2).

[0056] In one embodiment, in the multi-scale feature fusion enhancement, the specific steps of the feature fusion are as follows:

[0057] Feature Collection: Collect feature maps of different scales.

[0058] Similarity Calculation: Calculate the similarity between feature maps of different scales by calculating the cosine similarity between the feature maps. The cosine similarity formula is where F i and F j represent the i-th and j-th feature maps respectively, · represents the inner product operation, and ||·|| represents the norm of the vector.

[0059] Weight Calculation: According to the similarity metric, calculate the weight of each feature map through the softmax function. The softmax function formula is where w i represents the weight of the i-th feature map, and reference is a fixed reference feature map or the average feature map of all feature maps.

[0060] Feature Fusion: According to the calculated weights, fuse the feature maps of different scales. The weighted sum method is used to fuse the feature maps. The feature fusion formula is where F fused represents the fused feature map.

[0061] In one embodiment, in the multi-scale feature fusion enhancement, input the fused feature map into a segmentation network for preliminary segmentation, including the following steps:

[0062] Preliminary Segmentation: Obtain a feature map that fuses multi-modal and multi-scale information. The feature map is then input into a convolutional neural segmentation network for preliminary tumor segmentation.

[0063] Attention mechanism-guided segmentation: The attention mechanism calculates the weights at each position in the feature map, enabling the segmentation network to focus more on the tumor region. A probability map of the brain tissue is fused to generate an attention mask, which guides the segmentation network to focus on key regions. The formula for generating the attention mask is where F img is the image feature, F prod is the prior probability map, is the channel concatenation operation.

[0064] In one embodiment, in reinforcement learning optimization, the specific steps of using the reinforcement learning algorithm to optimize the parameters and structure of the segmentation network are as follows:

[0065] Define the reward function: Define a reward function to evaluate the quality of the segmentation result. The Dice coefficient is used as an indicator of the reward function. The higher the Dice coefficient, the closer the segmentation result is to the ground truth label, and the greater the reward;

[0066] Policy representation: Represent the policy of the segmentation network;

[0067] Sampling strategy: Use the Monte Carlo method to sample different policies to explore possible segmentation results in order to find the best segmentation strategy;

[0068] Update the policy: Evaluate the quality of the sampled policy according to the reward function, and use gradient ascent to update the policy parameters. The goal of the update is to maximize the reward, that is, to find the policy that produces the best segmentation result;

[0069] Iteration and convergence: Repeat the sampling strategy and the update strategy, continuously iterate and adjust the policy parameters until the convergence condition is reached (such as the reward function no longer improves significantly or reaches the preset number of iterations). At this time, it is considered that the best segmentation strategy has been found.

[0070] In one embodiment, in the optimization of uncertain regions, Monte Carlo Dropout is used to perform multiple inferences to calculate voxel-level uncertainty, identify the uncertain regions in the segmentation result, and use the CRF model to smooth the segmentation result of the uncertain regions;

[0071] The specific steps of using Monte Carlo Dropout to perform multiple inferences to calculate voxel-level uncertainty and identify the uncertain regions in the segmentation result are as follows:

[0072] Input image: Use the multi-modal MRI image as the input;

[0073] Model inference: Use a neural network model with a dropout layer to perform multiple inferences. In each inference, a part of the neurons are randomly discarded, thereby obtaining multiple different segmentation results;

[0074] Calculation of uncertainty: For each voxel, calculate the standard deviation or variance of its segmentation results in multiple inferences as the uncertainty of this voxel.

[0075] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-modal MRI brain tumor image segmentation method, characterized in that, It includes the following steps: Data preprocessing and feature extraction: Collect and process image data from different MRI sequences, preprocess the MRI image data, and use a convolutional neural network to extract multi-modal and multi-scale feature maps; Adaptive feature selection and alignment: According to the information of brain tumors, dynamically select the most representative modal features for alignment. During the feature selection process, dynamically adjust the weights of each modal feature through a dynamic weight generation network according to the characteristics of the input data, and use an alignment algorithm to align the feature maps of different modalities; Multi-scale feature fusion and enhancement: Introduce an additional network layer for feature fusion. The network layer is responsible for learning the weight assignment strategy. The weight assignment strategy dynamically adjusts the fusion weights and strategies according to the internal relationships and interactions between feature maps of different scales, and input the fused feature maps into the segmentation network for preliminary segmentation. On the basis of the preliminary segmentation, introduce an attention mechanism to guide the segmentation network to focus on the tumor area; Reinforcement learning optimization: Use a reinforcement learning algorithm to optimize the parameters and structure of the segmentation network. The reinforcement learning algorithm continuously iterates and adjusts the segmentation strategy to find the best segmentation result; Uncertain region optimization: For the uncertain regions in the segmentation result, use post-processing methods for further optimization; Three-dimensional tumor segmentation result: Output the three-dimensional tumor segmentation result.

2. The multimodal MRI brain tumor image segmentation method according to claim 1, wherein In data preprocessing and feature extraction, the MRI sequences include T1-weighted images, T2-weighted images, and FLAIR (fluid-attenuated inversion recovery) sequences. For the images of the T1 sequence, use Gaussian filtering to remove the noise in the images. For the images of the T2 and FLAIR sequences, use histogram equalization to enhance the contrast.

3. A multi-modal MRI brain tumor image segmentation method according to claim 1, characterized in that, In data preprocessing and feature extraction, input the preprocessed multi-modal image data into a convolutional neural network model, and extract the feature maps of each modality through convolutional layers. For the T1 sequence images, extract the feature maps of gray matter and white matter. For the T2 sequence images, extract the feature maps of edema and tumor tissues. For the FLAIR sequence images, extract the feature maps of brain tissues after cerebrospinal fluid suppression.

4. A multi-modal MRI brain tumor image segmentation method according to claim 1, characterized in that, The specific steps of the adaptive feature selection and alignment: Adjust the weights of each modal feature: Let the feature vector of the multi-modal MRI image be X = [x1, x2,..., x n , where x i represents the feature vector of the i-th modality, and the output of the dynamic weight generation network is the weight vector W = [w1, w2,..., w n , where w i represents the weight coefficient of the i-th modality, W = f DWGN (X), where f DWGN represents the function of the dynamic weight generation network; Feature selection: After obtaining the weight vector W, perform weighted summation on the features of each modality according to the weights to obtain the fused feature vector; Feature alignment: Using mutual information as a similarity metric, the transformation parameters that maximize the mutual information between the feature maps F1 and F2 of two modalities are found through an optimization algorithm. The transformation parameters are used to align F2 to the space of F1, thereby obtaining the aligned feature map F'2. The mutual information I(F1,F'2) is expressed as where p(f1,f'2) represents the joint probability distribution of F1 and F'2, and p(f1) and p(f'2) represent the marginal probability distributions of F1 and F'2 respectively. The goal of the optimization algorithm is to find the transformation parameters that maximize I(F1,F'2).

5. A multimodal MRI brain tumor image segmentation method according to claim 1, characterized in that, In multi-scale feature fusion and enhancement, the specific steps of the feature fusion: Feature collection: Collect feature maps of different scales; Similarity calculation: Calculate the similarity between feature maps of different scales by calculating the cosine similarity between the feature maps. The cosine similarity formula is where F i and F j represent the i-th and j-th feature maps respectively, · represents the inner product operation, and ||·|| represents the norm of the vector; Weight calculation: According to the similarity measure, calculate the weight of each feature map through the softmax function, and the formula of the softmax function is where w i represents the weight of the i-th feature map, and reference is a fixed reference feature map or the average feature map of all feature maps; Feature fusion: According to the calculated weights, feature maps of different scales are fused, and the weighted summation method is used to fuse the feature maps. The feature fusion formula is where F fused represents the fused feature map.

6. A multimodal MRI brain tumor image segmentation method according to claim 5, characterized in that In multi-scale feature fusion and enhancement, inputting the fused feature maps into the segmentation network for preliminary segmentation includes the following steps: Preliminary segmentation: Obtain a feature map that fuses multi-modal and multi-scale information. The feature map is then input into a convolutional neural segmentation network for preliminary tumor segmentation; Attention mechanism-guided segmentation: The attention mechanism calculates the weights of each position in the feature map, enabling the segmentation network to pay more attention to the tumor region. A probability map of the brain tissue is fused to generate an attention mask, which guides the segmentation network to focus on key regions. The formula for generating the attention mask is where F img is the image feature, and F prod is the prior probability map, and ⊕ represents the channel concatenation operation.

7. A multimodal MRI brain tumor image segmentation method according to claim 1, characterized in that, In reinforcement learning optimization, the specific steps of using a reinforcement learning algorithm to optimize the parameters and structure of the segmentation network: Define the reward function: Define a reward function to evaluate the quality of the segmentation results. The Dice coefficient is used as an indicator of the reward function. The higher the Dice coefficient, the closer the segmentation result is to the ground truth label, and the greater the reward. Policy representation: Represent the policy of the segmentation network. Sampling strategy: Explore possible segmentation results by sampling different policies using the Monte Carlo method to find the optimal segmentation policy. Update strategy: Evaluate the quality of the sampled policies according to the reward function and use gradient ascent to update the policy parameters. The goal of the update is to maximize the reward, that is, to find the policy that produces the best segmentation result. Iteration and convergence: Repeat the sampling strategy and the update strategy, continuously iterate and adjust the policy parameters until the convergence condition is reached. At this time, it is considered that the optimal segmentation policy has been found.

8. A multimodal MRI brain tumor image segmentation method according to claim 1, characterized in that In the optimization of the uncertain region, Monte Carlo Dropout is used for multiple inference to calculate the voxel-level uncertainty, identify the uncertain regions in the segmentation results, and use the CRF model to smooth the segmentation results of the uncertain regions. Specific steps for using Monte Carlo Dropout for multiple inference to calculate the voxel-level uncertainty and identify the uncertain regions in the segmentation results: Input image: Use the multi-modal MRI image as the input. Model inference: Use a neural network model with a dropout layer for multiple inferences. In each inference, a part of the neurons are randomly discarded to obtain multiple different segmentation results. Calculate uncertainty: For each voxel, calculate the standard deviation or variance of its segmentation results in multiple inferences as the uncertainty of the voxel.

Images