Brain tumor segmentation method based on multi-scale feature fusion

Abstract

The invention relates to the technical field of medical image processing, in particular to a brain tumor segmentation method based on multi-scale feature fusion, and the method comprises the steps: S1, data acquisition and preprocessing; s2, constructing a brain tumor segmentation model based on multi-scale feature fusion and Transform; s3, defining a cross entropy loss function, training the improved Unet model, and storing an optimal model weight in training for prediction; and S4, performing prediction by using the optimal model, storing a prediction result, performing online verification, calculating an evaluation index, and finally performing result comparison. According to the method, the segmentation precision of the MRI brain tumor image is improved, the disadvantage of a traditional segmentation method in long-distance modeling is improved, and feature recognition is more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a brain tumor segmentation method based on multi-scale feature fusion. Background Art

[0002] Brain tumors are a common heterogeneous intracranial tumor with a high incidence and extremely high mortality rate. Once the disease occurs, it will cause increased intracranial pressure and compress brain tissue, which will lead to severe central nervous system damage, causing patients to suffer from epileptic seizures, poor memory or physical dysfunction. If the tumor can be discovered as early as possible before it turns into a malignant tumor, it can greatly improve the patient's survival period and reduce treatment costs. In the past, neurosurgeons would analyze the nature of the tumor and draw the surgical work area based on the images of brain lesions, which not only consumed a lot of time, but also was affected by subjective factors and produced errors.

[0003] With the rapid development of modern artificial intelligence technology, people are increasingly exploring new methods for tumor diagnosis and surgical pre-evaluation intervention. Traditional manually designed feature methods have limitations in capturing the complexity of lesions. They not only rely on human-recognizable features, but also highly rely on the professional technical skills of doctors. Although semi-automatic segmentation technology started with human intervention, this intervention is often prone to introduce subjective biases and errors.

[0004] In order to accelerate the diagnosis process, simplify treatment plans, optimize monitoring effects, and assist surgeons to more accurately remove tumors, computer vision technology based on convolutional neural networks (CNNs) has shown excellent performance in the field of medical image segmentation in recent years. Therefore, it is particularly urgent to develop an automated, high-precision MRI image tumor segmentation method based on deep neural networks. This method will greatly promote the automation of medical image analysis and improve the accuracy and efficiency of tumor segmentation. Summary of the invention

[0005] The main purpose of the present invention is to overcome the problems in the prior art of brain tumor MRI image segmentation, such as slow convergence speed of traditional neural networks and poor segmentation results, and to provide a brain tumor segmentation method based on multi-scale feature fusion.

[0006] The technical solution adopted by the present invention to achieve its technical purpose is: a brain tumor segmentation method based on multi-scale feature fusion, the method comprising: S1, data acquisition and preprocessing; The original brain tumor dataset is preprocessed, and then the four modal brain tumor magnetic resonance images of Flair, T1, T1c, and T2 are processed. Since the dataset itself is a three-dimensional MRI image, and this method uses a two-dimensional U-Net as the main model structure, the image is first sliced ​​along the z-axis to segment the two-dimensional image data, and then the data is cropped, randomly flipped, etc. for data enhancement to expand the training dataset and improve the generalization ability of the model. Finally, the training set and the test set are divided.

[0007] S2. Build a brain tumor segmentation model based on multi-scale feature fusion and Transformer; A brain tumor segmentation method based on multi-scale feature fusion is constructed; an improved FViT module is added to the encoder part, and the feature image is sent to the FViT module after 3 CNN downsampling to capture the long-distance dependencies in the image. The network model uses 4 downsampling and 4 upsampling to retain the original U-shaped symmetric structure. The model introduces a bidirectional feature fusion module in the jump connection and decoder part. The feature map of the encoder part downsampling output can be fused with the feature map of the decoder part after the Transformer through the jump connection. When the input image passes through the encoder, the feature map will gradually become smaller due to the pooling operation. The decoder of the model restores the size of the feature map by upsampling, and finally obtains the segmentation result through a convolutional layer and a Sigmoid function.

[0008] S3. Define the cross entropy loss function and train the improved Unet model, while saving the optimal model weights during training for prediction; The training set was trained with the model constructed above, and the cross entropy classification loss function was adopted. After multiple trainings, the weight coefficient with the minimum loss of the validation set was selected as the final training weight in the training stage, and the network model with the best preservation results was selected.

[0009] S4. Use the optimal model to make predictions, save the prediction results, perform online verification, calculate the evaluation index, and finally compare the results. Use the saved model and weight parameters to segment the test set images into brain tumors and obtain the segmentation result map of the brain tumor images.

[0010] Preferably, the specific process in step S1 is as follows: (1) The original BraTS dataset, i.e., the MRI image data of patients with brain tumors, was divided into a training set and a test set according to the ratio; (2) Slice the four modalities of the three-dimensional MRI brain tumor image along the z-axis to obtain a two-dimensional tumor image; (3) Using data normalization methods to standardize MRI images of each modality; (4) Crop the processed image, delete invalid background information, reduce the calculation process, and use random flipping, rotation and other methods to enhance the data; (5) Input the preprocessed four modes into the network model.

[0011] Preferably, in step S3, the improved Unet model is improved on the basic Unet network structure, and the specific implementation method is: S21. Build an improved 2D-Unet network. The improved network architecture is an encoder and decoder structure with a skip connection architecture. The encoder part includes an improved Transformer encoder module, and the decoder part includes a designed bidirectional feature fusion module. S22, improve the original Transformer encoder structure by introducing Fourier transform (FFT) and inverse transform (IFFT), and add a learnable parameter matrix to perform weighted modulation on the frequency domain signal; S23. A bidirectional feature fusion module is designed in the jump connection part to fuse the features passed through CNN downsampling and the features upsampled through Transformer to enhance the ability to extract important features.

[0012] Preferably, the specific structure of the bidirectional feature fusion module is: The bidirectional feature fusion module includes three branches: first, the channel attention branch uses the SE attention mechanism module to perform channel attention calculation on the feature map after Transformer upsampling to enhance the key channel features; Secondly, the spatial attention branch processes the feature map after CNN downsampling through maximum pooling and average pooling, calculates the spatial attention score, and highlights the important spatial areas; Finally, the bilinear modeling branch fuses the feature maps of the first two branches through a bilinear pooling operation, then concatenates the feature maps of the three branches, and then performs deep processing through a residual block; To avoid overfitting, the entire fusion result will also be randomly deactivated through the Dropout layer.

[0013] Preferably, an improved FViT module is introduced in the encoder part of Unet, and the FViT module includes a linear projection layer, a patch embedding layer, a Transformer encoder layer and a feature mapping layer; The linear projection layer compresses the input image tensor into a vector form and then passes it to the patch embedding layer, which adds position encoding to integrate global information. Next, the Transformer encoder layer uses a multi-head attention mechanism to capture the dependencies between features. Through Fourier transform (FFT) and inverse transform (IFFT), the data is converted between the spatial domain and the frequency domain. The signal is modulated using a parameter matrix to enhance the recognition of local features. Finally, the final result is output through a multi-layer perceptron (MLP) basic module.

[0014] Compared with the prior art, the present invention has the following beneficial effects: This brain tumor segmentation method based on multi-scale feature fusion improves the 2D-Unet network model, which not only maintains the segmentation accuracy but also speeds up the convergence of the network. By introducing the improved FViT module, the image signal is transferred to the frequency domain for processing, which effectively reduces noise interference. This improvement not only retains the model's advantages in long-distance feature modeling, but also enhances the ability to recognize local features. In addition, the addition of a bidirectional feature fusion module ensures that key features are highlighted while eliminating the interference of irrelevant information, thereby significantly improving the accuracy of brain tumor segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the training process of the present invention.

[0016] Figure 2 Schematic diagram of the improved U-Net structure of the present invention.

[0017] Figure 3 This is the improved FViT module structure proposed by the present invention.

[0018] Figure 4 It is a bidirectional feature fusion module structure.

[0019] Figure 5 This is the data set of the method of the present invention and the brain tumor segmentation result diagram.

[0020] in: DETAILED DESCRIPTION

[0021] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the inventive product is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0022] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below through the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of the present invention. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention. Example 1

[0024] See also Figure 1 , a brain tumor segmentation method based on multi-scale feature fusion, the method comprising: S1, data acquisition and preprocessing; The original brain tumor dataset is preprocessed, and then the four modal brain tumor magnetic resonance images of Flair, T1, T1c, and T2 are processed. Since the dataset itself is a three-dimensional MRI image, and this method uses a two-dimensional U-Net as the main model structure, the image is first sliced ​​along the z-axis to segment the two-dimensional image data, and then the data is cropped, randomly flipped, etc. for data enhancement to expand the training dataset and improve the generalization ability of the model. Finally, the training set and the test set are divided.

[0025] The specific process in step S1 is as follows: (1) The original BraTS dataset, i.e., the MRI image data of patients with brain tumors, was divided into a training set and a test set according to the ratio; (2) Slice the four modalities of the three-dimensional MRI brain tumor image along the z-axis to obtain a two-dimensional tumor image; (3) Using data normalization methods to standardize MRI images of each modality; (4) Crop the processed image, delete invalid background information, reduce the calculation process, and use random flipping, rotation and other methods to enhance the data; (5) Input the preprocessed four modes into the network model.

[0026] S2. Build a brain tumor segmentation model based on multi-scale feature fusion and Transformer; A brain tumor segmentation method based on multi-scale feature fusion is constructed; an improved FViT module is added to the encoder part, and the feature image is sent to the FViT module after 3 CNN downsampling to capture the long-distance dependencies in the image. The network model uses 4 downsampling and 4 upsampling to retain the original U-shaped symmetric structure. The model introduces a bidirectional feature fusion module in the jump connection and decoder part. The feature map of the encoder part downsampling output can be fused with the feature map of the decoder part after the Transformer through the jump connection. When the input image passes through the encoder, the feature map will gradually become smaller due to the pooling operation. The decoder of the model restores the size of the feature map by upsampling, and finally obtains the segmentation result through a convolutional layer and a Sigmoid function.

[0027] S3. Define the cross entropy loss function and train the improved Unet model, while saving the optimal model weights during training for prediction; The training set was trained with the model constructed above, and the cross entropy classification loss function was adopted. After multiple trainings, the weight coefficient with the minimum loss of the validation set was selected as the final training weight in the training stage, and the network model with the best preservation results was selected.

[0028] In step S3, the improved Unet model is improved on the basic Unet network structure, and the specific implementation method is: S21. Build an improved 2D-Unet network. The improved network architecture is an encoder and decoder structure with a skip connection architecture. The encoder part includes an improved Transformer encoder module, and the decoder part includes a designed bidirectional feature fusion module. S22, improve the original Transformer encoder structure by introducing Fourier transform (FFT) and inverse transform (IFFT), and add a learnable parameter matrix to perform weighted modulation on the frequency domain signal; S23. A bidirectional feature fusion module is designed in the jump connection part to fuse the features passed through CNN downsampling and the features upsampled through Transformer to enhance the ability to extract important features.

[0029] Furthermore, the specific structure of the bidirectional feature fusion module is as follows: the bidirectional feature fusion module includes three branches: first, the channel attention branch uses the SE attention mechanism module to perform channel attention calculation on the feature map after Transformer upsampling to enhance the key channel features; secondly, the spatial attention branch processes the feature map after CNN downsampling through maximum pooling and average pooling, calculates the spatial attention score, and highlights the important spatial area; finally, the bilinear modeling branch fuses the feature maps of the first two branches through a bilinear pooling operation, and then splices the feature maps of the three branches, and then performs deep processing through the residual block; to avoid overfitting, the entire fusion result will also be randomly inactivated through the Dropout layer.

[0030] Furthermore, an improved FViT module is introduced in the encoder part of Unet. The FViT module includes a linear projection layer, a patch embedding layer, a Transformer encoder layer and a feature mapping layer. The linear projection layer compresses the input image tensor into a vector form and then passes it to the patch embedding layer, which adds position encoding to integrate global information. Then, the Transformer encoder layer adopts a multi-head attention mechanism to capture the dependencies between features. Through Fourier transform (FFT) and inverse transform (IFFT), the data is converted between the spatial domain and the frequency domain. The signal is modulated using a parameter matrix to enhance the recognition of local features. Finally, the final result is output through a multi-layer perceptron (MLP) basic module.

[0031] S4. Use the optimal model to make predictions, save the prediction results, perform online verification, calculate the evaluation index, and finally compare the results. Use the saved model and weight parameters to segment the test set images into brain tumors and obtain the segmentation result map of the brain tumor images.

[0032] Specifically, the main principle of the brain tumor segmentation method based on multi-scale feature fusion is as follows: first, the improved FViT module is introduced in the encoder part of Unet, and the input signal is converted from the time domain to the frequency domain using Fourier transform to enhance or suppress specific frequency components. Secondly, the frequency domain signal is weighted modulated using a learnable parameter matrix to highlight important frequency components and reduce noise interference, thereby enhancing the feature extraction capability. In addition, a bidirectional feature fusion (BiFF) module is designed in the jump connection part to fuse the features obtained by CNN convolution in the encoder part with the features extracted by Transformer, so that the model can pay more attention to the area of ​​interest during the upsampling process, avoid the interference of invalid information features, and improve the segmentation accuracy of brain tumors. Example 2

[0033] See also Figure 1-5 Based on the above embodiments, the brain tumor segmentation method based on multi-scale feature fusion is taken as an example in the embodiment described below.

[0034] In order to verify the segmentation accuracy of the model of the present invention, the experimental data set uses two brain tumor MRI image data sets BraTS 2020 and BraTS 2021 provided by the International Multimodal Brain Tumor Segmentation Challenge (BraTS Challenge). The labeled data is divided into label0: background, label1: necrotic tumor core area, label2: edema area around the tumor, label4: enhanced tumor area; BraTS2021 data set is selected for model testing. Reference Figure 1 , Figure 1 It is a schematic flow diagram of the present invention. In this embodiment, the following steps are performed: Step 1: First, obtain the BraTS original data set and slice the four modality MRI images of Flair, T1, T1c, and T2 along the z-axis to convert the images from the original three-dimensional to two-dimensional to meet the needs of model training.

[0035] The original data set, i.e., the MRI image data of patients with brain tumors, is divided into a training set and a test set according to the ratio. The data normalization method is used to normalize the MRI images of each modality.

[0036] The processed image is cropped to a size of 160×160, the invalid background information is deleted, the calculation process is reduced, and the data is enhanced by random flipping, rotation and other methods; Step 2: If Figure 2As shown in the figure, a brain tumor segmentation method based on multi-scale feature fusion is constructed; an improved FViT module is added to the encoder part, and the feature image is sent to the ViT module after 3 CNN downsampling to capture the long-distance dependencies in the image. The network model uses 4 downsampling and 4 upsampling to retain the original U-shaped symmetric structure. The model introduces a bidirectional feature fusion module in the jump connection and decoder part. The feature map of the encoder part downsampling output can be fused with the feature map of the decoder part after the Transformer through the jump connection. When the input image passes through the encoder, the feature map will gradually become smaller due to the pooling operation. The decoder of the model restores the size of the feature map by upsampling, and finally obtains the segmentation result through a convolutional layer and a Sigmoid function.

[0037] The improved network architecture is an encoder and decoder with a skip connection architecture. The decoder part contains a designed bidirectional feature fusion (BiFF) module, which replaces the direct feature map concatenation of the original encoder part to enhance the focus on important regional features.

[0038] The constructed network model maintains the original U-shaped symmetrical structure with a jump connection part in the middle. The last layer of the encoder uses the ViT model for feature downsampling to capture long-distance dependencies in the image.

[0039] The bidirectional feature fusion module is as follows Figure 3 As shown in the figure, it includes three branches: first, the channel attention branch uses the SE attention mechanism module to calculate the channel attention of the feature map after the Transformer is upsampled to enhance the key channel features. Secondly, the spatial attention branch processes the feature map after CNN downsampling through maximum pooling and average pooling, calculates the spatial attention score, and highlights the important spatial area; finally, the bilinear modeling branch fuses the feature maps of the first two branches through bilinear pooling operations, and then splices the feature maps of the three branches, and then performs deep processing through the residual block. To avoid overfitting, the entire fusion result will also be randomly inactivated through the Dropout layer.

[0040] FViT modules such as Figure 4 As shown in the figure, it includes a linear projection layer, a patch embedding layer, a Transformer encoder layer, and a feature mapping layer. The linear projection layer compresses the input image tensor into a vector form and then passes it to the patch embedding layer, which adds position encoding to integrate global information. Next, the Transformer encoder layer uses a multi-head attention mechanism to capture the dependencies between features. Through FFT and IFFT, the data is converted between the spatial domain and the frequency domain, and the signal is modulated using a learnable parameter matrix to enhance the recognition of local features. Finally, the final result is output through a multi-layer perceptron (MLP) basic module.

[0041] After the feature map is downsampled by ViT, it will undergo two basic convolutions and then enter the decoder. The decoder includes an upsampling layer process. The upsampling layer contains the proposed bidirectional feature fusion module, which receives the feature map transmitted by the jump connection and then splices it with the feature map output by the upsampling. Finally, it outputs a segmentation result prediction map with the same resolution as the input image.

[0042] Step 3: Training the model; using the cross entropy classification loss function, after 200 trainings, the experiment selects the weight coefficient with the minimum loss of the validation set as the final training weight of the training stage, and saves the network model with the best result.

[0043] Step 4: Figure 5 As shown in the figure, the brain tumor image segmentation application uses the saved model and weight parameters to segment the test set images into brain tumors, obtains the segmentation result map of the brain tumor image and compares it with some classic image segmentation models.

[0044] The solution in this embodiment can be selectively combined with the solutions in other embodiments for use.

[0045] It should be noted that although the above embodiments have been described in this article, the scope of patent protection of the present invention is not limited thereby. Therefore, based on the innovative concept of the present invention, changes and modifications made to the embodiments described herein, or equivalent structures, equivalent processes or equivalent functional transformations made using the contents of the present specification and drawings, directly or indirectly applying the above technical solutions to other related technical fields are all included in the scope of protection of the patent of the present invention.

Claims

1. A brain tumor segmentation method based on multi-scale feature fusion, characterized in that: The method comprises: S1, data acquisition and preprocessing; S2. Build a brain tumor segmentation model based on multi-scale feature fusion and Transformer; S3. Define the cross entropy loss function and train the improved Unet model, while saving the optimal model weights during training for prediction; S4. Use the optimal model to make predictions, save the prediction results, and perform online verification. At the same time, calculate the evaluation indicators and finally compare the results.

2. The brain tumor segmentation method based on multi-scale feature fusion according to claim 1, characterized in that: In step S1, the original brain tumor data set is preprocessed. First, the image is sliced ​​along the z-axis to segment the two-dimensional image data. Then, the data is cropped and randomly flipped for data enhancement to expand the training data set and improve the generalization ability of the model. Finally, the training set and the test set are divided.

3. The brain tumor segmentation method based on multi-scale feature fusion according to claim 1, characterized in that: In step S3, the improved Unet model is improved on the basic Unet network structure, and the specific implementation method is: S21. Build an improved 2D-Unet network. The improved network architecture is an encoder and decoder structure with a skip connection architecture. The encoder part includes an improved Transformer encoder module, and the decoder part includes a designed bidirectional feature fusion module. S22, improve the original Transformer encoder structure by introducing Fourier transform and inverse transform, and add parameter matrix to perform weighted modulation on frequency domain signal; S23. A bidirectional feature fusion module is designed in the jump connection part to fuse the features passed through CNN downsampling and the features upsampled through Transformer to enhance the ability to extract important features.

4. The brain tumor segmentation method based on multi-scale feature fusion according to claim 3, characterized in that: The improved FViT module is introduced in the encoder part of Unet. The FViT module includes a linear projection layer, a patch embedding layer, a Transformer encoder layer, and a feature mapping layer. The linear projection layer compresses the input image tensor into a vector form and then passes it to the patch embedding layer, which adds position encoding to integrate global information. Next, the Transformer encoder layer uses a multi-head attention mechanism to capture the dependencies between features. Through Fourier transform and inverse transform, the data is converted between the spatial domain and the frequency domain. The signal is modulated using a parameter matrix to enhance the recognition of local features. Finally, the final result is output through a multi-layer perceptron basic module.

5. The brain tumor segmentation method based on multi-scale feature fusion according to claim 4, characterized in that: The specific structure of the bidirectional feature fusion module includes three branches: First, the channel attention branch uses the SE attention mechanism module to perform channel attention calculation on the feature map after Transformer upsampling to enhance the key channel features; Secondly, the spatial attention branch processes the feature map after CNN downsampling through maximum pooling and average pooling, calculates the spatial attention score, and highlights the important spatial areas; Finally, the bilinear modeling branch fuses the feature maps of the first two branches through a bilinear pooling operation, then concatenates the feature maps of the three branches, and then performs deep processing through a residual block.

Images