3D brain tumor segmentation model combining efficient transformer and unet

Abstract

The application discloses a 3D brain tumor segmentation method based on deep learning and combining efficient Transform and UNet, and mainly relates to the technical field of medical image segmentation; including the following steps: S1, collecting multi-modal magnetic resonance imaging data of a brain tumor patient in a sufficient quantity; S2, data preprocessing, data set division and data enhancement; S3, constructing a 3D brain tumor segmentation model combining efficient Transform and UNet; S4, inputting the training set after preprocessing into the 3D brain tumor segmentation method combining efficient Transform and UNet for training; and S5, sending multi-modal magnetic resonance imaging data to be segmented into the trained 3D brain tumor segmentation model combining efficient Transform and UNet for segmentation; the application can solve the problem of poor automatic segmentation effect of the existing multi-modal magnetic resonance imaging data of a brain tumor patient.

Description

Technical Field

[0001] This invention relates to the field of medical image segmentation technology, specifically a 3D brain tumor segmentation model that combines efficient Transformer and UNet. Background Technology

[0002] Brain tumors pose a serious threat to human life and health. Different types of brain tumors vary significantly in treatment methods and prognosis; timely detection and treatment help improve overall patient survival. MRI is a commonly used non-invasive diagnostic technique in clinical practice, offering various imaging modalities, including T1-weighted (T1w), contrast-enhanced T1-weighted (CE_T1w), T2-weighted (T2w), and Fluid Attenuation Inversion Recovery (FLAIR), each providing unique contrasts in brain structure and pathology. Brain tumors consist of three overlapping regions: the edema region (ED), the enhanced tumor region (ET), and the necrotic core / non-enhancing tumor region (NCR / NET), each reflecting different biological characteristics.

[0003] In clinical practice, doctors often manually segment tumor subregions from MRI to determine a patient's condition, which is time-consuming, labor-intensive, and easily influenced by subjective factors. Therefore, automatically and accurately segmenting brain tumor regions from multimodal MRI can significantly improve the efficiency and accuracy of clinical diagnosis, but it also presents considerable challenges. First, due to different imaging principles, there are significant differences between different modal domains, resulting in low contrast and making it difficult to identify small targets. Second, different tumor subregions exhibit high heterogeneity in appearance and location, making it difficult to find commonalities through direct matching. Finally, the blurred boundaries between normal and diseased tissues make it difficult to accurately separate the tumor outline, hindering the use of strong clinical prior knowledge in segmentation models.

[0004] Traditional brain tumor segmentation methods include machine learning-based, threshold-based, and region-growing-based approaches. However, these methods rely on complex feature engineering and have relatively low feature representation capabilities, requiring manual intervention. Deep learning technology, on the other hand, can perform faster and more automated segmentation of large-scale medical images, reducing the workload of physicians. Therefore, developing more automated and accurate brain tumor segmentation models has become a hot research topic. Summary of the Invention

[0005] The purpose of this invention is to solve the problem that existing brain tumor segmentation methods based on multimodal MRI have poor automated segmentation results, especially in their insufficient ability to distinguish tumor sub-regions, which makes it difficult to meet the actual needs of medicine.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] A 3D brain tumor segmentation method combining efficient Transformer and UNet includes the following steps:

[0008] S1. Collect a sufficient number of multimodal MRIs of brain tumor patients, with cases from multiple centers to increase diversity;

[0009] S2. All the acquired raw MRI and label data are preprocessed, including registration, skull dissection, and spatial alignment. Then, the original dataset is augmented by methods such as intensity scaling, rotation, and random cropping.

[0010] S3. Construct a 3D brain tumor segmentation model that combines efficient Transformer and UNet;

[0011] S4. Input the training set obtained in step 2 into the 3D brain tumor segmentation model combining efficient Transformer and UNet constructed in step 3 for training. Use the Dice loss function for backpropagation to update the network parameters, and optimize the model through adaptive moment estimation (Adam) to obtain the trained 3D brain tumor segmentation model combining efficient Transformer and UNet.

[0012] S5. Input the multimodal MRI data of the brain tumor patient to be segmented and predicted into the trained 3D brain tumor segmentation model combining efficient Transformer and UNet obtained in step 4 for segmentation, and obtain the segmentation prediction results of the multimodal MRI data.

[0013] Preferably, in step S2, the 3D brain tumor segmentation model combining efficient Transformer and UNet mainly includes four DWFormer encoder modules, four ESCA spatial-channel attention modules, three MSFCA cross-attention modules, and four decoder modules; the four cascaded DWFormer encoders constitute the encoding path. Before entering the encoding path, each modality first passes through two 3×3×3 convolutional units (including activation and normalization layers) to increase the feature space, and then passes through the four DWFormer encoders in sequence. The output of the first DWFormer module is connected to the input of the second DWFormer, the output of the second DWFormer module is connected to the input of the third DWFormer, and the output of the third DWFormer module is connected to the input of the fourth DWFormer.

[0014] Preferably, each DWFormer encoder performs downsampling and local patching using a 3×3×3 convolutional unit with a stride of 2. Then, a DWFormer module performs local-global interaction on the feature map. While adhering to the overall Transformer structure, the DWFormer module uses lightweight depthwise separable blocks instead of self-attention operations. Specifically, the input features are first upscaled by a 1×1×1 convolution within the depthwise separable block, then spatial voxel features are extracted using a 7×7×7 large-kernel convolution to increase the receptive field, and finally, the number of channels is restored using a 1×1×1 convolution. For each 3×3×3 convolutional unit: all layers are concatenated sequentially; the input of the 3×3×3 convolution becomes the input of the 3×3×3 convolutional unit; the output of the 3×3×3 convolutional unit becomes the input of the batch normalization layer; the output of the batch normalization layer becomes the input of the ReLU activation function; and the output of the ReLU activation function becomes the output of the 3×3×3 convolutional unit.

[0015] Preferably, the output of the encoding path serves as the input to the ESCA module, with four ESCA modules cascaded in series. Each ESCA module consists of spatial attention and channel attention. For spatial attention, it is decoupled into axis-based interaction and window-based interaction. Axis-based interaction calculates axial attention along the three dimensions of the voxel feature map. The calculation method involves first converting the data from other axes to a batch axis using matrix transpose, and then performing the computation according to the standard Transformer process. Window-based interaction calculates window attention for tokens within each local 3D patch. For positional encoding, learnable absolute positional encoding is used for axis attention, and conditional positional encoding is used for window attention. After the outputs of the two attention branches are obtained, they are summed, and the final output is obtained through normalization and a multilayer perceptron layer.

[0016] Preferably, the MSFCA module is used to bridge the encoder and decoder. First, the feature maps are downsampled to the mean at different pooling rates (stride = 2, 4, 8). Then, the different feature maps are unified in dimension and concatenated by channel. A 1×1×1 convolution is then performed to adjust the number of channels, resulting in a feature map that integrates multi-scale information. This map then interacts with the feature map from the decoder. Finally, K and V are generated from the feature map from the encoder through a linear mapping, and Q is generated from the feature map from the decoder. The enhanced decoded features are obtained by calculating the correlation of the triples.

[0017] Preferably, each decoder consists of a transposed convolution with a stride of 2 and two consecutive 3×3×3 convolutional units. After passing through four decoders, the feature map resolution is restored to the original image size, and then the segmentation head obtains the segmentation results of three regions. The segmentation head consists of one 1×1×1 convolutional unit and one sigmoid activation function layer; the output of the fourth decoder module is connected to the input of the segmentation head; the input of the 1×1×1 convolutional unit forms the input of the segmentation module, the output of the 1×1×1 convolutional unit is connected to the input of the sigmoid activation function layer, and the output of the activation function layer forms the final output of the entire network.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0019] To fully explore the relevant information of each sub-region of brain tumor in multimodal MRI data, this invention models and learns multimodal MRI, designs a novel network to effectively extract features for each modality, and achieves efficient information fusion in the shallow and deep layers of the network. The deep learning model constructed in this invention can achieve relatively accurate segmentation and prediction of multimodal MRI brain tumors. Attached Figure Description

[0020] Figure 1 This is a basic flowchart of the 3D brain tumor segmentation model that combines the efficient Transformer and UNet of this invention;

[0021] Figure 2 This is a diagram of the deep learning-based model structure constructed in this invention;

[0022] Figure 3 This is a schematic diagram of the DWFormer encoder module;

[0023] Figure 4 This is a schematic diagram of the ESCA module;

[0024] Figure 5 This is a schematic diagram of the MSFCA module;

[0025] Figure 6This is a schematic diagram of the feature fusion and segmentation head structure; Detailed Implementation

[0026] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined in this application.

[0027] Example: The process is as follows. Figure 1 As shown, the present invention describes a 3D brain tumor segmentation model combining efficient Transformer and UNet, including the following steps:

[0028] S1: The publicly available multimodal brain tumor segmentation dataset BraTS2020 was used. This dataset contains a total of 369 labeled training samples from multiple medical centers. Each sample consists of four modalities: T1w, CE-T1w, T2w, and FLAIR.

[0029] S2: Preprocessing the collected data. The publicly available dataset has already undergone registration and skull stripping preprocessing, with a spatial resolution alignment of 1 mm³. All data were checked for completeness, and each modality and its corresponding label were cropped to 128×128×128 pixels. The dataset was divided into training and testing sets in a 7:3 ratio. Finally, data augmentation was performed on all data using methods such as standardization, intensity scaling, random rotation, and random cropping.

[0030] S3: Construct a 3D brain tumor segmentation model that combines efficient Transformer and UNet, and set the model parameters.

[0031] A 3D brain tumor segmentation model architecture combining efficient Transformer and UNet, such as Figure 2 As shown. The overall architecture is based on UNet, including the DWFormer encoder module, the ESCA high-efficiency spatial-channel attention module, the MSFCA multi-scale fusion cross-attention module, the decoder part, and the segmentation head.

[0032] 1) Encoder module

[0033] See the DWFormer block. Figure 3a. The entire encoding path consists of four encoders. Before being input into the encoding path, each modality first passes through two 3×3×3 convolutional units (containing activation and normalization layers) to enlarge its feature space, and then inputs it into the encoder. Each encoder performs downsampling and divides the local patch using a 3×3×3 convolutional unit with a stride of 2. Then, a DWFormer module performs local-global interaction on the feature map. While adhering to the overall Transformer structure, the DWFormer module uses lightweight depthwise separable blocks instead of self-attention operations. For example... Figure 3 As shown in b, the input features are first increased in dimensionality through a 1×1×1 convolution inside the depthwise separable block, and then spatial voxel features are extracted through a 7×7×7 large kernel convolution to increase the receptive field. Finally, the number of channels is restored through a 1×1×1 convolution.

[0034] like Figure 2 As shown on the left, the four modalities are sequentially input into the encoding path. Each modal first goes through two consecutive 3×3×3 convolutions to map the feature space with C channels, and then goes through four consecutive encoders to generate feature maps of different scales. Figure 6 'a' represents the multimodal feature fusion operation. Features extracted from different modalities are first concatenated along the channel dimension, then dimensionality is reduced using a 1×1×1 convolution, and finally, after passing through a normalization layer and an activation layer, features that fuse multimodal information are obtained to facilitate modeling in subsequent ESCA and MSFCA modules.

[0035] 2) ESCA module

[0036] See ESCA module Figure 4 The ESCA module consists of spatial attention and channel attention, used to model and enhance higher-order semantic information. For spatial attention, it is decoupled into axis-based interaction and window-based interaction. Axis-based interaction calculates axial attention along the three dimensions of the voxel feature map. The calculation method first transforms the data of other axes to batch processing axes using matrix transpose, and then performs the computation according to the standard Transformer process. Window-based interaction calculates window attention for tokens within each local 3D patch. For positional encoding, learnable absolute positional encoding is used for axis attention, and conditional positional encoding is used for window attention. After the outputs of the two attention branches are obtained, they are summed, and the final output is obtained through normalization and a multilayer perceptron layer.

[0037] 3) MSFCA module

[0038] See MSFCA module. Figure 5The MSFCA module bridges the encoder and decoder. Its core function is to calculate cross-attention between feature maps of different scales generated by the encoder and features from the decoder. This allows the decoder to adaptively retrieve valuable information from the encoder features to adapt to irregular lesions, thereby enhancing the lesion expression capability of the decoding path. First, the feature maps are downsampled at different pooling rates (stride = 2, 4, 8). Then, the different feature maps are unified in dimension and concatenated by channel. A 1×1×1 convolution is then performed to adjust the number of channels, resulting in a feature map that integrates multi-scale information. This map then interacts with the feature map from the decoder. To reduce the computational cost of the Transformer, a TokenLearner module based on spatial attention is used to reduce the number of tokens. Finally, K and V are generated from the feature map from the encoder through linear mapping, and Q is generated from the feature map from the decoder. The enhanced features are obtained by calculating the correlation of triples.

[0039] 4) Decoder

[0040] Each decoder consists of a transposed convolution with a stride of 2 and two consecutive 3×3×3 convolutional units. The transposed convolution upsamples the feature map from the previous decoder, doubling the resolution, while the 3×3×3 convolutional units further refine the enhanced decoded features. After four decoders, the feature map resolution is restored to the original image size.

[0041] Figure 6 b indicates that the segmentation head consists of one 1×1×1 convolutional unit and one sigmoid activation function layer; the output of the fourth decoder module is connected to the input of the segmentation head; the input of the 1×1×1 convolutional unit forms the input of the segmentation module, the output of the 1×1×1 convolutional unit is connected to the input of the sigmoid activation function layer, and the output of the activation function layer forms the final output of the entire network.

[0042] After building the deep learning model, the initial parameters of the network need to be set.

[0043] In this embodiment, the network parameters are set as follows: 500 training rounds, batch size of 1, initial learning efficiency of 10⁻⁴, learning rate using cosine annealing strategy, and warmup strategy used for preheating.

[0044] S4: Input the preprocessed training set from step 2 into the 3D brain tumor segmentation model based on deep learning combined with efficient Transformer and UNet constructed in step 3 for training. Use the Dice loss function for backpropagation to update the network parameters, and optimize the model through adaptive moment estimation (Adam) to obtain the trained 3D brain tumor segmentation model based on deep learning combined with efficient Transformer and UNet.

[0045] Dice loss function formula:

[0046]

[0047] Where N represents the number of samples, rn represents the label of sample n (1 for positive, 0 for negative), and pn represents the probability that sample n is predicted to be positive.

[0048] Step 5: Input the multimodal MRI data of the brain tumor patient to be segmented and predicted into the trained 3D brain tumor segmentation model combining efficient Transformer and UNet obtained in Step 4 for segmentation, and obtain the segmentation prediction results of the multimodal MRI data.

Claims

1. A method for constructing a 3D brain tumor segmentation model combining efficient Transformer and UNet, including the following steps: S1. Collect a sufficient number of multimodal magnetic resonance imaging (MRI) images of brain tumor patients from multiple centers to increase diversity; S2. All the acquired raw MRI and its label data are preprocessed by registration, skull dissection and spatial alignment. Then, the original dataset is augmented by intensity scaling, rotation and random cropping. S3. Construct a 3D brain tumor segmentation model that combines efficient Transformer and UNet; The 3D brain tumor segmentation model combining efficient Transformer and UNet includes four DWFormer encoder modules, four ESCA spatial-channel attention modules, three MSFCA cross-attention modules, and four decoder modules. The four cascaded DWFormer encoders form the encoding path. Before entering the encoding path, each modality first passes through two 3×3×3 convolutions, including activation and normalization layers, to enlarge the feature space of the units, and then passes through the four DWFormer encoders in sequence. The output of the first DWFormer module is connected to the input of the second DWFormer, the output of the second DWFormer module is connected to the input of the third DWFormer, and the output of the third DWFormer module is connected to the input of the fourth DWFormer. Each DWFormer encoder performs downsampling and divides the local patch using a 3×3×3 convolutional unit with a stride of 2. Then, a DWFormer module performs local-global interaction on the feature map. While following the overall structure of Transformer, the DWFormer module uses lightweight depthwise separable blocks to replace self-attention operations. Specifically, the input features are first upscaled by a 1×1×1 convolution inside the depthwise separable block, and then spatial voxel features are extracted by a 7×7×7 large kernel convolution to increase the receptive field. Finally, the number of channels is restored by a 1×1×1 convolution. The output of the encoded path serves as the input to the ESCA module, and the four ESCA modules are connected in series. The ESCA module consists of spatial attention and channel attention. For spatial attention, it is decoupled into axis-based interaction and window-based interaction. The axis-based interaction calculates axial attention along the three dimensions of the voxel feature map. The calculation method is to first transform the data of other axes into a batch axis by matrix transposition, and then perform the calculation according to the standard Transformer process. The window-based interaction calculates window attention for tokens within each local 3D patch. For position encoding, learnable absolute position encoding is used for axis attention and conditional position encoding is used for window attention. After the outputs of the two attention branch paths are obtained, they are summed and then normalized and processed through a multilayer perceptron layer to obtain the final output. S4. Input the training set obtained in step 2 into the 3D brain tumor segmentation model combining efficient Transformer and UNet constructed in step 3 for training. Use the Dice loss function for backpropagation to update the network parameters, and optimize the model through adaptive moment estimation and Adam to obtain the trained 3D brain tumor segmentation model combining efficient Transformer and UNet. S5. Input the multimodal MRI data of the brain tumor patient to be segmented and predicted into the trained 3D brain tumor segmentation model combined with efficient Transformer and UNet obtained in step 4 for segmentation, and obtain the segmentation prediction results of the patient's multimodal MRI.

2. The method for constructing a 3D brain tumor segmentation model combining efficient Transformer and UNet according to claim 1, characterized in that: For each 3×3×3 convolutional unit, all layers are concatenated in sequence. The input of the 3×3×3 convolution becomes the input of the 3×3×3 convolutional unit, the output of the 3×3×3 convolutional unit becomes the input of the batch normalization layer, the output of the batch normalization layer becomes the input of the ReLU activation function, and the output of the ReLU activation function becomes the output of the 3×3×3 convolutional unit.

3. The method for constructing a 3D brain tumor segmentation model combining efficient Transformer and UNet according to claim 1, characterized in that: The MSFCA module is used to bridge the encoder and decoder. First, it performs mean downsampling operations on the feature maps with different pooling rates stride=2, 4, and 8. Then, it unifies the dimensions of the different feature maps and concatenates them by channel. After adjusting the number of channels through a 1×1×1 convolution, it obtains a feature map that integrates multi-scale information. This map then interacts with the feature map from the decoder. Finally, K and V are generated from the feature map from the encoder through linear mapping, and Q is generated from the feature map from the decoder. The enhanced decoded features are obtained by calculating the correlation of triples.

4. The method for constructing a 3D brain tumor segmentation model combining efficient Transformer and UNet according to claim 3, characterized in that: Each decoder consists of a transposed convolution with a stride of 2 and two consecutive 3×3×3 convolutional units; after passing through four decoders, the feature map resolution is restored to the original image size, and then the segmentation head obtains the segmentation results of three regions.

5. The method for constructing a 3D brain tumor segmentation model combining efficient Transformer and UNet according to claim 4, characterized in that: The segmentation head consists of one 1×1×1 convolutional unit and one sigmoid activation function layer; the output of the fourth decoder module is connected to the input of the segmentation head; the input of the 1×1×1 convolutional unit forms the input of the segmentation module, the output of the 1×1×1 convolutional unit is connected to the input of the sigmoid activation function layer, and the output of the activation function layer forms the final output of the entire network.

Images