Brain tumor segmentation method based on Dual-SwinTransBTS
By combining Swing-Transformer and CNN, a Dual-SwinTransBTS model was constructed. Multimodal feature fusion was performed using a dual-branch encoder and STFusion module, which solved the problem of neglecting modal complementarity and improved the accuracy of brain tumor segmentation.
Patent Information
- Application Number
- CN202410748171.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-11
- Publication Date
- 2025-12-12
AI Technical Summary
Existing brain tumor segmentation models neglect the complementarity between modalities during multimodal fusion, leading to information loss or dilution and affecting segmentation accuracy.
By combining Swin-Transformer and CNN, a brain tumor segmentation method based on Dual-SwinTransBTS is constructed. Multimodal image features are extracted through a dual-branch encoder and STFusion module, and feature fusion is performed through a modal cross-attention module.
It improves the accuracy of brain tumor segmentation, especially in the representation of local features at lesion boundaries and the ability to extract remote features from three-dimensional images, thereby enhancing segmentation precision.
Smart Images
Figure CN121120661A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of computer vision, deep learning technology and medical image segmentation, and specifically relates to a medical image segmentation method combining Swin-Transformer and CNN, which realizes accurate segmentation of brain tumor images. BACKGROUND
[0002] Automatic brain tumor segmentation plays an important role in brain tumor diagnosis, surgical planning and treatment evaluation. However, due to the high variability of brain lesion structure, fuzzy tissue boundary, limited data and class imbalance, it still faces great challenges.
[0003] Brain tumor segmentation in magnetic resonance imaging (MRI) is of great significance in brain tumor diagnosis, tumor management and research. With the great success of the Brain tumor segmentation (BTS) ten-year challenge and the progress of CNN and Transformer algorithms, many excellent BTS models have been proposed to solve the difficulties of BTS in different technical aspects. Conventional brain tumor magnetic resonance imaging includes four types of images: T1 weighted image (T1), T2 weighted image (T2), T1 contrast enhanced image (T1 Gd) and FLAIR image (T2 FLAIR), which provides valuable information for the clinical diagnosis and treatment plan of glioma. Generally speaking, doctors evaluate glioma by integrating diagnostic information from different imaging methods, among which the enhanced area, tumor necrosis and peritumoral edema are most concerned, for example, it is generally believed that the higher the enhancement intensity, the larger the necrosis and edema area, and the higher the grade of glioma. Therefore, automatic and accurate segmentation of lesions is an important step for neurology precision medicine.
[0004] Convolutional Neural Networks (CNNs), due to their powerful feature representation capabilities, are widely used in medical image segmentation tasks and have achieved excellent performance. In recent years, the Vision Transformer (ViT) has introduced one of the most powerful techniques from natural language processing into the fields of computer vision and medical imaging. Based on a self-attention mechanism, the Transformer can capture long-range information in 3D volumetric data and has been rapidly adapted for brain tumor segmentation in 3D MRI sequences. Based on these two popular techniques, many excellent brain tumor segmentation methods have been proposed to address challenges such as lesion location, morphological uncertainty, low contrast, and annotation bias. However, most existing models fuse modalities at the input or feature level, neglecting the complementarity between modalities, leading to the loss or dilution of information from some modalities. Therefore, how to reasonably fuse multiple modalities is one of the factors in improving the accuracy of brain tumor segmentation. Summary of the Invention
[0005] This invention aims to combine Swin-Transformer and CNN to propose a brain tumor segmentation method based on Dual-SwinTransBTS. It extracts multimodal image features through a dual-branch encoder and fuses these features using the proposed STFusion module. The purpose of this invention is to improve the accuracy of brain tumor segmentation by enhancing the precise representation of local features at lesion boundaries and the ability to extract long-range features from 3D images.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows.
[0007] The brain tumor segmentation method based on Dual-SwinTransBTS is implemented through the following steps:
[0008] Step 1: Select a public dataset, divide the dataset into training, validation and test sets, and perform image augmentation on the images in the dataset to improve the model's segmentation performance.
[0009] Step 2: Construct a multimodal cross-attention (MCA) module based on Swin-Transformer. First, replace the MLP module in Swin-Transformer with a pyramidal convolution (PyConv) module. Then, use cross-modal attention (CMA) to achieve cross-modal interaction, thereby capturing information at different scales in the image. This helps the model learn more global image features and improves the model's performance.
[0010] Step 3: Construct Swin-Transformer Feature Fusion Module (STFusion). The STFusion module establishes global correlation between features of different scales through convolution and average pooling modules, and is used to fuse multi-scale features from the dual-branch encoder to obtain high-quality semantic segmentation performance.
[0011] Step 4: Construct a dual-branch encoder based on a modal cross-attention module (MCA) and a CNN-based decoder, and combine the STFusion to construct a brain tumor segmentation model Dual-SwinTransBTS.
[0012] The data preprocessing steps described in Step 1 are as follows:
[0013] The medical image segmentation dataset is the BraTS2021 dataset. First, perform image standardization to adjust all images to a voxel size of 128x128x128, which can help the model better learn features. Second, use random scaling, three-direction random flipping, Gaussian noise, Gaussian blur, and random contrast to enhance the images.
[0014] The specific process of constructing the multi-modal cross-attention module (MCA) based on Swin-Transformer in Step 2 is as follows:
[0015] Step 2.1: To consider both local and long-range information, combine Swin-Tansformer and CNN in the MCA module. The MCA block consists of two identical Swin-Tansformer branches, each used to extract features from two modalities. Each branch consists of two cascaded modules, namely the self-modal module and the cross-modal module.
[0016] Step 2.2: The self-modal module is used for feature extraction of each single modality and is a Swin-Transformer-CNN hybrid module. First, use Swin-Transformer to capture long-range information. Then use a Pyramidal Convolution (PyConv) layer instead of the MLP layer to extract multi-scale information. Taking T1 and T1 Gd as examples, the calculation formula of the self-modal module is: wherein, , These represent the input features of the MCA module; , These represent the outputs after the normalization layer and the multi-head self-attention module, respectively. , These represent the outputs after the normalization layer and the pyramid convolution module, respectively.
[0017] Step 2.3: The cross-modal module employs a sliding window self-attention Swin-Transformer, replacing the MLP layer with PyConv. It exchanges information between two related modalities through Cross-Modal Attention (CMA). The calculation formula for the cross-modal attention part is as follows: in, , They represent , The output after passing through the cross-modal attention (CMA) module. The final output after passing through the cross-modal module is calculated using the following formula: in, , These represent the outputs after window-shifting multi-head self-attention and cross-modal attention modules, respectively. , These represent the final outputs of the MCA module.
[0018] The design process for constructing the Swin-Transformer feature fusion module (STFusion) in step 3 is as follows:
[0019] Step 3.1: Since the encoder is a hybrid model composed of Swin-Transformer and CNN, and the decoder is a purely CNN-based design, a Swin-Transformer feature fusion module was designed to bridge the semantic gap between the features of the encoder and decoder, fusing features from the MCA module and the decoder.
[0020] Step 3.2: Let the outputs of the MCA module and the decoder be respectively... , The output after STFusion module The calculation is as follows: where, and denote the tensors of two branches in the dual-branch hybrid encoder, respectively.
[0021] Step 3.3: In the STFusion module, since both the input and feature tensors are three-dimensional volumes. To make full use of the three-dimensional information, the and are average-pooled in three directions, respectively, and the calculation formula is as follows: The vectors in three directions are reshaped into the same shape, connected, and compressed through a 1 x 1 x 1 convolution layer. They are reshaped to the original dimension to obtain new feature vectors in three directions and . Next, the features from the two branches are aggregated through convolution and concatenation operations, and the calculation formula is as follows: Through matrix multiplication operation of three vectors, the calibrated attention tensor A can be obtained: Finally, the final output of the STFusion module is obtained by concatenating the features from the MCA module and the convolution module , and the calculation formula is as follows: .
[0022] Step 4 constructs a dual-branch encoder based on the modal cross-attention module (MCA) and a CNN-based decoder, and combines STFusion to construct a brain tumor segmentation model Dual-SwinTransBTS, and the steps are as follows:
[0023] Step 4.1: Build the framework of the proposed model, including the MCA encoder, the STFusion feature fusion module, and the decoder.
[0024] Step 4.2: Divide the training set, test set, and validation set in the processed data set.
[0025] Step 4.3: Design a good training and prediction strategy, set the initial learning rate lr, the number of iterations Epochs, the batch size BatchSize, and the optimizer Adam; the loss function is selected as the Dice loss, and the calculation formula is as follows: Wherein, T represents GroundTruth, and P represents the segmentation result.
[0026] Step 4.4: input the segmented image in the verification set into the trained Dual-SwinTransBTS brain tumor segmentation network for segmentation, evaluate the model using the Dice coefficient, Hausdorff distance (HD) and sensitivity (Sensitivity), optimize the parameters of the network, until the training reaches convergence, and obtain the final segmentation result.
[0027] The brain tumor segmentation method based on Dual-SwinTransBTS proposed by the application has the following advantages:
[0028] The Dual-SwinTransBTS proposed by the application can be used to process 3D MRI images and sequence data, which reorganizes the input modalities and improves the accuracy of segmentation.
[0029] The application proposes a dual-branch hybrid encoder based on a modal correlation cross-attention block (MCA) for extracting multi-modal image features, and the dual-encoder structure combines the advantages of CNN and Swin-Transformer, has the ability of local feature representation of accurate lesion boundary and long-range feature extraction of 3D volume image.
[0030] The Swin-Transformer feature fusion module (STFusion) designed by the application establishes the global correlation between features of different scales through convolution and average pooling, thereby effectively fusing features from the encoder and the decoder, and obtaining high-quality semantic segmentation performance. BRIEF DESCRIPTION OF DRAWINGS
[0031] Figure 1 The flowchart of the brain tumor segmentation method based on Dual-SwinTransBTS proposed by the application.
[0032] Figure 2 The Multimodal Cross-Attention (MCA) module proposed by the application.
[0033] Figure 3A Swin-Transformer Feature Fusion (STFusion) module designed for the application.
[0034] Figure 4 A schematic diagram of the overall framework of the Dual-SwinTransBTS network proposed in the application. DETAILED DESCRIPTION
[0035] In order to better show the purpose, structure and function of the application, the Dual-SwinTransBTS-based brain tumor segmentation method of the application will be further described in detail below in combination with the drawings. It should be pointed out that those skilled in the art can make some improvements to the application without departing from the core idea of the application, and these all belong to the protection scope of the application.
[0036] The application proposes a Dual-SwinTransBTS-based brain tumor segmentation method, as shown in Figure 1 The method comprises the following steps:
[0037] Step 1: data preprocessing step.
[0038] The public dataset BraTS2021 is selected, and first, image standardization operation is performed to adjust all images to a volume size of 128x128x128, which can help the model learn features better. Secondly, random scaling, three-direction random flipping, Gaussian noise, Gaussian blur and random contrast are used to perform image enhancement operation.
[0039] Step 2: build a Swin-Transformer-based multi-modal cross-attention module (MCA), as shown in Figure 2 The specific process is as follows:
[0040] Step 2.1: In order to consider local information and long-range information at the same time, Swin-tansformer and CNN are combined in the MCA module. The MCA block is composed of two identical Swin-tansformer branches, which are used to extract features from two modalities respectively, and each branch is composed of two cascaded modules, i.e. self-modal module and cross-modal module.
[0041] Step 2.2: The self-modal module is used for feature extraction of each single modality, which is a Swin-Transformer-CNN hybrid module. First, Swin-Transformer is used to capture long-range information. Then, a pyramid convolution layer (PyConv) is used to replace the MLP layer to extract multi-scale information. Taking T1 and T1 Gd as an example, the calculation formula of the self-modal module is: wherein, , respectively represent the input features of the MCA module; , respectively represent the outputs after the normalization layer and the multi-head self-attention module; , respectively represent the outputs after the normalization layer and the pyramid convolution module.
[0042] Step 2.3: The cross-modal module adopts Swin-Transformer with sliding window self-attention, and replaces the MLP layer with PyConv. It exchanges information between two related modalities through cross-modal attention (CMA), and the calculation formula of the cross-modal attention part is as follows: wherein, , respectively represent the outputs of the cross-modal attention (CMA) module. The calculation formula of the final output after the cross-modal module is as follows: wherein, , respectively represent the outputs of the window shift multi-head self-attention and cross-modal attention modules, , respectively represent the final outputs of the MCA module. The construction of the Swin-Transformer feature fusion module (STFusion) in Step 3 is shown in
[0043] The design process of the STFusion module is as follows: Figure 3
[0044] Step 3.1: Since the encoder is a hybrid model composed of Swin-Transformer and CNN, the decoder is a purely CNN-based design to make up for the semantic gap between the features of the encoder and the decoder. A Swin-Transformer feature fusion module is designed to fuse the features from the MCA module and the decoder.
[0045] Step 3.2: Let the outputs of the MCA module and the decoder be , , the output of the STFusion module is calculated as follows: where and represent the tensors of the two branches in the dual-branch hybrid encoder.
[0046] Step 3.3: In the STFusion module, since the input and feature tensors are three-dimensional volumes. To make full use of the three-dimensional information, we perform average pooling on and in three directions, respectively, and the calculation formula is as follows: Reshape the three-direction vectors into the same shape, concatenate them, and compress the channels through a 1x1x1 convolution layer. Reshape them to the original dimensions to obtain new feature vectors in three directions and . Next, aggregate the features from the two branches through convolution and concatenation operations, and the calculation formula is as follows: By performing matrix multiplication on the three vectors, we can obtain the calibrated attention tensor A: Finally, the final output of the STFusion module is obtained by concatenating the features from the MCA module and the convolution module , and the calculation formula is as follows: .
[0047] The step 4 constructs a dual-branch encoder based on a modal cross-attention module (MCA) and a CNN-based decoder, and constructs a brain tumor segmentation model Dual-SwinTransBTS based on STFusion as shown in Figure 4 As shown, the model design steps are as follows:
[0048] Step 4.1: Build the framework of the proposed model, including the MCA dual-branch encoder, the STFusion feature fusion module, and the decoder.
[0049] Step 4.2: Divide the training set, validation set, and test set in the processed data set.
[0050] Step 4.3: Design a good training and prediction strategy, set the initial learning rate to 0.0001, the optimizer to Adam, and the BatchSize to 500. The loss function is selected as the Dice loss, and the calculation formula is as follows: Where T represents GroundTruth, and P represents the segmentation result.
[0051] Step 4.4: Input the segmented images in the validation set into the trained Dual-SwinTransBTS brain tumor segmentation network for segmentation, use the average Dice coefficient (DSC), Hausdorff distance (HD), and sensitivity (Sensitivity) to evaluate the model, and optimize the network parameters until the training reaches convergence, and obtain the final segmentation result. The calculation formulas of DSC, HD, and Sensitivity are as follows: Where A and B represent Ground Truth and prediction result, respectively. Where P is the predicted segmentation map, and G is the label. Where TP represents predicting positive samples as positive samples, and FN represents predicting positive samples as negative samples.
Claims
1. A brain tumor segmentation method based on Dual-SwinTransBTS, characterized in that, Includes the following steps: Step 1: Select a public dataset, divide the dataset into training, validation and test sets, and perform image augmentation on the images in the dataset to improve the model's segmentation performance; Step 2: Construct a multimodal cross-attention (MCA) module based on Swin-Transformer. First, replace the MLP module in Swin-Transformer with a pyramidal convolution (PyConv) module, and then use cross-modal attention (3D Cross-Modal Attention, CMA) to achieve cross-modal interaction, thereby capturing information at different scales in the image, which helps the model learn more global image features and improves the model's performance. Step 3: Construct the Swin-Transformer Feature FusionModule (STFusion). The STFusion module establishes global correlations between features at different scales through convolution and average pooling modules, which is used to fuse multi-scale features from the dual-branch encoder-decoder to obtain high-quality semantic segmentation performance. Step 4: Construct a dual-branch encoder based on modal cross-attention module (MCA) and a CNN-based decoder, and combine it with STFusion to build a brain tumor segmentation model, Dual-SwinTransBTS.
2. The method of brain tumor segmentation based on Dual-SwinTransBTS according to claim 1, wherein, The medical image segmentation dataset in step 1 is the BraTS2021 dataset; during preprocessing, all images are adjusted to a voxel size of 128×128×128.
3. The method of brain tumor segmentation based on Dual-SwinTransBTS according to claim 1, wherein, The specific process of constructing the Swin-Transformer-based Multimodal Cross-Attention (MCA) module in step 2 is as follows: Step 2.1: To consider both local and remote information simultaneously, the Swin-Tansformer and CNN are combined in the MCA module. The MCA block consists of two identical Swin-Tansformer branches, which are used to extract features from the two modalities respectively. Each branch consists of two cascaded modules, namely the self-modal module and the cross-modal module; Step 2.2: The self-modal module is used for feature extraction of each single modality and is a Swin-Transformer-CNN hybrid module. First, a Swin-Transformer is used to capture long-range information. Then, a Pyramidal Convolution (PyConv) layer is used instead of the MLP layer to extract multi-scale information. Taking inputs T1 and T1 Gd as an example, the calculation formula for the self-modal module is: wherein, , respectively represent the input features of the MCA module; , respectively represent the outputs of the normalization layer and the multi-head self-attention module; , respectively represent the outputs of the normalization layer and the pyramid convolution module; Step 2.3: The cross-modal module employs a sliding window self-attention Swin-Transformer, replacing the MLP layer with PyConv. It exchanges information between two related modalities through Cross-Modal Attention (CMA). The calculation formula for the cross-modal attention part is as follows: wherein, , respectively represent , the output of the cross-modal attention (CMA) module. The final output after passing through the cross-modal module is calculated as follows: in, , These represent the outputs after window-shifting multi-head self-attention and cross-modal attention modules, respectively. , These represent the final outputs of the MCA module.
4. The brain tumor segmentation method based on Dual-SwinTransBTS according to claim 1, characterized in that, Step 3, which involves constructing the Swin-Transformer Feature Fusion Module (STFusion), specifically includes the following steps: Step 3.1: Since the encoder is a hybrid model composed of Swin-Transformer and CNN, and the decoder is a purely CNN-based design, a Swin-Transformer feature fusion module was designed to bridge the semantic gap between the features of the encoder and decoder, fusing features from the MCA module and the decoder. Step 3.2: Let the outputs of the MCA module and the decoder be respectively... , The output after STFusion module The calculation is as follows: in, and These represent the tensors of the two branches in the dual-branch hybrid encoder, respectively. Step 3.3: In the STFusion module, since both the input and feature tensors are three-dimensional volumes, in order to fully utilize the three-dimensional information, [the following steps are taken]. and Average pooling is performed in three directions, and the calculation formula is as follows: The vectors in the three directions are reshaped into the same shape, concatenated, and then compressed through a 1 × 1 × 1 convolutional layer. This reshaping of the vectors back to their original dimensions yields new feature vectors in the three directions. and Next, features from the two branches are aggregated through convolution and concatenation operations, calculated as follows: The calibrated attention tensor A can be obtained through matrix multiplication of three vectors: Finally, the final output of the STFusion module is obtained by concatenating features from the MCA module and the convolutional module. The calculation formula is as follows: 。 5. The brain tumor segmentation method based on Dual-SwinTransBTS according to claim 1, characterized in that, The steps in step 4 of constructing the brain tumor segmentation model based on a modal cross-attention module (MCA) dual-branch encoder and a CNN-based decoder are as follows: Step 4.1: Build the framework of the proposed model, including the MCA dual-branch encoder, STFusion feature fusion module, and decoder; Step 4.2: Divide the processed dataset into training set, test set, and validation set; Step 4.3: Design a good training and prediction strategy, select a loss function and optimizer, and use the training samples selected and constructed in Step 4.2 to effectively train the Dual-SwinTransBTS network and calculate the loss. Step 4.4: Input the images to be segmented from the validation set into the trained Dual-SwinTransBTS network for segmentation. Use the validation set to evaluate the performance of the model, adjust the hyperparameters based on the performance, and optimize the parameters of the Dual-SwinTransBTS network until the training converges to obtain the final segmentation result.
6. The brain tumor segmentation method based on Dual-SwinTransBTS according to claim 5, characterized in that, Step 4.3 involves designing training and prediction strategies, selecting a loss function and optimizer, and using the training samples selected and constructed in Step 4.2 to effectively train the Dual-SwinTransBTS network, optimizing the parameters of the Dual-SwinTransBTS network until training converges. This specifically includes the following steps: Step 4.3.1: The necessary settings for the network's hyperparameters are as follows: set the initial learning rate to 0.0001, select Adam as the optimizer, and set the number of iterations to 500; Step 4.3.2: The loss function selected is Dice loss, and the calculation formula is as follows: In the formula, T represents Ground Truth and P represents the segmentation result.
Citation Information
Cited By
Double-branch Laplace gated graph polymerization multi-mode meningioma segmentation method
CN122116088A