Brain tumor image segmentation method based on physical information

Abstract

The invention discloses a brain tumor image segmentation method based on physical information, and belongs to the field of medical image segmentation. Comprising the following steps: extracting image features: firstly, carrying out dimensionality reduction on an image, and then coding feature vectors to obtain the features of a sequence; adaptive feature fusion: after sequence features are spliced, channel attention is used to distinguish importance of each channel, space attention is used to make a model pay attention to important areas in a feature map, and unique fusion features suitable for different subtasks are obtained; the three-branch multi-task decoder is used for reserving the stage characteristics of the encoder through multi-stage up-sampling and jump connection and considering the loss function of each sub-task, so that the accuracy of different segmentation tasks is improved; and edge feature enhancement: obtaining an average curvature value of each feature channel by using a method based on an average curvature, selecting a plurality of channels with the highest average curvature value in the features, and combining the selected features with the original features to enhance the edge features. According to the invention, the precision of brain tumor image segmentation is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image segmentation, and in particular to a brain tumor image segmentation method based on physical information Background Art

[0002] The brain, a vital organ that oversees all bodily functions and is a central component of the nervous system, can be affected by one of the most lethal brain diseases: brain tumors. Early-stage brain tumors have significantly higher treatment success rates than late-stage ones, making timely detection crucial for subsequent treatment. Brain tumor image segmentation is a crucial task in medical image analysis. Using automated methods for brain tumor segmentation to assist medical professionals in diagnosing brain tumors and determining subsequent treatment plans for patients is highly valuable.

[0003] Current brain tumor image segmentation methods can be primarily categorized into traditional methods and deep learning-based methods. Traditional methods have the advantages of simplicity and computational speed, but they struggle to achieve improved accuracy. Due to the high variability in image intensity, these techniques are ineffective for all MR image types. Compared to traditional methods, deep learning-based methods can solve the problem on an end-to-end basis, segmenting MR images with significantly improved segmentation accuracy. In brain tumor segmentation, three regions of varying sizes are typically required: 1. The entire tumor region (WT) (including the necrotic tumor core, enhancing tumor, and peritumoral edema); 2. The tumor core (TC) (including the necrotic tumor core and enhancing tumor region); and 3. The enhancing tumor (ET). Common magnetic resonance imaging (MRI) sequences include T1-weighted (T1), T2-weighted (T2), T1-enhanced (T1CE), and fluid-attenuated inversion recovery (FLAIR). Different sequence features contribute differently to the segmentation of different brain tumor subregions. However, current methods ignore the unique characteristics of different brain tumor subregions, and few methods quantitatively distinguish the contributions of different sequences to segmentation. Therefore, different sequence features are required for different brain tumor subregions. Summary of the Invention

[0004] The purpose of this invention is to propose a physical information-based brain tumor image segmentation method that incorporates a variety of clinical knowledge into the model, extracts contextual representations specific to brain tumor subregions from features of different sequences, and finally fuses these representations to achieve better segmentation results. This invention is achieved through the following technologies:

[0005] S1. Sequence Feature Extraction Encoder: The encoder is responsible for capturing contextual information and extracting features from the input MRI image. First, the Patch Embedding layer converts the original 2D image into a series of 1D patches. Then, the Spatial Reduction Self-Attention (SRSA) module is used to encode the embedded feature vectors to initially obtain contextual features for different sequence images.

[0006] S2, Adaptive Feature Fusion Module: This module aims to fuse information from different MRI sequences to form multi-scale features that are shared across multiple tasks. It first concatenates features from different sequences, then uses channel attention to learn weight coefficients for each channel, enabling the model to distinguish the importance of different channels. Spatial attention is also used to direct the model's attention to important regions in the feature map, reducing the introduction of noise and ultimately generating unique fused features suitable for different subtasks.

[0007] S3, three-branch multi-task decoder: This decoder introduces the multi-task concept and splits the original brain tumor segmentation task into three subtasks: the entire tumor region (WT), the tumor core region (TC), and the enhanced tumor region (ET). During the model training process, the loss function of each subtask is considered to improve the accuracy of different segmentation tasks.

[0008] S4, Edge Feature Enhancement Module: This module is primarily designed to enhance the edge features of the ET region, thereby improving segmentation accuracy. To better capture the information content of each feature channel, a mean curvature-based calculation method is first introduced to obtain the mean curvature value of each channel. The channels with the highest mean curvature values are then selected and combined with the original features.

[0009] S5, output layer: This layer mainly converts the three different feature representations obtained into the final segmentation results.

[0010] The present invention has the following beneficial effects:

[0011] This is a comprehensive framework tailored for the task of brain tumor segmentation. The model integrates the unique physical properties of MRI within a multi-task learning framework to improve the robustness of the model and segmentation accuracy by better capturing tumor heterogeneity. In order to extract the features of specific modalities separately, a multi-branch network architecture was developed to minimize the risk of interference from less relevant MRI modalities in the segmentation of specific tumor sub-regions. In order to effectively merge multimodal MRI data into multi-scale features, an adaptive feature fusion (AFF) module was designed to obtain fused features suitable for different brain tumor sub-regions to ensure richer feature representation. In order to meet the segmentation needs of different sub-regions, a three-minute multi-task decoder was designed. During the model training process, the loss function of each small task was considered to improve the accuracy of different segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below.

[0013] Figure 1 The overall flow chart of the method of the present invention is

[0014] Figure 2 This is a model framework diagram of the method described in the present invention

[0015] Figure 3 This is the structural diagram of the adaptive feature fusion module of the present invention

[0016] Figure 4 This is the experimental result diagram for verifying the prior knowledge of the present invention.

[0017] Figure 5 This is a specific flow chart of the edge feature enhancement module of the present invention DETAILED DESCRIPTION

[0018] To more clearly illustrate the present invention and the technical solutions in the prior art, the present invention will be further described below with reference to the accompanying drawings and specific embodiments. Obviously, the embodiments described are only a portion of the present invention, not all of it. Based on the embodiments of the present invention, any other embodiments derived by those skilled in the art without inventive effort are considered within the scope of protection of the present invention. The following embodiments are intended only to illustrate the present invention and are not intended to limit its scope.

[0019] like Figure 1 and Figure 2 As shown in FIG, the brain tumor image segmentation method based on physical information includes the following steps:

[0020] S1, Sequence Feature Extraction Encoder: For a given feature X of size H×W×C, where H, W, and C represent the height, width, and number of channels respectively. First, the image is divided into HW / P 2 Then perform the Patch Embedding operation, and the process can be expressed as follows:

[0021] α=Norm(XW T +b) (1)

[0022] β=Reshape(α)+P (2)

[0023] Among them, Norm represents layer normalization, Reshape(.) represents the deformation operation, and W,b are learnable parameters.

[0024] The encoder consists of two residual networks. The first residual network can be expressed as:

[0025] γ=SRA(Norm(β))+β (3) where SRA stands for spatial reduced attention and is expressed as:

[0026] SRA(Norm(β))=Concat(h1,h2,…,h k )W O (4)

[0027] h i Expressed as:

[0028]

[0029] Here, Concat(·) represents the concatenation operation, and SR(·) is an operation for reducing the spatial dimension of the input sequence, which is expressed as:

[0030] SR(Norm(β))=Norm(Reshape(Norm(β))W S ) (6)

[0031] The second residual network is defined as:

[0032] ρ=FFN(Norm(γ))+γ (7)

[0033] Where FFN(.) represents the feedforward network, which is given by:

[0034] FFN(Norm(γ))=Max(0;(Norm(γ))W A +b A )W B +b B +γ (8)

[0035] Finally, the output Y of the encoder is given by:

[0036] Y=Reshape(ρ) (9)

[0037] S2. Adaptive feature fusion module: This module aims to fuse information from different MRI sequences to form multi-scale features shared across multiple tasks. First, the sequence features obtained by the sequence feature extraction encoder are obtained, and the sequence features are fused. Then, the channel attention is used to learn the weight coefficients of each channel to enable the model to distinguish the importance of different channels. At the same time, spatial attention is used to make the model pay attention to important areas in the feature map, reduce the introduction of noise, and finally obtain unique fusion features suitable for different subtasks. The specific structure and operation process of this module are as follows Figure 3 shown.

[0038] First, the features of different sequences are spliced together, expressed as the following formula:

[0039]

[0040] in They represent the feature maps from T1, T1ce, T2 and FLAIR generated in the i-th stage respectively.

[0041] Next, a global average pooling operation is applied to reduce the spatial dimensions from H×W to 1×1, summarizing the global information of each feature map. The pooled result is then passed through two convolutional layers. The first convolution reduces the channel dimension by a factor of R, followed by a nonlinear operation (Leaky ReLU). The second convolutional layer restores the original dimension. The output is then passed through a sigmoid function to generate channel attention weights:

[0042]

[0043] Here, Conv() represents the convolution operation, Pooling() represents the pooling operation, and LeakyReLU() and Sigmoid() are activation functions.

[0044] The generated weight θ i It is used to recalibrate the original input feature map and emphasize important features. After performing element multiplication, the weighted feature map is obtained:

[0045]

[0046] Then use the convolution operation to fuse the multiple features obtained:

[0047]

[0048] Finally, element-wise feature augmentation is used to recalibrate the input by multiplying it with the weights generated by the average pooling operation. This process can be expressed as:

[0049]

[0050] Through three such different AFF modules, unique fusion features suitable for three sub-tasks are finally generated.

[0051] S3, the three-branch multi-task decoder is responsible for receiving fused features suitable for different tasks, and through multi-stage downsampling operations and skip connections, it retains the features obtained during the upsampling process and produces the final brain tumor segmentation results. After obtaining the fused features obtained by the AFF module, the decoder first reduces the dimension of the features through a downsampling operation:

[0052]

[0053] Among them represents the output of the AFF module, where X can represent WT, TC and ET respectively. Since the sensitivity of each brain tumor sub-region to each modality is different (such as Figure 4 As shown), the specific operation of each branch decoder can be expressed as follows:

[0054]

[0055] Among them, F 4-i represents the common features of the 4-i levels in the encoder, and represents the WT, TC and ET features at the decoder level i, and represents the WT, TC and ET features at the i+1th stage of the decoder.

[0056] The loss of the final model is composed of sub-losses for each task, as follows:

[0057] Loss total =λ1Loss WT +λ2Loss TC +λ3Loss ET . (19)

[0058] The sum of λ1, λ2, and λ3 is 1, and the specific values are adaptively determined during the training process.

[0059] S4, edge feature enhancement module: This module is mainly used to enhance the edge features of the ET area. In order to better obtain the amount of information in each channel of the feature, a curvature-based calculation method is first introduced, because the average curvature can well represent the amount of information contained in the image. The curvature value of each channel is obtained by convolution, and then sorted according to the size of the obtained curvature value. The information-rich channel is combined with the original feature to enhance the edge feature. The specific structure of the EFE module is as follows Figure 5 shown.

[0060] This module is based on the principles of information theory, where greater variability in eigenvalues indicates greater information content. While information entropy is a common measure of information, its calculation involves logarithmic operations, which can be time-consuming. Therefore, curvature was chosen as a computationally efficient alternative to measure information content. Curvature is calculated using a convolution operation with a predefined kernel, defined as:

[0061]

[0062] The feature enhancement module first calculates the curvature of each channel. Then, it selects the top KC channel with the highest curvature value, combines it with the original feature map and inputs it into the decoder, thereby enhancing the feature representation of the channel containing the most important information.

[0063] S5, the output layer will get three features X WT , X TC and X ET Converted into the final segmentation result, the specific process is shown in the following formula:

[0064] O i =Conv(Norm(Reshape(X i ))) (twenty one)

[0065] P i =Sigmoid(O i ) (twenty two)

[0066] The i here can be expressed as WT, TC and ET respectively, and Sigmoid() is the activation function.

[0067] This paper uses the most extensive BraTS datasets, BraTS2019 and BraTS2020. The BraTS dataset series is provided by the Medical Image Computing and Computer-Assisted Intervention Consortium (MICCAI) and uses data from patients in hospitals worldwide. The original data volume for each modality is 240 × 240 × 155 pixels, and 2D slices are generated by slicing in the axial plane. The slices are cropped to remove unused background areas, resulting in 224 × 224 pixel data blocks.

[0068] Table 1 Dataset distribution

[0069]

[0070] The experiments were implemented using PyTorch and MindSpore. The initial learning rate was set to 1e-2, and the poly strategy was used to adjust the learning rate of the SGD optimizer. The batch size for model training was set to 12, and the number of training epochs was 100. Furthermore, the experiments were conducted on a PyCharm platform running on Windows 10, using Cuda version 11.3, Python version 3.8, and an RTX8000 (48GB) GPU for acceleration.

[0071] The evaluation indicators of the present invention are Dice score, 95% Hausdorff distance (HD95) and Sensitivity

[0072]

[0073] in represents the predicted segmentation result (0 or 1), and y is the ground truth label (0 or 1). Here, i refers to the pixel index, N is the total number of pixels per sample, and M is the total number of samples.

[0074]

[0075] Let y∈Tr, Where Pr and Tr represent the predicted value and true value of each point respectively.

[0076]

[0077] Where TP and FN represent the total number of true positives and false negatives, respectively.

[0078] As shown in Tables 2 and 3, the experimental results obtained using this invention on two datasets are compared with previous works, most of which are retrieved from their respective papers. The best results are in bold and the second best results are underlined.

[0079] Table 2 BraTS2019 experimental results

[0080] Models Dice HD95 Sens WT TC ET Mean WT TC ET Mean WT TC ET Mean UNet 0.944 0.922 0.890 0.919 2.261 1.620 1.434 1.772 0.941 0.939 0.905 0.928 IVD-Net 0.928 0.902 0.870 0.900 2.340 1.670 1.430 1.810 0.917 0.923 0.882 0.907 UNet++ 0.945 0.924 0.894 0.921 2.251 1.603 1.371 1.742 0.940 0.930 0.904 0.925 AttentionU 0.947 0.926 0.896 0.923 <![CDATA[ 2.220 ]]> 1.580 <![CDATA[ 1.333 ]]> 1.711 0.940 0.931 0.898 0.923 TransUnet 0.947 0.932 0.896 0.925 2.261 1.580 1.382 1.741 0.951 0.939 0.909 0.933 UctransNet 0.949 0.934 0.903 0.929 2.209 <![CDATA[ 1.558 ]]> 1.329 <![CDATA[ 1.699 ]]> 0.944 0.938 0.909 0.930 <![CDATA[F 2 Net]]> <![CDATA[ 0.952 ]]> <![CDATA[ 0.941 ]]> <![CDATA[ 0.900 ]]> <![CDATA[ 0.931 ]]> 2.222 1.652 1.348 1.741 0.954 <![CDATA[ 0.949 ]]> <![CDATA[ 0.905 ]]> <![CDATA[ 0.936 ]]> SPA-Net 0.896 0.833 0.771 0.833 5.950 6.120 3.530 5.200 - - - - EA-DFFTU 0.902 0.815 0.737 0.818 4.900 4.600 5.400 4.967 <![CDATA[ 0.955 ]]> 0.814 0.713 0.827 <![CDATA[S 2 CA-Net]]> 0.910 0.844 0.801 0.852 4.001 5.860 3.177 4.346 - - - - Ours 0.952 0.944 0.904 0.933 2.180 1.540 1.337 1.690 0.960 0.951 0.898 0.932

[0081] Table 3 BraTS2020 experimental results

[0082] Models Dice HD95 Sens WT TC ET Mean WT TC ET Mean WT TC ET Mean UNet 0.943 0.922 0.896 0.920 2.735 1.970 2.30 2.348 0.941 0.932 0.906 0.926 IVD-Net 0.931 0.906 0.874 0.904 2.771 2.527 2.153 2.484 0.920 0.926 0.885 0.910 UNet++ 0.945 0.923 0.897 0.922 2.466 2.165 2.287 2.306 0.943 0.931 0.904 0.926 AttentionU 0.942 0.914 0.891 0.916 2.200 <![CDATA[ 1.534 ]]> 1.332 1.687 0.938 0.932 0.905 0.925 TransUnet 0.947 0.932 0.899 0.926 2.222 1.551 1.380 1.718 0.945 0.940 0.902 0.929 UctransNet 0.947 0.934 <![CDATA[ 0.904 ]]> 0.928 2.211 1.550 1.343 1.701 0.940 0.938 <![CDATA[ 0.913 ]]> 0.930 <![CDATA[F 2 Net]]> <![CDATA[ 0.951 ]]> <![CDATA[ 0.940 ]]> 0.902 <![CDATA[ 0.931 ]]> <![CDATA[ 2.195 ]]> 1.544 <![CDATA[ 1.313 ]]> <![CDATA[ 1.684 ]]> <![CDATA[ 0.953 ]]> 0.951 0.916 0.940 SPA-Net 0.900 0.832 0.778 0.837 4.600 6.550 32.20 14.45 - - - - EA-DFFTU 0.937 0.855 0.806 0.866 3.700 3.200 2.500 3.133 0.943 0.928 0.907 0.926 <![CDATA[S 2 CA-Net]]> 0.925 0.889 0.824 0.879 3.043 3.877 2.712 3.211 - - - - Ours 0.953 0.946 0.909 0.935 2.168 1.477 1.272 1.641 0.960 0.950 0.908 0.937

[0083] The performance of the present invention is better than a series of previous models, including F 2Net,EA-DFFTU,S 2 CA-Net, which shows the advantage of the proposed model framework. In the BraTS2019 dataset, compared with the second best model F 2 Compared with the best model F, the average Dice score of the present invention is improved by 0.2%, among which the ET region with the highest segmentation difficulty is improved by 0.4%. On the BraTS2020 dataset, the average Dice score of the present invention is better than that of the second best model F. 2 The Net improved by 0.4% and the ET area improved by 0.7%. Although the ET area's performance in terms of HD95 and Sensitivity is not as good as that of previous models, the average performance is better.

Claims

1. A brain tumor image segmentation method based on physical information, characterized by: The steps include: S1. Sequence Feature Extraction Encoder: The encoder is responsible for capturing contextual information and extracting features from the input MRI image. It first uses the Patch Embedding layer to convert the original 2D image into a series of 1D patches. It then uses the Spatial Reduction Self-Attention (SRSA) module to encode the embedded feature vectors to initially obtain contextual features for different sequence images. S2, Adaptive Feature Fusion Module: This module aims to fuse information from different MRI sequences to form unique fusion features across multiple tasks. It first concatenates features from different sequences, then uses channel attention to learn the weight coefficients of each channel to enable the model to distinguish the importance of different channels. It also uses spatial attention to focus the model on important areas in the feature map, reducing the introduction of noise and ultimately obtaining unique fusion features suitable for different subtasks. S3, Three-branch Multi-task Decoder: This decoder introduces a multi-task concept, splitting the original brain tumor segmentation task into three subtasks: the entire tumor region (WT), the tumor core region (TC), and the enhanced tumor region (ET). During model training, the loss function of each subtask is considered to improve the accuracy of different segmentation tasks. S4, edge feature enhancement module: This module is mainly used to enhance the edge features of the ET area, thereby improving its segmentation accuracy. In order to better obtain the information content of each feature channel, a calculation method based on average curvature is first used to obtain the average curvature value of each channel. Then, multiple channels with the highest average curvature value in the feature are selected and the selected features are combined with the original features. S5, output layer: This layer mainly converts the three different feature representations obtained into the final segmentation results.

2. The brain tumor image segmentation method based on physical information according to claim 1, characterized in that: The specific steps of step S1 are: For a given feature X of size H×W×C, where H, W, and C represent the height, width, and number of channels respectively, the image is first divided into HW / P 2 Then, the Patch Embedding operation is performed. The process can be expressed by the following formula: α=Norm(XW T +b) (1) β=Reshape(α)+P (2) Among them, Norm represents layer normalization, Reshape(.) represents the deformation operation, and W,b are learnable parameters; The encoder consists of two residual networks. The first residual network can be expressed as: γ=SRA(Norm(β))+β (3) Where SRA stands for spatial reduced attention, and the formula is: SRA(Norm(β))=Concat(h1,h2,…,h k )W O (4) h i Expressed as: Here, Concat(·) represents the concatenation operation, and SR(·) is an operation for reducing the spatial dimension of the input sequence, which is expressed as: SR(Norm(β))=Norm(Reshape(Norm(β))W S ) (6) The second residual network is defined as: ρ = FFN(Norm(γ)) + γ (7) where FFN(.) represents the feedforward network, which is given by: FFN(Norm(γ))=Max(0;(Norm(γ))W A +b A )W B +b B +g (8) Finally, the output Y of the encoder is given by: Y=Reshape(ρ) (9) where Reshape means changing the shape of the array.

3. The brain tumor image segmentation method based on physical information according to claim 1, characterized in that: The specific steps of step S2 are: First, the features of different sequences are spliced together, expressed as the following formula: in Represent the feature maps from T1, T1ce, T2 and FLAIR generated by the third stage of the encoder respectively; Next, a global average pooling operation is applied to reduce the spatial dimension from H×W to 1×1, thereby summarizing the global information of each feature map. The combined result is then passed through two convolutional layers. The first convolution reduces the channel dimension by a factor of R, followed by a nonlinear operation (Leaky ReLU). The second convolutional layer restores the original dimension, and then passes the output through a Sigmoid function to generate channel attention weights: Here Conv() represents the convolution operation, Pooling() represents the pooling operation, LeakyReLU() and Sigmoid() are activation functions, and the generated weight θ i Used to recalibrate the original input feature map, emphasize important features, and after performing element multiplication, obtain the weighted feature map: Then use the convolution operation to fuse the multiple features obtained: f i =Conv3(θ i ) (i=1,2,3) (13) Finally, element-wise feature augmentation is used to recalibrate the input by multiplying it with the weights generated by the average pooling operation. This process can be expressed as: Among them F i is the generated fusion feature.

4. The brain tumor image segmentation method based on physical information according to claim 1, characterized in that: The specific steps of step S3 are: The three-branch multi-task decoder is responsible for receiving fused features suitable for different tasks, and retains the features obtained during the upsampling process through multi-stage downsampling operations and skip connections to produce the final brain tumor segmentation results. After obtaining the fused features obtained by the AFF module, the decoder first reduces the number of feature channels through a downsampling operation: Among them Represents the output of the AFF module, where X can represent WT, TC and ET respectively. The specific operation of each branch decoder can be expressed as follows: Among them, F 4-i represents the common features of the 4-i levels in the encoder, and represents the WT, TC and ET features at the decoder level i, and represents the WT, TC and ET features at the i+1th level of the decoder; The loss of the final model is composed of sub-losses for each task, as follows: Loss total =λ1 Loss WT +λ2 Loss TC +λ3 Loss ET (19) The sum of λ1, λ2, and λ3 is 1, and the specific values are adaptively determined during the training process.

5. The brain tumor image segmentation method based on physical information according to claim 1, characterized in that: The specific steps of step S4 are: Among the three segmentation tasks of WT, TC, and ET, ET segmentation has the worst performance. To improve the accuracy of ET segmentation, the EFE module is used in the ET decoder. This module is based on the principles of information theory, where the greater the variability of the eigenvalues, the greater the amount of information. Although information entropy is a common information measure, its calculation involves logarithmic operations, which can be time-consuming. Finally, curvature was chosen as a computationally efficient alternative to measure information content. Curvature is calculated using a convolution operation with a predefined kernel, which is defined as: The feature enhancement module first calculates the curvature of each channel. Then, it selects the top KC channel with the highest curvature value, combines it with the original feature map and inputs it into the decoder, thereby enhancing the feature representation of the channel containing the most important information.

6. The brain tumor image segmentation method based on physical information according to claim 1, characterized in that: The specific steps of step S5 are: The output layer will get three features X WT , X TC and X ET Converted into the final segmentation result, the specific process is shown in the following formula: O i =Conv(Norm(Reshape(X i ))) (21) P i =Sigmoid(O i ) (22) The i here can be expressed as WT, TC and ET respectively, and Sigmoid() is the activation function.

Images