A brain tumor segmentation method based on three-plane fusion edge U-Net

Abstract

The application discloses a brain tumor segmentation method based on a three-plane fusion edge U-Net, and comprises the following steps: dividing the obtained brain tumor MR image into a training set and a verification set, and pre-processing the training set image; based on the U-Net architecture, performing three-plane segmentation on the brain tumor MR image, taking an edge auxiliary module as a skip connection to construct an edge U-Net model, and fusing the extracted image features and edge features; adopting a combined loss function combining a cross-entropy loss function and a boundary loss function to train the edge U-Net model, and obtaining an optimal model; inputting the pre-processed brain tumor image dataset and the edge feature image dataset together as data of different planes into three network structures to train optimal parameters, and obtaining an optimal network segmentation result; and obtaining a three-dimensional probability graph according to the optimal segmentation result. The application can improve the segmentation precision and segment different parts of the whole tumor.

Description

Technical Field

[0001] This invention relates to the field of neural networks in deep learning, and more particularly to a brain tumor segmentation method based on a three-plane fused edge U-Net. Background Technology

[0002] Brain tumor segmentation is an important field in medical image processing, aiming to accurately segment brain tumors from medical images (MRI / CT). Achieving brain tumor segmentation not only accurately delineates the tumor portion but also greatly assists doctors in clinical diagnosis. With the continuous increase in medical image data, the workload of brain tumor image segmentation is constantly increasing. If batch, automated, and accurate segmentation of brain tumors could be achieved, it would significantly reduce the workload of expert physicians.

[0003] However, developing an automated segmentation program is not an easy task for current brain tumor segmentation, because lesions are defined based on intensity changes compared to surrounding tissues. Even segmentation performed by radiologists shows significant variations due to differences in partial volume effects, bias field artifacts, and intensity gradients between adjacent structures. Furthermore, tumor structures vary in expansion, location, and size among different patients, thus affecting strong prior segmentation of anatomical structures.

[0004] Manually segmenting brain tumors in MR medical images is a highly challenging task, both time-consuming and susceptible to evaluator variability. Therefore, reliable automatic and semi-automatic brain tumor segmentation techniques have garnered significant attention over the past two decades, resulting in hundreds of different algorithms. Currently, many expert physicians still rely on manual segmentation of brain tumors. Different subregions of gliomas vary considerably in appearance and shape due to their biological conditions. Accurate segmentation of brain tumors remains a critical challenge due to various difficulties such as location uncertainty, morphological uncertainty, low-contrast imaging, annotation bias, and data imbalance. Even in segmentation across different datasets by expert clinicians, discrepancies often arise among high-scoring individuals. Manually labeling tumor locations by medical experts is extremely time-consuming and may lack sufficient accuracy. Accurate tumor identification can help physicians implement more rational diagnostic and treatment strategies. Given the massive amount of data available for processing, a method for automatically segmenting individual brain tumor regions is needed. Due to the powerful performance of deep learning methods, many deep learning-based approaches have been applied to brain tumor segmentation, automatically extracting feature representations and achieving accurate and stable performance. In particular, the emergence of the U-Net network framework has made medical image segmentation extremely popular. Other network models segment the tumor as a whole. However, for doctors who need to study lesions in different tumor regions, an overall network architecture is needed to segment tumors in different locations separately. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a brain tumor segmentation method based on a three-plane fused edge U-Net that can improve the segmentation boundary of brain tumors and has good segmentation performance.

[0006] Technical solution: The brain tumor segmentation method of the present invention includes the following steps:

[0007] S1. The acquired brain tumor MR images are divided into training set and validation set, and the training set images are preprocessed to obtain preprocessed brain tumor image dataset A and edge feature image dataset B.

[0008] S2, an encoder and decoder based on the U-Net architecture, performs three-plane segmentation on brain tumor MR images, constructs an edge U-Net model with edge auxiliary modules as skip connections, and fuses the extracted image features and edge features.

[0009] S3 uses a combined loss function that combines the cross-entropy loss function and the boundary loss function to train the edge U-Net model, optimize the parameters of the edge U-Net model, and obtain the best model;

[0010] S4. The preprocessed brain tumor image dataset A and the edge feature image dataset B are used as data for different planes and are input into the three network structures of the edge U-Net network model to train the optimal parameters and obtain the optimal network segmentation result; the three-dimensional probability map is obtained based on the optimal segmentation result.

[0011] Furthermore, in step S1, the preprocessing steps for the acquired brain tumor MR images are as follows:

[0012] S11, Obtain brain tumor case image datasets from publicly available data sources;

[0013] S12, the brain tumor case image dataset is denoised and edge extracted using Gaussian filtering;

[0014] S13 performs slicing operations on the three planes of axial, sagittal and coronal directions, and adjusts them to appropriate dimensions as input.

[0015] Further, in step S2, the encoder includes an image partial encoder and an edge partial encoder. The image partial encoder includes five convolutional blocks: the first convolutional block contains three 64-channel convolutional layers; the second convolutional block contains three convolutional layers with a total of 128 channels; the third convolutional block consists of four convolutional layers, each with 256 channels; the fourth convolutional block consists of four convolutional layers with 512 channels; and the last convolutional block includes two convolutional layers, each with 1024 channels, and a dropout layer. After each convolutional block, a max pooling layer is added.

[0016] The edge encoder consists of four convolutional blocks. The first convolutional block contains three 64-channel convolutional layers; the second convolutional block contains three convolutional layers with a total of 128 channels; the third convolutional block consists of three convolutional layers, each with 256 channels; and the fourth convolutional block consists of three convolutional layers with 512 channels. An average pooling operation is added after each convolutional block.

[0017] Furthermore, in step S2, the decoder includes four convolutional blocks, each convolutional block including four convolutional layers, two layers with a kernel size of 2×2 and a ReLU activation function, and two layers with a 3×3 convolutional layer and a ReLU activation function;

[0018] The decoder fuses the four outputs to generate four features with different numbers of channels for image edges at different scales.

[0019] Furthermore, in step S2, the edge assist module consists of two convolutional layers. The first convolutional layer uses edge mapping, with a kernel size of 3×3, a sigmoid activation layer, and a stride of 1. The second convolutional layer has a kernel size of 3×3, a ReLU activation layer, and a stride of 1.

[0020] In the edge-assistance module, fusion M i and The feature, the expression for the edge-aided module, is as follows:

[0021] α = M i ⊙ρ

[0022]

[0023] in, It is a boundary feature, M i α represents the features of the image itself; α and ρ represent the attributes generated by the image encoder and edge encoder, respectively; and δ represents the convolution operation.

[0024] Furthermore, in step S3, the combined loss function expression of cross-entropy loss CE and boundary loss function ML is as follows:

[0025] L = CE + ML

[0026] The function definition of cross-entropy loss is:

[0027]

[0028] The boundary loss function is defined as follows:

[0029]

[0030] Where, p c (t) represents the output of the soft-max layer, and C is the number of classes in the label; y c (t)∈{0,1} represents the binary value of each class at voxel t; t is the position of the voxel; j is the index value, S j This represents the significant difference between the actual ground value and the predicted value. This represents the IOU between the actual boundary and the estimated boundary.

[0031] Furthermore, in step S4, the specific implementation steps for obtaining the three-dimensional probability map based on the optimal segmentation result are as follows: different tumor parts are divided according to the threshold, and the segmented tumors are then post-processed to obtain the axial, sagittal, and coronal segmentation results. The axial, sagittal, and coronal segmentation results are then assembled into a three-dimensional probability map. The size of each three-dimensional probability map is then adjusted to the original size, and finally the three-dimensional maps are averaged to obtain the final probability map.

[0032] Compared with the prior art, the significant advantages of this invention are as follows:

[0033] 1. This invention constructs a deep learning model based on the U-Net network structure, mainly focusing on the boundary of brain tumors, and constructs an edge U-Net model to achieve more accurate tumor segmentation of complex edges, including location and shape;

[0034] 2. Based on the three-plane fusion edge U-Net model, an edge assist module was designed to fuse extracted image features and boundary features. The results show that the edge assist module provides richer boundary semantic information, improves tumor localization detection and shape recognition capabilities, and thus obtains better segmentation results.

[0035] 3. The edge U-Net model with three planes in this invention can better obtain tumor information from three planes. The segmentation results from the three planes can more fully reflect the tissue contrast information, thereby improving the segmentation accuracy. Furthermore, different parts of the whole tumor can be segmented by using a threshold method.

[0036] 4. This invention introduces a loss function that combines cross-entropy loss and boundary loss to identify better boundaries, thereby obtaining better pixel values ​​and improving the overall performance of the model. Attached Figure Description

[0037] Figure 1 This is a flowchart of the present invention;

[0038] Figure 2 This is a diagram of the edge U-Net network model structure.

[0039] Figure 3 This is a schematic diagram of the edge assist module;

[0040] Figure 4 This is a schematic diagram of the multi-threshold edge U-Net of the present invention. Detailed Implementation

[0041] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0042] like Figure 1 The flowchart shown is a process for brain tumor segmentation based on a three-plane fused edge U-Net according to the present invention. The specific implementation steps are as follows:

[0043] Step 1: Obtain the dataset and perform preprocessing.

[0044] This invention uses the MR image dataset published at BraTs2019, which includes 335 training cases and 125 validation cases (without segmentation labels), but no test set. The training dataset is divided into two types of cases: HGG and LGG. Each case has four modalities (T1, T2, Flair, and T1ce), and the size of the obtained MR image dataset is 240×240×155.

[0045] First, Gaussian filtering is used to denoise the acquired MR image dataset, resulting in a preprocessed brain tumor image dataset A. Then, tumor edges are extracted, yielding an edge feature image dataset B. Next, both the preprocessed brain tumor image dataset A and the edge feature image dataset B are sliced, with the slice sizes adjusted to fit the model's input size of 256×256. In essence, this process involves cropping and resizing the preprocessed brain tumor dataset A and the edge feature image dataset B in three directions (axial, sagittal, and coronal) to provide data for use as input.

[0046] Step 2: Build the network architecture

[0047] A brain tumor segmentation method based on a three-plane architecture edge U-Net (MA-Unet) structure is constructed. An edge-aiding module is designed to fuse extracted image features and boundary features. The edge U-Net primarily focuses on the tumor boundary in MR images. For example... Figure 2 As shown, the specific implementation process is as follows:

[0048] The basic U-Net architecture consists of two branches: an encoder branch and a decoder branch. An edge U-Net network model is then proposed based on the segmentation of three planes in an MRI dataset. Using three planes from a preprocessed brain tumor image dataset A and edge feature image data B as input, an edge U-Net network model is constructed for segmentation. The model is then evaluated based on the segmentation results, and post-processing operations are performed to achieve the tumor segmentation path.

[0049] The input to the edge U-Net network model is divided into two parts: a preprocessed brain tumor dataset A and an edge feature image dataset B. In this embodiment, a 256×256 image is used as the input image in the edge U-Net network model. The input image is progressively filtered and downsampled (downsampling factor of 2) after each encoder block. After the last encoder block, the output size is reduced to 1 / 16 of the input image, that is, 1024×16×16 is the dimension of the feature map after the encoder block.

[0050] The encoder section includes an image partial encoder and an edge partial encoder. The image partial encoder consists of five convolutional blocks, with an input image size of 256×256. Each convolutional block is designed to extract features specific to a given input scale. The first convolutional block contains three 64-channel convolutional layers. The second convolutional block contains three convolutional layers with a total of 128 channels. The third convolutional block consists of four convolutional layers, each with 256 channels. The fourth convolutional block consists of four convolutional layers with 512 channels. The last convolutional block consists of two convolutional layers, each with 1024 channels, and a dropout layer with a factor of 0.5. All convolutional blocks have a kernel size of 3×3 and a stride of 1 for the ReLU activation function, except for the last convolutional block which has a kernel size of 1×1. After each convolutional block, a max-pooling layer with a kernel size of 2×2 and a stride of 1 is added. The edge partial encoder consists of four outputs that generate four features with different numbers of channels for image edges at different scales. The input is a 256×256 edge image, which should learn discriminative salient features. Each edge encoder branch has four blocks with feature maps of 64, 128, 256, and 512. Each layer is responsible for extracting unique features at a given input scale.

[0051] Decoder section: The decoder consists of 4 convolutional blocks. It concatenates images and edges of different scales in the image part and the edge part to form feature maps, and then performs upsampling. The decoder section also has 4 blocks with feature maps of 64, 128, 256 and 512. Each convolutional block includes two convolutional layers with a kernel size of 2×2 and a ReLU activation function, as well as two 3×3 convolutional layers and ReLU activation functions. After four passes, the original output image can be obtained.

[0052] Information loss is a problem faced by the decoder. To connect encoder and decoder features, the U-Net model for segmentation tasks includes skip connections between the encoder and decoder networks. This embodiment utilizes an edge-guided module to identify and highlight boundary features in the input. Adding an edge-aided module as a skip connection addresses the decay and saturation problems in boundary feature extraction in the preprocessed brain tumor dataset A, improving the features extracted for each block and aiding in the recovery of the input image. Figure 3 As shown, the edge assistance module consists of two convolutional layers. The first convolutional layer uses edge mapping with a kernel size of 3×3, a sigmoid activation layer, and a stride of 1. The second convolutional layer has a kernel size of 3×3, a ReLU activation layer, and a stride of 1. To ensure that edge features are passed to the next layer, the edge assistance module is fused into layered features.

[0053] The expression for the edge-aid module is as follows:

[0054] α = M i ⊙ρ(1)

[0055]

[0056] in, For boundary features, M i These are features of the image itself; in the edge assistance module, M is fused. i and Features; α and ρ are the attributes generated by the image encoder and edge encoder, respectively, and δ is the convolution operation.

[0057] The edge assist module in this invention provides more boundary information, improving the ability to detect tumor location and shape recognition.

[0058] Based on different characteristics of tumor MR imaging, tumors can be divided into necrotic and non-enhanced portions, enhanced portions, and edematous portions. Different tumor sites are analyzed using a thresholding method and defined as follows:

[0059] f(x, y) > P (3)

[0060] Where x and y represent the coordinates of the threshold; P represents the threshold, which classifies voxels into different tumor parts to represent the segmentation of each lesion in the entire tumor.

[0061] like Figure 4 As shown, when combined with a threshold, the tumor is divided into three parts based on the defined tumor boundaries: necrosis plus non-enhancing, enhancing, and edematous parts.

[0062] Step 3: Train the model, optimize the parameters, and obtain the corresponding evaluation metrics.

[0063] A new loss function is introduced for edge deep learning models, which combines cross-entropy loss and boundary loss to better identify boundaries, thereby obtaining better pixel values ​​and thus better overall performance.

[0064] The primary goal of the loss function is to reduce its value to a minimum level that achieves accurate segmentation. The constructed loss function aims to increase the symmetric optimization minimum set between the predicted and actual boundaries.

[0065] L=CE+ML (4)

[0066] The cross-entropy loss function is defined as follows:

[0067]

[0068] Where, p c (t) represents the output of the soft-max layer, C is the number of classes in the label, and y c (t)∈{0,1} represents the binary value of each class at voxel t.

[0069] Entropy loss function is widely used in medical image segmentation. Cross-entropy function supports the completion of segmentation task by preserving the global optimization of the entire image.

[0070] The loss function defined at the boundary is as follows:

[0071]

[0072] S j This represents the significant difference between the actual ground value and the predicted value. This represents the IOU between the actual boundary and the estimated boundary.

[0073]

[0074] Where, p c (t) represents the output of the soft-max layer, C is the number of classes in the label, and y c (t)∈{0,1} represents the binary value of each class at voxel t; t is the position of the voxel, j is the index value, and S j This represents the significant difference between the actual ground value and the predicted value. This represents the Interchange of Value (IOU) between the true boundary and the estimated boundary. The boundary loss function clusters the portion at the boundary, while the cross-entropy loss function clusters the remaining portion of the tumor.

[0075] Step 4: Model Training and Post-processing

[0076] A complete three-plane fusion edge U-Net network structure was constructed and trained on it. The preprocessed brain tumor dataset A and the edge feature image dataset B were used as inputs to the three network structures of the edge U-Net network model, serving as data for different planes, to train the optimal parameters and obtain the optimal network segmentation results. Then, different tumor lesion parts were segmented based on thresholds. Finally, post-processing operations were performed on the segmented tumors to remove small false positives and scattered points at the edges. Axial, sagittal, and coronal segmentation results were obtained, and these results were then assembled into a three-dimensional probability map. The size of each three-dimensional probability map was adjusted to its original size, and finally, the three-dimensional maps were averaged to obtain the final probability map.

[0077] This invention is a method for brain tumor segmentation using a three-plane fusion edge U-Net based on the three-dimensional characteristics of MR images. It utilizes preprocessed image edge features plus an edge-aiding module to achieve tumor edge segmentation. The edge U-Net model mainly focuses on the edges, resulting in a higher segmentation result.

Claims

1. A three-plane fused edge U-Net based brain tumor segmentation method, characterized in that, The method comprises the following steps: S1, the acquired brain tumor MR image is divided into a training set and a validation set, and the training set image is preprocessed to obtain a preprocessed brain tumor image dataset A and an edge feature image dataset B; S2, based on the encoder and decoder of the U-Net architecture, the brain tumor MR image is segmented in the axial, sagittal and coronal directions, and an edge auxiliary module is used as a skip connection to construct an edge U-Net model, and the extracted image features and edge features are fused; S3, a combined loss function combining a cross-entropy loss function and a boundary loss function is used to train the edge U-Net model, optimize the parameters of the edge U-Net model, and obtain an optimal model; S4, the preprocessed brain tumor image dataset A and the edge feature image dataset B are used as data in different planes and input into three network structures of the U-Net network model to train the optimal parameters and obtain an optimal network segmentation result; and a three-dimensional probability map is obtained according to the optimal segmentation result; In step S2, the encoder comprises an image part encoder and an edge part encoder, the image part encoder comprises five convolutional blocks, the first convolutional block comprises three 64-channel convolutional layers; the second convolutional block comprises three convolutional layers with a total of 128 channels; the third convolutional block is composed of four convolutional layers, each convolutional layer has 256 channels; the fourth convolutional block is composed of four convolutional layers, and has 512 channels; and the last convolutional block comprises two convolutional layers, each convolutional layer has 1024 channels, and a dropout layer; after each convolutional block, a maximum pooling layer is added; The edge part encoder comprises four convolutional blocks, the first convolutional block comprises three 64-channel convolutional layers; the second convolutional block comprises three convolutional layers with a total of 128 channels; the third convolutional block is composed of three convolutional layers, each convolutional layer has 256 channels; the fourth convolutional block is composed of three convolutional layers, and has 512 channels; and after each convolutional block, an average pooling is added; The edge auxiliary module is composed of two convolutional layers, the first convolutional layer uses edge mapping, the kernel size is 3x3, the sigmoid activation layer and the step are both 1; the second convolutional kernel size is 3x3, the ReLU activation layer and the step are both 1; In the edge-assisted module, the fusion M i and The characteristic, the expression of the edge-assisted module is as follows: α = M i ⊙ p wherein, is a boundary feature, M i is a feature of the image itself; a, p are attributes produced by the image part encoder and the edge part encoder, respectively, and d is a convolution operation; In step S3, the combined loss function expression of the cross-entropy loss CE and the boundary loss function ML is as follows: L = CE + ML The function definition of the cross-entropy loss is as follows: The boundary loss function is defined as follows: where p c (t) denotes the output of the soft-max layer, C is the number of classes in the label; y c (t) ∈ {0, 1} denotes the binary value for each class at voxel t; t is the position of the voxel; j is the index value, S j denotes the loss between the ground truth and the predicted saliency, denotes the IOU between the real boundary and the estimated boundary.

2. The method of claim 1, wherein the three-plane fused margin U-Net based brain tumor segmentation method is characterized by, In step S1, the implementation steps of preprocessing the acquired brain tumor MR image are as follows: S11, acquiring brain tumor case image dataset from a public data source; S12, performing denoising processing and edge extraction on the brain tumor case image dataset through Gaussian filtering; S13, performing slice processing operation on the axial, sagittal and coronal planes, and adjusting the slice size to the input size of the appropriate model.

3. The method of claim 1, wherein the three-plane fused margin U-Net based brain tumor segmentation method is characterized by, In step S2, the decoder comprises four convolutional blocks, each convolutional block comprises four convolutional layers, two layers of kernel size are 2x2 and one ReLU activation function, and two layers of 3x3 convolutional layers and ReLU activation function; The decoder fuses the four outputs into four features with different channel numbers for image edge generation at different scales.

4. The method of claim 1, wherein the three-plane fused margin U-Net based brain tumor segmentation method is characterized by, In step S4, the specific implementation steps of obtaining the three-dimensional probability map according to the optimal segmentation result are as follows: different tumor parts are divided according to a threshold, and finally post-processing operation is performed on the divided tumor to obtain the segmentation results in the axial direction, the sagittal direction and the coronal direction, and then the segmentation results in the axial direction, the sagittal direction and the coronal direction are assembled into a three-dimensional probability map. Then, the size of each three-dimensional probability map is adjusted to the original size, and finally the three-dimensional maps are averaged to obtain the final probability map.

Images