A multi-branch liver tumor segmentation method based on UNet

Abstract

The application discloses a multi-branch liver tumor segmentation method based on UNet, and comprises the following steps: first, a multi-branch liver tumor segmentation network based on UNet is constructed; then, a training sample set constructed by CT images in which liver tumors have been segmented in advance is used to perform deep learning training on the multi-branch liver tumor segmentation network based on UNet, so that a trained multi-branch liver tumor segmentation network based on UNet is obtained; finally, a CT image currently in need of liver tumor segmentation is sent to the trained multi-branch liver tumor segmentation network based on UNet, so that the CT image in which the liver tumor has been segmented is obtained. By strengthening the receptive field of the segmentation network feature map and fully utilizing the channel and spatial structure information, pixel-level details and spatial information can be better captured, so that the segmentation performance of the network on medical images is improved. Different from existing methods, the network architecture proposed in the application can capture fine details of local edges and global multi-scale information at the same time, so that spatial consistency is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image segmentation, in particular to a multi-branch liver tumor segmentation method based on UNet. BACKGROUND

[0002] Liver cancer is the most common and highest mortality tumor in the world, which seriously threatens people's life and health. According to the data of the National Cancer Center, the incidence of liver cancer in China ranks 5th among all malignant tumors, and the mortality rate ranks 2nd. With the development of science and technology, computer application technology and medical informatization level have been greatly developed, and medical image facilities have also been more perfect. Computed tomography (CT) has the characteristics of fast scanning time, high image resolution, etc., and is the current diagnosis method for liver lesions. At present, the segmentation of liver tumor in clinic is usually manually annotated by experienced doctors, which is not only time-consuming and laborious, but also the tumor area of the liver CT image of the same patient may cause different results under the annotation of different doctors, and is seriously dependent on the experience and skill of doctors. Therefore, it is of great significance to study accurate and efficient automatic liver tumor segmentation method for the clinical diagnosis and treatment of liver cancer.

[0003] Traditional medical image segmentation methods include threshold, level set, region growing, etc. However, due to the reasons that the liver tumor in the CT image usually has low contrast, fuzzy boundary, size, shape, position, number and other reasons, these traditional medical image segmentation methods need manual intervention, are difficult to effectively adapt to the complexity and diversity of liver tumors, have poor segmentation accuracy of the target, low efficiency, and cannot realize automatic segmentation of the tumor region. In recent years, deep learning technology has developed rapidly and is widely applied in the field of medical image segmentation. Ronneberger et al. proposed a U-shaped convolutional network (Unet), which first introduced a skip connection into the convolutional network. The network realizes end-to-end semantic segmentation of the image through encoding-decoding operation. The encoder performs down-sampling on the extracted features to capture the context information of the image, and the decoder performs up-sampling on the extracted features to accurately locate the segmentation region. Li et al. proposed a bottleneck supervised Unet model (BS-Unet). The model is a hybrid tight connection structure, which performs segmentation by fully utilizing the information between each layer of the network. Schlemper et al. incorporated attention mechanism into the Unet network and proposed an A-Unet (attention unet) model. The model can automatically learn the region features related to the segmentation task and suppress irrelevant features. Although these networks are widely used in the field of liver tumor segmentation, there are still problems such as inaccurate liver tumor boundary segmentation, difficulty in detecting small tumors, and difficulty in tumor segmentation caused by data class imbalance. SUMMARY

[0004] The present application solves the problem of existing deep learning networks having difficulty in liver tumor segmentation, and provides a multi-branch liver tumor segmentation method based on UNet.

[0005] To solve the above problems, the present application is realized by the following technical scheme:

[0006] A multi-branch liver tumor segmentation method based on UNet, comprising the following steps:

[0007] First, a multi-branch liver tumor segmentation network based on UNet is constructed;

[0008] The UNet-based multi-branch liver tumor segmentation network is composed of 1 input layer, 9 multi-branch modules, 4 maximum pooling layers, 4 up-sampling modules, 1 1x1 convolution layer and 1 output layer; the input of the input layer is taken as the input of the UNet-based multi-branch liver tumor segmentation network; the output of the input layer is connected to the input of the first multi-branch module; the output of the first multi-branch module is connected to the input of the first maximum pooling and the first input of the fourth up-sampling; the output of the first maximum pooling is connected to the input of the second multi-branch module; the output of the second multi-branch module is connected to the input of the second maximum pooling and the first input of the third up-sampling; the output of the second maximum pooling is connected to the input of the third multi-branch module; the output of the third multi-branch module is connected to the input of the third maximum pooling and the first input of the second up-sampling; the output of the third maximum pooling is connected to the input of the fourth multi-branch module; the output of the fourth multi-branch module is connected to the input of the fourth maximum pooling and the first input of the first up-sampling; the output of the fourth maximum pooling is connected to the input of the fifth multi-branch module; the output of the fifth multi-branch module is connected to the second input of the first up-sampling, and the output of the first up-sampling is connected to the input of the sixth multi-branch module; the output of the sixth multi-branch module is connected to the second input of the second up-sampling, and the output of the second up-sampling is connected to the input of the seventh multi-branch module; the output of the seventh multi-branch module is connected to the second input of the third up-sampling, and the output of the third up-sampling is connected to the input of the eighth multi-branch module; the output of the eighth multi-branch module is connected to the second input of the fourth up-sampling, and the output of the fourth up-sampling is connected to the input of the ninth multi-branch module; the output of the ninth multi-branch module is connected to the input of the 1x1 convolution layer, and the output of the 1x1 convolution layer is connected to the input of the output layer; the output of the output layer is taken as the output of the UNet-based multi-branch liver tumor segmentation network; then, a training sample set is constructed by using CT images in which liver tumors have been segmented in advance, and the UNet-based multi-branch liver tumor segmentation network constructed is subjected to deep learning training by using the training sample set, to obtain a trained UNet-based multi-branch liver tumor segmentation network; finally, a CT image currently in need of liver tumor segmentation is sent to the trained UNet-based multi-branch liver tumor segmentation network, to obtain a CT image in which a liver tumor has been segmented.

[0009] In the above scheme, the multi-branch module is composed of two 3*3 convolution layers with a hollow rate of 2, two 3*3 convolution layers with a hollow rate of 4, two maximum pooling layers with a convolution kernel of 2, two feature addition layers, two batch normalization layers and two activation layers; the inputs of the first 3*3 convolution layer with a hollow rate of 2, the first 3*3 convolution layer with a hollow rate of 4 and the first maximum pooling layer with a convolution kernel of 2 jointly form the input of the multi-branch module; the outputs of the first 3*3 convolution layer with a hollow rate of 2, the first 3*3 convolution layer with a hollow rate of 4 and the first maximum pooling layer with a convolution kernel of 2 are simultaneously connected to the inputs of the feature addition layer, the output of the feature addition layer is connected to the input of the first batch normalization layer, and the output of the first batch normalization layer is connected to the input of the first activation layer; the output of the first activation layer is simultaneously connected to the inputs of the second 3*3 convolution layer with a hollow rate of 2, the second 3*3 convolution layer with a hollow rate of 4 and the second maximum pooling layer with a convolution kernel of 2; the outputs of the second 3*3 convolution layer with a hollow rate of 2, the second 3*3 convolution layer with a hollow rate of 4 and the second maximum pooling layer with a convolution kernel of 2 are simultaneously connected to the inputs of the feature addition layer, the output of the feature addition layer is connected to the input of the second batch normalization layer, and the output of the second batch normalization layer is connected to the input of the second activation layer; the output of the second activation layer forms the output of the multi-branch module.

[0010] In the above scheme, the upsampling module is composed of one bilinear interpolation layer and one splicing layer; the input of the bilinear interpolation layer is the first input of the upsampling module, the output of the bilinear interpolation layer is connected to one input of the splicing layer, the other input of the splicing layer is the second input of the upsampling module, and the output of the splicing layer is the output of the upsampling module.

[0011] Compared with the prior art, the present application can better capture pixel-level details and spatial information by strengthening the receptive field of the segmentation network feature map and fully utilizing the channel and spatial structure information, thereby improving the segmentation performance of the network on medical images. Unlike existing methods, the network architecture proposed in the present application can capture fine details of local edges while also capturing global multi-scale information, thereby ensuring spatial consistency. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 It is a structural diagram of the multi-branch liver tumor segmentation network based on UNet.

[0013] Figure 2 It is a structural diagram of the multi-branch module

[0014] Figure 3 It is a schematic diagram of the dilated convolution.

[0015] Figure 4 It is a structural diagram of the upsampling module. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below with specific examples.

[0017] A UNet-based multi-branch liver tumor segmentation method, comprising the following steps:

[0018] 1) Construct a UNet-based multi-branch liver tumor segmentation network.

[0019] The UNet-based multi-branch liver tumor segmentation network is composed of 1 input layer, 9 multi-branch modules, 4 maximum pooling layers, 4 up-sampling modules, 1 1x1 convolution layer and 1 output layer, as shown in Figure 1 The input of the input layer serves as the input of the UNet-based multi-branch liver tumor segmentation network. The output of the input layer is connected to the input of the first multi-branch module. The output of the first multi-branch module is connected to the input of the first maximum pooling and the first input of the fourth up-sampling. The output of the first maximum pooling is connected to the input of the second multi-branch module. The output of the second multi-branch module is connected to the input of the second maximum pooling and the first input of the third up-sampling. The output of the second maximum pooling is connected to the input of the third multi-branch module. The output of the third multi-branch module is connected to the input of the third maximum pooling and the first input of the second up-sampling. The output of the third maximum pooling is connected to the input of the fourth multi-branch module. The output of the fourth multi-branch module is connected to the input of the fourth maximum pooling and the first input of the first up-sampling. The output of the fourth maximum pooling is connected to the input of the fifth multi-branch module. The output of the fifth multi-branch module is connected to the second input of the first up-sampling, and the output of the first up-sampling is connected to the input of the sixth multi-branch module. The output of the sixth multi-branch module is connected to the second input of the second up-sampling, and the output of the second up-sampling is connected to the input of the seventh multi-branch module. The output of the seventh multi-branch module is connected to the second input of the third up-sampling, and the output of the third up-sampling is connected to the input of the eighth multi-branch module. The output of the eighth multi-branch module is connected to the second input of the fourth up-sampling, and the output of the fourth up-sampling is connected to the input of the ninth multi-branch module. The output of the ninth multi-branch module is connected to the input of the 1x1 convolution layer, and the output of the 1x1 convolution layer is connected to the input of the output layer. The output of the output layer serves as the output of the UNet-based multi-branch liver tumor segmentation network.

[0020] The UNet-based multi-branch liver tumor segmentation network constructed by the present application mainly consists of an encoder, a decoder and a 1x1 convolution. The encoding path and the decoding path are symmetrical to each other. The encoding path is composed of a multi-branch structure and a pooling structure, which is used to obtain context information, i.e. local features. The decoding path is composed of a multi-branch structure and an up-sampling structure, which is used for accurate positioning. The down-sampling depth of the encoding is 5, and the up-sampling depth of the decoding is 5.

[0021] The multi-branch structure, i.e. the multi-branch module, can fuse multi-scale features and better obtain context information, and its structure is as shown inFigure 2 The multi-branch structure introduces a cavity convolution with different cavity rates. Without increasing the computational cost, the receptive field is better than the ordinary convolution, and the receptive field calculation is shown in the following formula, which efficiently extracts the features of the image.

[0022] N = k + (k - 1) (d - 1)

[0023] Where N is the receptive field range, k is the original convolution kernel size, and d is the cavity rate. For a convolutional deep network, the larger the receptive field, the more global information it contains, effectively reducing the loss of spatial information. When k = 3 and d = 1, the receptive field N is 3, which is a normal 3x3 convolution. When k = 3 and d = 2, the receptive field N is 5, which is a standard 5x5 convolution.

[0024] At the same time, in order to avoid the cavity rate of the cavity convolution being too large, resulting in too few effective neurons in the convolution kernel, the multi-branch structure only applies the cavity convolution with a cavity rate of 1 and 2. The multi-branch structure introduces a max-pooling layer, which reduces information redundancy, extracts effective semantic information in the image, and further fuses the information features extracted by the cavity convolution; secondly, it strengthens the invariance of image features, making it more robust to image shifts, rotations, etc.; thirdly, it prevents overfitting during training due to multiple convolutions.

[0025] The multi-branch module is composed of two 3x3 convolution layers with a hollow rate of 2, two 3x3 convolution layers with a hollow rate of 4, two maximum pooling layers with a convolution kernel of 2, two feature addition layers, two batch normalization layers and two activation layers. The input of the first 3x3 convolution layer with a hollow rate of 2, the first 3x3 convolution layer with a hollow rate of 4 and the first maximum pooling layer with a convolution kernel of 2 forms the input of the multi-branch module. The output of the first 3x3 convolution layer with a hollow rate of 2, the first 3x3 convolution layer with a hollow rate of 4 and the first maximum pooling layer with a convolution kernel of 2 is connected to the input of the feature addition layer, the output of the feature addition layer is connected to the input of the first batch normalization layer, and the output of the first batch normalization layer is connected to the input of the first activation layer. The output of the first activation layer is connected to the input of the second 3x3 convolution layer with a hollow rate of 2, the second 3x3 convolution layer with a hollow rate of 4 and the second maximum pooling layer with a convolution kernel of 2. The output of the second 3x3 convolution layer with a hollow rate of 2, the second 3x3 convolution layer with a hollow rate of 4 and the second maximum pooling layer with a convolution kernel of 2 is connected to the input of the feature addition layer, the output of the feature addition layer is connected to the input of the second batch normalization layer, and the output of the second batch normalization layer is connected to the input of the second activation layer. The output of the second activation layer forms the output of the multi-branch module. The multi-branch module aggregates receptive fields of different scales, and the input feature maps are respectively subjected to 3x3 dilated convolution with a hollow rate of 2, 3x3 dilated convolution with a hollow rate of 4 and maximum pooling operation with a convolution kernel of 2, and then subjected to feature addition as the output of the multi-branch module, so as to solve the problem of different scales of the same segmentation target. As shown in Figure 3 , the dilated convolution can expand the receptive field without losing resolution, which can detect large targets such as liver and accurately locate small targets such as tumors.

[0026] The pooling structure is a maximum pooling layer, which adopts a 2x2 maximum pooling operation to reduce the size of the feature map. The up-sampling structure is an up-sampling module, and its structure is shown in Figure 4 . The up-sampling module is composed of one bilinear interpolation layer and one splicing layer. The input of the bilinear interpolation layer is the first input of the up-sampling module, the output of the bilinear interpolation layer is connected to one input of the splicing layer, the other input of the splicing layer is the second input of the up-sampling module, and the output of the splicing layer is the output of the up-sampling module. The bilinear interpolation is used to expand the image size, and the expanded feature map is spliced with the corresponding feature map in the encoding path to achieve better feature reconstruction effect. Finally, 1x1 convolution is used to restore the size of the input network feature map.

[0027] 2) Use the CT image in which the liver tumor has been segmented in advance to construct a training sample set, and use the training sample set to perform deep learning training on the constructed UNet-based multi-branch liver tumor segmentation network to obtain the trained UNet-based multi-branch liver tumor segmentation network.

[0028] 3) The CT image currently requiring liver tumor segmentation is sent to the trained UNet-based multi-branch liver tumor segmentation network to obtain the CT image with segmented liver tumor.

[0029] It should be noted that the above embodiments of the present application are illustrative, but not limiting the present application, therefore the present application is not limited to the above specific embodiments. Any other embodiments obtained by those skilled in the art under the inspiration of the present application without departing from the principles of the present application are considered to be within the protection of the present application.

Claims

1. A multi-branch liver tumor segmentation method based on UNet, characterized in that, The steps include the following: First, a multi-branch liver tumor segmentation network based on UNet is constructed; The UNet-based multi-branch liver tumor segmentation network consists of one input layer, nine multi-branch modules, four max-pooling layers, four upsampling modules, one 1×1 convolutional layer, and one output layer. The input of the input layer serves as the input to the UNet-based multi-branch liver tumor segmentation network, and the output of the input layer is connected to the input of the first multi-branch module. The output of the first multi-branch module is connected to the input of the first max pooling module and the first input of the fourth upsampling module; the output of the first max pooling module is connected to the input of the second multi-branch module. The output of the second multi-branch module is connected to the input of the second max pooling module and the first input of the third upsampling module; the output of the second max pooling module is connected to the input of the third multi-branch module. The output of the third multi-branch module is connected to the input of the third max pooling and the first input of the second upsampling; The output of the third max-pooling module is connected to the input of the fourth multi-branch module; The output of the fourth multi-branch module is connected to the input of the fourth max pooling module and the first input of the first upsampling module; the output of the fourth max pooling module is connected to the input of the fifth multi-branch module. The output of the fifth multi-branch module is connected to the second input of the first upsampled module, and the output of the first upsampled module is connected to the input of the sixth multi-branch module. The output of the sixth multi-branch module is connected to the second input of the second upsampled module, and the output of the second upsampled module is connected to the input of the seventh multi-branch module. The output of the seventh branch module is connected to the second input of the third upsampled module, and the output of the third upsampled module is connected to the input of the eighth multi-branch module. The output of the eighth multi-branch module is connected to the second input of the fourth upsampled module, and the output of the fourth upsampled module is connected to the input of the ninth multi-branch module. The output of the ninth multi-branch module is connected to the input of a 1×1 convolutional layer, and the output of the 1×1 convolutional layer is connected to the input of the output layer; the output of the output layer serves as the output of the UNet-based multi-branch liver tumor segmentation network. The aforementioned multi-branch module consists of two 3×3 convolutional layers with a dilation rate of 2, two 3×3 convolutional layers with a dilation rate of 4, two max-pooling layers with a kernel of 2, two feature addition layers, two batch normalization layers, and two activation layers. The inputs of the first 3×3 convolutional layers with a dilation rate of 2, the first 3×3 convolutional layers with a dilation rate of 4, and the first max-pooling layer with a kernel of 2 together form the input of the multi-branch module. The outputs of the first 3×3 convolutional layers with a dilation rate of 2, the first 3×3 convolutional layers with a dilation rate of 4, and the first max-pooling layer with a kernel of 2 are simultaneously connected to the input of the feature addition layers, and the output of the feature addition layers is connected to the first batch normalization layer. The input of the normalization layer and the output of the first batch of normalization layers are connected to the input of the first activation layer. The output of the first activation layer is simultaneously connected to the input of the second 3×3 convolutional layer with a dilation rate of 2, the second 3×3 convolutional layer with a dilation rate of 4, and the second max pooling layer with a kernel of 2. The outputs of the second 3×3 convolutional layer with a dilation rate of 2, the second 3×3 convolutional layer with a dilation rate of 4, and the second max pooling layer with a kernel of 2 are simultaneously connected to the input of the feature addition layer. The output of the feature addition layer is connected to the input of the second batch of normalization layers. The output of the second batch of normalization layers is connected to the input of the second activation layer. The output of the second activation layer forms the output of the multi-branch module. Then, a training sample set is constructed using CT images of liver tumors that have been segmented in advance, and the constructed UNet-based multi-branch liver tumor segmentation network is trained using the training sample set to obtain the trained UNet-based multi-branch liver tumor segmentation network. Finally, the CT image that needs to be segmented for liver tumor is fed into the trained UNet-based multi-branch liver tumor segmentation network to obtain the segmented CT image of the liver tumor.

2. The multi-branch liver tumor segmentation method based on UNet according to claim 1, characterized in that, The upsampling module consists of one bilinear interpolation layer and one stitching layer; The input of the bilinear interpolation layer serves as the first input of the upsampling module. The output of the bilinear interpolation layer is connected to one input of the stitching layer. The other input of the stitching layer serves as the second input of the upsampling module, and the output of the stitching layer serves as the output of the upsampling module.

Images