A 2.5D deep learning segmentation algorithm for liver tumors based on contextual information perception

Abstract

The present invention discloses a 2.5D deep learning segmentation algorithm for liver tumors based on contextual information perception. The present invention proposes a 2.5D segmentation network model based on contextual perception by combining a two-dimensional network and a three-dimensional network, which effectively reduces the computational cost while comprehensively utilizing contextual information. On the one hand, multiple continuous slices are used as network inputs, and on the other hand, three-dimensional convolution is used in the encoder part to extract deep features and inter-slice features, thereby optimizing the segmentation results by utilizing inter-layer continuity. At the same time, a hole space convolution pooling module is used at the top layer of the feature extraction network to introduce global semantic information, thereby alleviating the problem of large differences in the scale of the segmentation target. In addition, a two-dimensional network is used in the decoder part to reduce the amount of parameters, and a deep separable convolution is used in the network instead of a conventional convolution to reduce the computational cost. At the same time, the weighted loss function designed in combination with the category cross entropy loss function and the Dice loss function can also further improve the category imbalance problem in medical images.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning image processing, and in particular to a medical image segmentation algorithm utilizing contextual semantic information. Background Art

[0002] As an important metabolic organ of the human body, the liver can directly determine the health of the human body. According to statistics, there are about one million new liver cancer patients in the world every year, among which China has the highest incidence of liver malignant tumors and liver surgery rate in the world. The traditional method of diagnosing liver lesions is for radiologists to judge and identify them with the naked eye, which is not only time-consuming and laborious, but also the accuracy of the diagnosis results depends to a large extent on the doctor's subjective judgment and clinical experience. Therefore, it is of great significance to promote the development and implementation of automated detection and segmentation technology for liver tumor lesions.

[0003] As one of the research directions in the field of machine learning, deep learning builds more complex network models through nonlinear fitting and deeper network layers, which is more suitable for more complex problem scenarios. Compared with traditional machine learning algorithms, deep neural networks do not need to manually extract and select features, but instead obtain deep features and high-level semantic information of images through multi-layer convolution training and learning. At present, deep learning related algorithms have been widely used in the fields of medical image classification, detection, segmentation and registration. Applying deep learning models to liver tumor datasets to locate and classify tumors and further fine segmentation at the pixel level can effectively assist doctors in diagnosis and improve diagnostic efficiency.

[0004] At present, the commonly used method in the field of image processing is the two-dimensional deep neural segmentation network, which fuses the feature maps of corresponding resolutions in the encoding-decoding structure through jump connections, making the image detail features richer and more suitable for medical image segmentation tasks with small-scale targets. However, the two-dimensional segmentation network only extracts the spatial feature information within the two-dimensional slices, and does not utilize the correlation between slices. Therefore, some studies have proposed using a three-dimensional segmentation network with a three-dimensional convolution module to process three-dimensional medical images, but the three-dimensional network has higher requirements on the performance of the computing device, and has a larger number of parameters and calculations. Summary of the invention

[0005] Common medical images are mostly three-dimensional stereo images. The existing two-dimensional segmentation network only focuses on the intra-slice features of the input image, while ignoring the inter-slice correlation between continuous slices of three-dimensional medical images. The present invention focuses on this. In order to avoid the problems of large amount of calculation and large number of parameters caused by the three-dimensional network, a two-and-a-half-dimensional segmentation network model based on context perception is proposed by combining the two-dimensional network and the three-dimensional network, which effectively reduces the calculation cost while comprehensively utilizing the context information. The present invention, on the one hand, uses multiple continuous slices as network input, and on the other hand, uses three-dimensional convolution in the encoder part to extract deep features and inter-slice features, thereby optimizing the segmentation results by using inter-layer continuity. At the same time, the hollow space convolution pooling module is used at the top layer of the feature extraction network to introduce global semantic information, alleviating the problem of large differences in the scale of the segmentation target. In addition, a two-dimensional network is used in the decoder part to reduce the number of parameters, and a deep separable convolution is used in the network instead of a conventional convolution to reduce the computational cost. At the same time, the weighted loss function designed by combining the category cross entropy loss function and the Dice loss function can also further improve the category imbalance problem in medical images.

[0006] The present invention proposes a medical image segmentation algorithm, which is a 2.5D deep learning segmentation algorithm for liver tumors based on contextual information perception. From the perspective of using contextual information to improve segmentation results, a 2.5D segmentation network model is established based on the characteristics of information continuity between continuous slices of medical images and the advantages of two-dimensional and three-dimensional networks. Multiple continuous slices are used as network input, and the three-dimensional network is used to extract inter-layer feature information and use deep separable convolution to further reduce the computing cost. At the same time, a dilated spatial convolution pooling pyramid is added to the deep layer of the network model encoder to introduce global semantic information. In addition, a weighted loss function that is more suitable for medical image segmentation tasks is designed during the training process to help the network more effectively deal with the pixel imbalance problem between liver tumors and backgrounds.

[0007] Technical Solution

[0008] The present invention discloses a 2.5D deep learning segmentation algorithm for liver tumors based on contextual information perception. The existing two-dimensional segmentation network ignores the inter-slice feature correlation of three-dimensional images. The three-dimensional network takes the three-dimensional image as input and can capture the contextual information within and between slices. However, the three-dimensional segmentation network often has a large number of parameters and a high computational cost. The present invention thus constructs a segmentation network model that can fully utilize the feature correlation between consecutive slices without excessively increasing the amount of calculation.

[0009] The specific steps include:

[0010] S1, liver tumor CT image data is used and preprocessed;

[0011] S2, builds a segmentation network model, takes the preprocessed continuous images as input, performs deep feature extraction in the 3D encoding stage, restores the original feature resolution in the 2D decoding stage and outputs the segmentation prediction results of the middle layer slices, connected in the middle by a dimensionality reduction module;

[0012] S3, considering the sample category problem in the data set, based on the cross entropy loss function and Dice loss function commonly used in segmentation tasks, designed a weighted loss function that is more suitable for medical segmentation tasks, and used this loss function to train the network model;

[0013] S4, train the 2.5D segmentation network and use the 3D fully connected conditional random field to refine the final liver tumor segmentation result. The algorithm model is evaluated using common evaluation indicators for segmentation tasks such as Dice similarity coefficient, voxel overlap error, and voxel relative error.

[0014] The image preprocessing process S1 of the above dataset specifically includes the following steps:

[0015] S11, read the pictures in the dataset;

[0016] S12, calculating the corresponding window width and window position according to the maximum and minimum HU values ​​of the tumor in the case (the window width refers to the HU value range contained in the CT image, and the window position refers to the average value of the upper and lower limits of the window width), and adjusting the HU value to the range of [-200, 250] by setting the values ​​of the window width and window position, so as to display a clearer target area;

[0017] S13, histogram equalization is used to further enhance the contrast and the voxel values ​​are normalized to [-1, 1];

[0018] S14, standardize the data set using the standard deviation of the training set;

[0019] S15, use random cropping, flipping transformation, brightness transformation and other methods to enhance the data set.

[0020] In the above step S2, the segmentation network includes a 3D encoder sub-network and a 2D decoder sub-network, and a dimension reduction-skip connection module is used in the middle to connect the feature maps of the same resolution in the encoding stage and the decoding stage. Specifically, the following steps are included:

[0021] S21, after the data set is preprocessed, each target image is concatenated with t consecutive adjacent images and input into the network to extract features. ·,·,k , stitch the slices from kt to k+t together, and their corresponding consecutive t adjacent slices V ·,·,{k-t,…,k,…k+t} As shown below:

[0022] V ·，·{k-t，...，k，...k+t}=[V ·，·，k-t , …, V ·，·，k , …, V ·，·，k+t ]

[0023] S22, performs feature extraction on the input image in the encoding stage, and uses 3D depthwise separable convolution to better mine intra-layer and inter-layer information.

[0024] S23, the encoder subnetwork consists of four layers. At the end of each layer, a downsampling operation is performed to reduce the resolution and double the number of feature map channels, so as to fully learn the features of images within and between image slice layers.

[0025] S24, a three-dimensional atrous spatial convolution pyramid pooling module is used on the feature image output by the fourth layer of the encoder to increase the size of the receptive field and retain more global context information.

[0026] S25, the decoder subnetwork is a four-layer deconvolution operation, using two-dimensional depth-wise separable convolution to reduce the dimension and parameter amount of the network, where upsampling and convolution operations constitute a deconvolution structure for parameter learning.

[0027] S26, connecting the encoder sub-network and the decoder sub-network through a dimension reduction-skip connection module to transmit image details from the encoder to the decoder.

[0028] S27, restore the image size and perform semantic segmentation at the decoder end.

[0029] In the above step S2, a dilated spatial convolutional pyramid pooling module is added to the top layer of the encoder subnetwork to increase the receptive field size and retain more global context information.

[0030] The specific steps include:

[0031] S21, uses 1×1×1 convolution and 3D depthwise separable convolution with dilation rates of 2, 4, and 6 in the three spatial dimensions to sample the input feature map;

[0032] S22, concatenates the feature maps generated at each void rate and passes them through a point-by-point convolutional layer to generate a feature map with more global information.

[0033] In the above step S3, based on the cross entropy loss function and Dice loss function commonly used in segmentation tasks, a weighted loss function more suitable for medical segmentation tasks is designed and used during the training process. Specifically, the following steps are included:

[0034] S31, introduces the category weighting factor w based on the pixel-by-pixel cross entropy loss function class , the definition of the class-balanced cross entropy loss function is as follows:

[0035]

[0036]

[0037] Among them, y i represents the true value image annotated by the doctor, p i represents the predicted image generated by the model, N represents the batch size, It is determined by the ratio of pixels in the entire image.

[0038] S32, Dice loss function is as follows:

[0039]

[0040] S33, the weighted loss function is designed by combining the cross entropy loss function and the Dice loss function. The final weighted loss function is as follows:

[0041] Loss = αL BCE +βL Dice

[0042] Among them, α and β are weight coefficients. During the experiment, the weight coefficients need to be adjusted according to the experimental results to obtain better experimental results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a 2.5-dimensional deep learning segmentation algorithm for liver tumors based on contextual information perception provided by an embodiment of the present invention

[0044] Figure 2 This is a comparison chart of the data set before and after preprocessing

[0045] Figure 3 This is a complete network structure diagram of the present invention.

[0046] Figure 4 It is the weighted loss function effect verification DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] The present invention aims to solve the segmentation difficulties of liver tumor images, such as blurred edges, low contrast, and large scale differences. From the perspective of using contextual information, the present invention combines the advantages of two-dimensional and three-dimensional networks to establish a two-and-a-half-dimensional segmentation network model based on the characteristics of information continuity between consecutive slices of medical images. At the same time, the deep separable convolution is used to further reduce the number of parameters and computational costs. In addition, considering the imbalance of liver tumor and background categories, a weighted loss function more suitable for segmentation tasks is designed to optimize the algorithm performance.

[0049] The 2.5-dimensional deep learning segmentation algorithm for liver tumors based on contextual information perception provided by the present invention comprises the following steps:

[0050] Step 1, data preprocessing module, in order to more clearly display the target tissues and organs, eliminate the interference of other irrelevant organs in the abdomen, and increase the contrast between the liver and other tissues, the HU value is adjusted to the range of [-200, 250] by setting the window width and window position. The corresponding window width and window position of each case are calculated based on the maximum and minimum HU values ​​of the tumor in that case. On the basis of the above operations, the contrast is further enhanced through histogram equalization, and the voxel value is normalized to [-1, 1]. The normalized voxel value is subtracted from the average voxel value in the training set, and the data set is standardized using the standard deviation of the training set. In addition, data enhancement such as random cropping and flipping transformation is performed. For comparison before and after preprocessing, please refer to Figure 2 .

[0051] Step 2: Build a 2.5D segmentation network model, which consists of a 3D encoder subnetwork and a 2D decoder subnetwork. A dimension reduction-skip connection module is used in the middle to connect the feature maps of the same resolution in the encoding stage and the decoding stage. In the encoding stage, the input image is extracted, and 3D convolution is used to better mine the intra-layer and inter-layer information. The decoder subnetwork uses 2D convolution to reduce the dimension and parameter amount of the network, and restores the image size and performs semantic segmentation at the end of the decoder. The network model can be referenced Figure 3 .

[0052] The encoder subnetwork has four layers, and downsampling is performed at the end of each layer to reduce the resolution and double the number of feature map channels, so as to fully learn the features of the images within and between CT slice layers. All convolutions in the first four layers of the encoder subnetwork are zero-padded convolution operations in the x, y, and z directions. The decoder subnetwork is a four-layer deconvolution operation, in which upsampling and convolution operations form a deconvolution structure for parameter learning. The network transmits image details from the encoder to the decoder through a dimension reduction-skip connection module, thereby improving the edge segmentation prediction accuracy. In addition, a residual structure is used in each layer of the encoder and decoder subnetworks to accelerate the convergence of the network.

[0053] Step three, the design of the dimension reduction jump connection module. The feature map of the encoder subnetwork is a three-dimensional structure, and the feature map of the decoder subnetwork is a two-dimensional structure. Therefore, it is necessary to introduce a dimension reduction module on the jump connection to connect the encoder and decoder at the same level. Through a non-zero-filled 3×3×3 convolution, the input three-dimensional features are converted into two-dimensional features through the ReLU activation function and batch normalization operation to form a dimension reduction-jump connection. Compared with conventional jump connections, the dimension reduction-jump connection module proposed in the present invention uses a three-dimensional convolution that is not filled with zeros in the z-axis direction of the feature map, and the convolution kernel size is 3×3×3. Zeros are filled in the spatial directions of the x and y axes, so the spatial resolution remains unchanged, and the z-axis dimension is reduced from 3 to 1.

[0054] Step 4: Design of loss function. Since there is an imbalance problem among liver pixels, tumor pixels, and background pixels in the liver tumor segmentation task, a weighted loss function is designed by combining the cross entropy loss function and the Dice loss function commonly used in the segmentation task. The final weighted loss function is shown below:

[0055] Loss = αL BCE +βL Dice

[0056] Among them, α and β are weight coefficients, which are used to balance the proportion of cross entropy loss function and Dice loss function in the weighted loss function. During the training process, it is necessary to adjust the size of the weight coefficient according to the experimental results to obtain better experimental results, so as to achieve the purpose of paying more attention to categories with fewer samples. The effect of weighted loss function can be referred to Figure 4 .

[0057] Step 5: Model training and experimental verification. Taking the liver tumor LiTS2017 dataset as an example, 131 cases can be divided into 110 training sets, and the remaining 21 cases are used as test sets. The training uses the Adam optimizer to update the parameters, and the initial learning rate is set to 0.002, and is reduced to 0.1 times the previous one after 20 and 40 epochs, respectively. The comparison with other algorithms also follows this set of unified standards to train and test different models.

Claims

1. A 2.5D deep learning segmentation algorithm for liver tumors based on contextual information perception, characterized in that: In view of the difficulties in segmentation of liver tumor images, such as blurred edges, low contrast, and large scale differences, we established a 2.5D segmentation network model from the perspective of utilizing contextual information, taking into account the characteristics of information continuity between consecutive slices of medical images and combining the advantages of 2D and 3D networks. We used deep separable convolution to further reduce the number of parameters and computational costs. We introduced global semantic information through the atrous spatial convolution pooling module at the top level of the feature extraction network. In addition, considering the imbalance of liver tumor and background categories, we designed a weighted loss function that is more suitable for segmentation tasks by combining the category cross entropy loss function and the Dice loss function to optimize the algorithm performance. The 2.5D deep learning segmentation algorithm for liver tumors specifically includes the following steps: (1) Using liver tumor CT image data and preprocessing; (2) Construct a 2.5D segmentation network model, use the preprocessed continuous image as input, perform deep feature extraction in the 3D encoding stage, restore the original feature resolution in the 2D decoding stage and output the segmentation prediction results of the intermediate layer slices, connected in the middle by a dimensionality reduction module; (3) In order to further increase the receptive field, a dilated spatial convolutional pyramid pooling module is used at the top layer of the encoding subnetwork to introduce more global semantic information; (4) Considering the sample category problem in the dataset, based on the cross entropy loss function and Dice loss function commonly used in segmentation tasks, a weighted loss function more suitable for medical segmentation tasks was designed, and the 2.5D segmentation network model was trained using this loss function; (5) Training the 2.5D segmentation network model in step (2), and using 3D fully connected conditional random fields to refine the final liver tumor segmentation results; using Dice similarity coefficient, voxel overlap error, and voxel relative error, which are commonly used evaluation indicators for segmentation tasks, to evaluate the 2.5D segmentation network model and algorithm; Step (2) describes the overall network structure design. The 2.5D segmentation network model includes a 3D encoder sub-network and a 2D decoder sub-network, with a dimension reduction-skip connection module in the middle to connect the feature maps of the same resolution in the encoding stage and the decoding stage; The following steps are involved: (2.1) Each target image is concatenated with t consecutive adjacent images as the input image and input into the network to extract features; for the kth slice of volume V along the z axis , stitch together the slices from layer kt to layer k+t, and their corresponding consecutive t adjacent slices As shown below: ； (2.2) Feature extraction of the input image in the encoding stage, using 3D depth-separable convolution to better mine intra-layer and inter-layer information; (2.3) The encoder subnetwork consists of four layers. The last layer of each layer is downsampled to reduce the resolution and double the number of feature map channels, so as to fully learn the features of images within and between image slice layers. (2.4) Use a three-dimensional atrous spatial convolutional pyramid pooling module on the feature image output by the fourth layer of the encoder to increase the size of the receptive field and retain more global context information; (2.5) The decoder subnetwork is a four-layer deconvolution operation, using two-dimensional depth-separable convolution to reduce the dimension and number of parameters of the network, where upsampling and convolution operations form a deconvolution structure for parameter learning; (2.6) Connect the encoder sub-network and the decoder sub-network through the dimension reduction-skip connection module to transfer image details from the encoder to the decoder; (2.7) Restore image size and perform semantic segmentation at the decoder end; In step (4), based on the cross entropy loss function and Dice loss function commonly used in segmentation tasks, a weighted loss function more suitable for medical segmentation tasks was designed and used during the training process; The following steps are involved: (4.1) Introducing a category weighting factor based on the pixel-by-pixel cross entropy loss function , the definition of the category-balanced cross entropy loss function is as follows: in, represents the true value image annotated by the doctor, represents the predicted image generated by the 2.5D segmentation network model, N represents the batch size, Determined by the proportion of pixels in the entire image; (4.2) The Dice loss function is as follows: ； (4.3) The weighted loss function is designed by combining the cross entropy loss function and the Dice loss function. The final weighted loss function is as follows: in and is the weight coefficient. During the experiment, the weight coefficient is adjusted according to the experimental results to obtain better experimental results.

2. The context-aware 2.5-dimensional deep learning segmentation algorithm for liver tumors according to claim 1 is characterized in that: In step (1), the image preprocessing includes the following steps: (1.1) Read the images in the dataset; (1.2) Calculate the corresponding window width and window position based on the maximum and minimum HU values ​​of the tumor in the case (window width refers to the HU value range contained in the CT image, and window position refers to the average value of the upper and lower limits of the window width). By setting the values ​​of the window width and window position, adjust the HU value to the range of [-200, 250] to display a clearer target area; (1.3) Use histogram equalization to further enhance the contrast and normalize the voxel values ​​to [-1, 1]; (1.4) Use the standard deviation of the training set to standardize the data set; (1.5) Use random cropping, flipping, and brightness transformation to enhance the data set.

3. The context-aware 2.5D deep learning segmentation algorithm for liver tumors according to claim 1 is characterized in that: Step (3) describes the specific design of the pyramid pooling module, which adds a dilated spatial convolutional pyramid pooling module to the top layer of the encoder subnetwork responsible for feature extraction to increase the receptive field size and retain more global context information; The following steps are involved: (3.1) Use 1×1×1 convolution and 3D depthwise separable convolution with dilation rates of 2, 4, and 6 in the three spatial dimensions to sample the input feature map; (3.2) The feature maps generated at different dilation rates are concatenated and passed through a point-by-point convolutional layer to generate a feature map with more global information.

Images