Kidney tissue segmentation method and device based on CT images

Abstract

The method and device for renal tissue segmentation based on CT images can improve the accuracy of kidney, renal artery, renal vein and renal tumor segmentation. The method includes: (1) obtaining an image to be segmented; (2) slicing the image to be segmented; (3) preprocessing the sliced ​​image; (4) inputting the image into a pre-trained region of interest extraction model to obtain the region of interest; (5) inputting the image within the region of interest into a pre-trained segmentation model; (6) processing the segmentation result to remove some erroneous slices; (7) visualizing the segmentation result in three dimensions.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for segmenting kidney tissue based on CT images, and a device for segmenting kidney tissue based on CT images. Background Art

[0002] Renal CT is a CT scan of the kidneys, which is a method of examining the kidneys through CT images.

[0003] It has the following meanings:

[0004] 1. It can determine the location, size, shape, and invasion range of the tumor; it can identify the tumor as a cystic, solid, fatty, or calcified lesion, and thus make a qualitative diagnosis.

[0005] 2. When intravenous urography shows a non-functioning kidney, CT can determine the location, nature or congenital developmental abnormality of the lesion.

[0006] 3. It can detect small calcifications, stones or negative stones that cannot be seen by ordinary X-ray examinations.

[0007] 4. It is of great value in the diagnosis of renal tuberculosis and can show intrarenal destruction, pathogenic calcification and perinephric abscess, etc.

[0008] 5. It can determine the location, extent and perinephric hematoma of kidney damage, as well as postoperative complications.

[0009] However, it is difficult to improve the segmentation accuracy of CT images of the kidney, renal artery, renal vein, and renal tumor. Summary of the invention

[0010] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide a kidney tissue segmentation method based on CT images, which can improve the accuracy of kidney, renal artery, renal vein and renal tumor segmentation.

[0011] The technical solution of the present invention is: the kidney tissue segmentation method based on CT images comprises the following steps:

[0012] (1) Obtain the image to be segmented;

[0013] (2) Slicing the image to be segmented;

[0014] (3) Preprocessing the sliced ​​images;

[0015] (4) The image is input into a pre-trained region of interest extraction model to obtain the region of interest;

[0016] (5) Input the image in the region of interest into the pre-trained segmentation model;

[0017] (6) Process the segmentation results and remove some erroneous fragments;

[0018] (7) Three-dimensional visualization of segmentation results;

[0019] In step (5), the segmentation model is a multi-scale fusion Attention-DenseUnet network. The network architecture is divided into two parts: an encoding downsampling path and a decoding upsampling path. The downsampling path contains four dense connection modules. From top to bottom, the size of each dense connection module is 6 layers, 12 layers, 16 layers, and 24 layers respectively; in the dense connection module, the input of each layer is not only related to the output of the i-1 layer, but also to the output of all previous layers, which is formula (1):

[0020] X l =H l ([X 0 ,X 1 ,…X l-1 ]) (1)

[0021] Where [] represents X 0 To X l-1 All outputs of the layer are organized together by channel, H l Represents nonlinear transformation. This network structure uses a combination of BatchNorm+ReLU+Conv1×1+Conv3×3 for nonlinear transformation.

[0022] The region of interest extraction model of the present invention is a ResUNet network, which can accurately capture the features of the kidney part on the CT image, and output the eight vertex coordinates of a bounding box containing the entire kidney part structure. The segmentation model of the present invention is a multi-scale fusion Attention-DenseUNet model, and the result of the region of interest extraction is input into the segmentation model for fine segmentation. The multi-scale feature fusion mechanism used in the segmentation model is an important method to improve network performance. The low-level features contain more position information, and the high-level features have the phenomenon of information loss, and the perception ability of details is reduced. Therefore, the low-level features and high-level features are fused, and the multi-scale spatial information can be effectively used to improve the accuracy of segmentation. The model performs 4 upsamplings, and a feature map is output after each upsampling. After each layer of feature images is convolved with a convolution kernel size of 1*1, features with the same number of channels are obtained, and the final feature map is added. The final feature map is activated by the Sigmoid function to obtain the final segmentation result. After the segmentation is completed, the segmentation results need to be morphologically post-processed to remove some erroneous fragments, and then the segmentation results are visualized in three dimensions, which can further improve the accuracy of kidney, renal artery, renal vein, and renal tumor segmentation.

[0023] A CT image-based renal tissue segmentation device is also provided, comprising:

[0024] an acquisition module configured to acquire an image to be segmented;

[0025] A slicing module configured to slice the image to be segmented;

[0026] A preprocessing module configured to preprocess the sliced ​​image;

[0027] A region of interest extraction module is configured to input an image into a pre-trained region of interest extraction model to obtain a region of interest;

[0028] a segmentation module configured to input an image within a region of interest into a pre-trained segmentation model;

[0029] The post-processing module is configured to process the segmentation results and remove some erroneous fragments;

[0030] a display module configured to perform three-dimensional visualization of the segmentation results;

[0031] In the segmentation module, the segmentation model is a multi-scale fusion Attention-DenseUNet network. The network architecture is divided into two parts: an encoding downsampling path and a decoding upsampling path. The downsampling path contains four dense connection modules. From top to bottom, the size of each dense connection module is 6 layers, 12 layers, 16 layers, and 24 layers respectively. In the dense connection module, the input of each layer is not only related to the output of the i-1 layer, but also to the output of all previous layers, as shown in formula (1):

[0032] X l =H l ([X 0 ,X 1 ,…X l-1 ]) (1)

[0033] Where [] represents X 0 To X l-1 All outputs of the layer are organized together by channel, H l Represents nonlinear transformation. This network structure uses a combination of BatchNorm+ReLU+Conv1×1+Conv3×3 for nonlinear transformation.

[0034] A spatial attention module is added to the skip connection layer between the downsampling layer and the upsampling layer to further improve the segmentation accuracy;

[0035] In the decoding part, a multi-scale fusion strategy is adopted. The model performs 4 upsamplings and outputs a feature map after each upsampling. After each layer of feature image is convolved with a convolution kernel size of 1*1, features with the same number of channels are obtained and added to obtain the final feature map. The final feature map is activated by the Sigmoid function to obtain the final segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the kidney tissue segmentation method based on CT images according to the present invention.

[0037] Figure 2 ResUnet for region of interest extraction is shown.

[0038] Figure 3 A 6-layer densely connected module (Dense Block) is shown.

[0039] Figure 4 The structure of the entire Attention module is shown.

[0040] Figure 5 This is the multi-scale fusion Attention-DenseUNet network structure diagram. DETAILED DESCRIPTION

[0041] like Figure 1 As shown, this CT image-based kidney tissue segmentation method includes the following steps:

[0042] (1) Obtain the image to be segmented;

[0043] (2) Slicing the image to be segmented;

[0044] (3) Preprocessing the sliced ​​images;

[0045] (4) The image is input into a pre-trained region of interest extraction model to obtain the region of interest;

[0046] (5) Input the image in the region of interest into the pre-trained segmentation model;

[0047] (6) Process the segmentation results and remove some erroneous fragments;

[0048] (7) Three-dimensional visualization of segmentation results;

[0049] like Figure 2As shown, in step (5), the segmentation model is a multi-scale fusion Attention-DenseUNet network. The network architecture is divided into two parts: an encoding downsampling path and a decoding upsampling path. The downsampling path contains four dense connection modules. From top to bottom, the size of each dense connection module is 6 layers, 12 layers, 16 layers, and 24 layers respectively; in the dense connection module, the input of each layer is not only related to the output of the i-1 layer, but also to the output of all previous layers, as shown in formula (1):

[0050] X l =H l ([X 0 ,X 1 ,…X l-1 ]) (1)

[0051] Where [] represents X 0 To X l-1 All outputs of the layer are organized together by channel, H l Represents nonlinear transformation. This network structure uses a combination of BatchNorm+ReLU+Conv1×1+Conv3×3 for nonlinear transformation.

[0052] The region of interest extraction model of the present invention is a ResUNet network, which can accurately capture the features of the kidney part on the CT image, and output the eight vertex coordinates of a bounding box containing the entire kidney part structure. The segmentation model of the present invention is a multi-scale fusion Attention-DenseUNet model, and the result of the region of interest extraction is input into the segmentation model for fine segmentation. The multi-scale feature fusion mechanism used in the segmentation model is an important method to improve network performance. The low-level features contain more position information, and the high-level features have the phenomenon of information loss, and the perception ability of details is reduced. Therefore, the low-level features and high-level features are fused, and the multi-scale spatial information can be effectively used to improve the accuracy of segmentation. The model performs 4 upsamplings, and a feature map is output after each upsampling. After each layer of feature images is convolved with a convolution kernel size of 1*1, features with the same number of channels are obtained, and the final feature map is added. The final feature map is activated by the Sigmoid function to obtain the final segmentation result. After the segmentation is completed, the segmentation results need to be morphologically post-processed to remove some erroneous fragments, and then the segmentation results are visualized in three dimensions, which can further improve the accuracy of kidney, renal artery, renal vein, and renal tumor segmentation.

[0053] like Figure 5As shown, in step (5), the segmentation model is a multi-scale fusion Attention-DenseUNet. In the encoder part of the model, four Dense blocks are added, which are 6 layers, 12 layers, 36 layers and 24 layers from top to bottom. The Dense block ensures the maximum information flow between layers and improves the gradient flow, thereby reducing the burden of finding the optimal solution in the deep neural network. In order to reduce the amount of calculation and increase the receptive field, a downward transition layer is used after each dense block. Each transition layer is composed of a ReLU activation function, batch normalization (BN), a bottleneck layer (i.e., a 1×1 convolution) and an average pooling layer (2×2). The decoding stage of the model performs multi-scale fusion. The upsampling part in the decoder will definitely lose some detail information when it is restored from low resolution to high resolution, which affects the final segmentation result. By adding a multi-scale fusion module at the decoding end, multi-scale spatial information can be effectively used to improve the accuracy of segmentation. The features of each layer of the decoder in the multi-scale fusion segmentation model are upsampled and restored to high resolution before being fused. In this way, the extracted feature information is superimposed in parallel to achieve the complementarity of multiple feature information, making the segmentation more accurate. The model performs 4 upsamplings, and a feature map is output after each upsampling. After each layer of feature images is convolved with a convolution kernel size of 1×1, features with the same number of channels are obtained, and the final feature map is added. The final feature map is activated by the Sigmoid function to obtain the final segmentation result. In the traditional UNet structure, the connection between the downsampling encoding part and the upsampling decoding part is often directly spliced ​​by the Concatenate mechanism to directly splice the previous and next features. Therefore, there is a problem of insufficient feature extraction of the downsampling part during the splicing process. This model uses the attention module to connect the upsampling and downsampling parts. It can further improve the segmentation accuracy.

[0054] Preferably, in step (2), the three-dimensional image to be segmented is decomposed into single two-dimensional slices in the coronal direction and saved as a matrix file.

[0055] Preferably, in step (3), the size of a single slice is fixed to 512×512 using bilinear interpolation, and then the grayscale Hounsfield unit value of the slice is adjusted to the range [-1024, 600] HU, and then the data is normalized to zero mean and unit variance.

[0056] Preferably, in step (4), the model uses a cross entropy loss function to combine the labels of the tumor and kidney parts into one category for calculation.

[0057] Preferably, if Figure 3As shown, in step (5), each dense connection module (Dense) is connected using a Transition layer, and the Transition layer consists of a Conv (1×1) and a Maxpool.

[0058] Preferably, in step (5), the upsampling path includes spatial attention and dense supervision mechanisms. The spatial attention module increases the weight of the target area for the network and improves the segmentation accuracy. The dense supervision converts each upsampled feature map channel into a 3×1×1 convolution to reduce the computational loss according to the size of the target image. In this way, each upsampling will be as similar to the target as possible. The structure of the entire Attention-DenseUNet is as follows: Figure 5 shown.

[0059] Among them, the spatial attention mechanism is implemented by the spatial attention module, such as Figure 4 As shown. The feature map g of the same layer of the downsampling layer i A 1*1*1 convolution operation is performed, and the feature map of the previous layer of the upsampling layer is also subjected to a 1*1*1 convolution; the two feature maps are added and then ReLU is performed, and then the final attention weight is obtained through a 1*1*1 convolution and a Sigmoid function; in each upsampling module, a 1*1 convolution is performed on the feature map, and it is converted into a three-channel matrix for Loss calculation, so that each upsampling is as similar to the target as possible.

[0060] Preferably, in step (6), for the segmentation results of the blood vessels, a morphological closing operation is first performed, and then an operation of first dilation and then corrosion is performed to connect some broken segments; the segmentation results of the kidney and tumor are processed using a 3D connected component method to remove some erroneous segments.

[0061] Preferably, in step (7), the segmentation result is three-dimensionally modeled using the VTK open source framework to facilitate direct observation by the doctor.

[0062] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, CD, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a kidney tissue segmentation device based on CT images, which is usually represented in the form of functional modules corresponding to the steps of the method. The device includes an acquisition module, which is configured to acquire the image to be segmented;

[0063] an acquisition module configured to acquire an image to be segmented;

[0064] A slicing module configured to slice the image to be segmented;

[0065] A preprocessing module configured to preprocess the sliced ​​image;

[0066] A region of interest extraction module is configured to input an image into a pre-trained region of interest extraction model to obtain a region of interest;

[0067] a segmentation module configured to input an image within a region of interest into a pre-trained segmentation model;

[0068] The post-processing module is configured to process the segmentation results and remove some erroneous fragments;

[0069] a display module configured to perform three-dimensional visualization of the segmentation results;

[0070] In the segmentation module, the segmentation model is a multi-scale fusion Attention-DenseUNet network. The network architecture is divided into two parts: an encoding downsampling path and a decoding upsampling path. The downsampling path contains four dense connection modules. From top to bottom, the size of each dense connection module is 6 layers, 12 layers, 16 layers, and 24 layers respectively. In the dense connection module, the input of each layer is not only related to the output of the i-1 layer, but also to the output of all previous layers, as shown in formula (1):

[0071] X l =H l ([X 0 ,X 1 ,…X l-1 ]) (1)

[0072] Where [] represents X 0 To X l-1 All outputs of the layer are organized together by channel, H l Represents nonlinear transformation. This network structure uses a combination of BatchNorm+ReLU+Conv1×1+Conv3×3 for nonlinear transformation.

[0073] A spatial attention module is added to the skip connection layer between the downsampling layer and the upsampling layer to further improve the segmentation accuracy;

[0074] In the decoding part, a multi-scale fusion strategy is adopted. The model performs 4 upsamplings and outputs a feature map after each upsampling. After each layer of feature image is convolved with a convolution kernel size of 1*1, features with the same number of channels are obtained and added to obtain the final feature map. The final feature map is activated by the Sigmoid function to obtain the final segmentation result.

[0075] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A method for segmenting kidney tissue based on CT images, characterized in that: It includes the following steps: (1) Obtain the image to be segmented; (2) Slicing the image to be segmented; (3) Preprocessing the sliced ​​images; (4) The image is input into a pre-trained region of interest extraction model to obtain the region of interest; (5) Input the image in the region of interest into the pre-trained segmentation model; (6) Process the segmentation results and remove some erroneous fragments; (7) Three-dimensional visualization of segmentation results; In step (5), the segmentation model is a multi-scale fusion Attention-DenseUnet network. The network architecture is divided into two parts: an encoding downsampling path and a decoding upsampling path. The downsampling path contains four dense connection modules. From top to bottom, the size of each dense connection module is 6 layers, 12 layers, 16 layers, and 24 layers respectively; in the dense connection module, the input of each layer is not only related to the output of the i-1 layer, but also to the output of all previous layers, which is formula (1): X l =H l ([X0,X1,…X l-1 ]) (1) Where [] represents the value of X0 to X l-1 All outputs of the layer are organized together by channel, H l Represents nonlinear transformation. This network structure uses a combination of BatchNorm+ReLU+Conv1×1+Conv3×3 for nonlinear transformation.

2. The method for segmenting kidney tissue based on CT images according to claim 1, characterized in that: In the step (2), the three-dimensional image to be segmented is decomposed into single two-dimensional slices in the coronal direction and saved as a matrix file.

3. The method for segmenting kidney tissue based on CT images according to claim 2, characterized in that: In step (3), the size of a single slice is fixed to 512×512 using bilinear interpolation, the Hounsfield unit value of the slice data is adjusted to the range [-1024, 600] HU, and then the grayscale value of the slice is normalized to zero mean and unit variance.

4. The method for segmenting kidney tissue based on CT images according to claim 3, characterized in that: In step (4), the model uses a cross entropy loss function to combine the labels of the tumor and kidney into one category for calculation.

5. The method for segmenting kidney tissue based on CT images according to claim 4, characterized in that: In the step (5), each densely connected module is connected using a Transition layer, and the Transition layer consists of a Conv (1×1) and a Maxpool.

6. The method for segmenting kidney tissue based on CT images according to claim 5, characterized in that: In the step (5), a multi-scale fusion strategy is adopted in the decoding part, and the model performs 4 upsamplings. After each upsampling, a feature map is output. After each layer of feature image is convolved with a convolution kernel size of 1*1, features with the same number of channels are obtained, and the final feature map is added together; the final feature map is activated by the Sigmoid function to obtain the final segmentation result.

7. The method for segmenting kidney tissue based on CT images according to claim 6, characterized in that: In the step (5), the upsampling path includes spatial attention and dense supervision mechanisms. The spatial attention module increases the weight of the target area for the network and improves the segmentation accuracy; the dense supervision converts each upsampled feature map channel into a 3*1*1 convolution to reduce the computational loss according to the size of the target image; Feature map g of the same layer of the downsampling layer i Perform a 1*1*1 convolution operation, and also perform a 1*1*1 convolution on the feature map of the previous layer of the upsampling layer; add the two feature maps and perform ReLU, and then use a 1*1*1 convolution and a Sigmoid function to get the final attention weight; In each upsampling module, the feature map is convolved 1*1 and converted into a three-channel matrix for Loss calculation, so that each upsampling is as similar to the target as possible.

8. The method for segmenting kidney tissue based on CT images according to claim 7, characterized in that: In the step (6), the segmentation results of the blood vessels are first subjected to a morphological closing operation, and then to an operation of first dilation and then corrosion, so as to connect some of the disconnected parts of the segmentation; the segmentation results of the kidney and the tumor are processed using the 3D connected component method to remove some erroneous fragments.

9. The method for segmenting kidney tissue based on CT images according to claim 8, characterized in that: In the step (7), the segmentation results are three-dimensionally modeled using the VTK open source framework to facilitate direct observation by doctors.

10. A kidney tissue segmentation device based on CT images, characterized in that: It includes: an acquisition module configured to acquire an image to be segmented; A slicing module configured to slice the image to be segmented; A preprocessing module configured to preprocess the sliced ​​image; A region of interest extraction module is configured to input an image into a pre-trained region of interest extraction model to obtain a region of interest; a segmentation module configured to input an image within a region of interest into a pre-trained segmentation model; The post-processing module is configured to process the segmentation results and remove some erroneous fragments; a display module configured to perform three-dimensional visualization of the segmentation results; In the segmentation module, the segmentation model is a multi-scale fusion Attention-DenseUnet network. The network architecture is divided into two parts: an encoding downsampling path and a decoding upsampling path. The downsampling path contains four dense connection modules. From top to bottom, the size of each dense connection module is 6 layers, 12 layers, 16 layers, and 24 layers respectively. In the dense connection module, the input of each layer is not only related to the output of the i-1 layer, but also to the output of all previous layers, as shown in formula (1): X l =H l ([X0,X1,…X l-1 ]) (1) Where [] represents the value of X0 to X l-1 All outputs of the layer are organized together by channel, H l Represents nonlinear transformation. This network structure uses a combination of BatchNorm+ReLU+Conv1×1+Conv3×3 for nonlinear transformation. A spatial attention module is added to the skip connection layer between the downsampling layer and the upsampling layer to further improve the segmentation accuracy; In the decoding part, a multi-scale fusion strategy is adopted. The model performs 4 upsamplings and outputs a feature map after each upsampling. After each layer of feature image is convolved with a convolution kernel size of 1*1, features with the same number of channels are obtained and added to obtain the final feature map. The final feature map is activated by the Sigmoid function to obtain the final segmentation result.

Images