Liver Couinaud segmentation method based on point cloud and voxel fusion strategy

Abstract

The invention relates to a liver Couinaud segmentation method based on a point cloud and voxel fusion strategy, and the method comprises the following steps: S1, extracting a liver mask from a CT image, and generating a liver blood vessel attention map; s2, carrying out continuous point sampling on the liver in a 3D space based on the liver blood vessel attention map; step S3, voxelizing the 3D point cloud obtained by sampling; and S4, based on the topological relation of the coordinate points in the 3D space and the semantic information of the voxel grid, constructing a multi-scale point cloud voxel fusion network, and performing liver Couinaud segmentation prediction. Compared with the prior art, the method has the advantage of high precision.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation, and in particular, to a method for segmenting the liver Couinaud segments based on a point cloud and voxel fusion strategy. Background Art

[0002] Primary liver cancer is one of the most common cancers with the highest fatality rate at present. The most effective treatment method is liver resection surgery. Couinaud segmentation divides the liver into eight completely independent liver segments according to the physiological structure of the liver, which is of great significance for surgical operations. In practice, it is very time-consuming and cumbersome for doctors to manually divide the Couinaud liver segments. Automatically and accurately segmenting the Couinaud segments of the liver based on the patient's liver CT images can effectively help doctors formulate surgical plans.

[0003] Existing methods only use the 3D voxels of liver CT to train convolutional neural networks without considering the spatial relationships of different liver segments, resulting in poor segmentation results in areas with similar image contrast; the CNN-based liver segment segmentation methods treat all features in liver CT equally, which causes other irrelevant features of liver tissue to interfere with the segmentation results.

[0004] After retrieval, Chinese invention patent application CN114937147A discloses a segmentation intelligent recognition model and recognition method for liver CT images. This method uses a liver segmentation model based on a 3D-UNet network to segment the liver parenchymal region of abdominal CT images to obtain a liver mask, then multiplies the liver mask by the original input abdominal CT image to obtain a liver localization image, and finally inputs the liver localization image into a liver segmentation model, and the liver segmentation model performs Couinaud segmentation on the liver region.

[0005] However, due to the relatively close image intensity and contrast of each liver segment in liver CT images, as well as the problem of relatively blurred boundaries, the above method ignores the guiding information of the vascular structure in the liver area for liver segment segmentation, resulting in poor and rough liver segment segmentation results. The accuracy of Couinaud liver segment segmentation needs to be further improved, especially the segmentation at the boundary positions of adjacent liver segments is blurred and cannot be applied to assist doctors in clinical practice.

[0006] Therefore, there is an urgent need to design a method for segmenting the liver Couinaud segments with high accuracy. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for segmenting the liver Couinaud segments with high accuracy based on a point cloud and voxel fusion strategy to overcome the defects of the above-mentioned existing technologies.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] According to the first aspect of the present invention, a method for segmenting the liver Couinaud based on a point cloud and voxel fusion strategy is provided, and the method includes the following steps:

[0010] Step S1: Extract a liver mask from the CT image and generate a liver vascular attention map;

[0011] Step S2: Perform continuous point sampling on the liver in 3D space based on the liver vascular attention map;

[0012] Step S3: Voxelize the sampled 3D point cloud;

[0013] Step S4: Based on the topological relationship of coordinate points in 3D space and the semantic information of the voxel grid, construct a multi-scale point cloud voxel fusion network and perform liver Couinaud segmentation prediction.

[0014] Preferably, in step S1, a 3D-Unet network model is used to generate the liver vascular attention map. Specifically, first, a pre-trained 3D-Unet model outputs a vascular binary mask M, and then morphological dilation is used to include the vascular pixels in the area covered by the vascular binary mask M to generate the liver vascular attention map M'.

[0015] Preferably, step S2 includes the following sub-steps:

[0016] Step S21: The image coordinates I = {i 1 , i 2 ,..., i t , i t ∈ R 3} of each voxel in the CT image are converted into world coordinates P = {p 1 , p 2 ,..., p t , p t ∈ R 3}, and the expression is:

[0017] P = I * Spacing * Sirection + Origin

[0018] where Spacing is the voxel spacing of the CT image, Direction is the scanning direction, and Origin is the coordinate of the origin of the image coordinate system in the world coordinate system;

[0019] Step S22: Perform continuous point sampling based on the liver vascular attention map.

[0020] Preferably, the step S22 is specifically as follows: In each training cycle, T points are randomly sampled from the CT image, where T / 2 points fall within the first range area covered by the liver vascular attention map. A random perturbation deviation Offset = (Δx, Δy, Δz) within the range of [-1, 1] is adopted, so that each point p t =(x t , y t , z t ) in the first range area generates a corresponding new point p' t =(x t +Δx, y t +Δy, z t +Δz), and the image intensity of the corresponding coordinates is obtained by trilinear interpolation;

[0021] In the training stage, the label of the new point p' t is calculated based on the coordinate points processed by the rounding function.

[0022] Preferably, the step S3 includes:

[0023] Converting the point data (p t , f t ) to the voxel grid V u，v，w through voxelization for subsequent semantic information extraction; where f t ∈R c is the feature of the point p t ;

[0024] Normalize the point coordinates p t to the range of [0, 1] to obtain

[0025] Take the average value of the feature values of all points falling within the voxel grid (u, v, w) as the feature of the voxel grid, thereby transforming the normalized point cloud data to the voxel grid V u，v,w .

[0026] Preferably, the calculation expression of the voxel grid is:

[0027]

[0028] where r is the voxel resolution, I[·] is a binary indicator function, which takes the value of 1 when the point belongs to the voxel grid (u, v, w), otherwise 0, f t,c represents the feature of the point in the c-th channel, and N u,v,w is the number of points falling within the voxel grid (u, v, w).

[0029] Preferably, the multi-scale point cloud voxel fusion network in step S4 includes a convolutional neural network branch based on point cloud and a convolutional neural network branch based on voxel. The feature fusion process is specifically as follows:

[0030] Input the point data (p t , f t ) into the convolutional neural network branch based on point cloud, and extract fine-grained features with topological information through the multi-layer perceptron E P ; where f t ∈ R c is the feature of point p t .

[0031] Input the voxel grid V u,v,w transformed from the point data into the convolutional neural network branch E v based on voxel to aggregate the features of surrounding points and learn the semantic information in the 3D space of the liver;

[0032] Use trilinear interpolation to re-transform the features extracted from the convolutional neural network branch based on voxel into point data representation, and combine the point data representation with the fine-grained features extracted from the convolutional neural network branch based on point cloud to obtain the fused point data and corresponding point features;

[0033] Voxelize the fused point data and corresponding features again and transfer them to the two branches. After iterating the point voxel operation a set number of times, splice the obtained point features and input them into the per-point decoder for liver Couinaud segmentation prediction.

[0034] Preferably, the combination of the point data representation and the fine-grained features extracted from the convolutional neural network branch based on point cloud to obtain the fused point data and corresponding point features is specifically as follows:

[0035]

[0036] Wherein, The superscript 1 of represents is the fused point data and corresponding features obtained through the first round of point voxel operation

[0037] According to the second aspect of the present invention, an electronic device is provided, including a memory and a processor. A computer program is stored on the memory, and when the processor executes the program, any one of the methods described above is implemented.

[0038] According to the third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, any one of the methods described above is implemented.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] 1) Based on the strategy of voxel and point cloud fusion, the present invention introduces the spatial topological relationship of each liver segment in the liver segment segmentation process, making the liver segment segmentation more precise, especially at the edges of each liver segment.

[0041] 2) The present invention adopts a continuous spatial point sampling method based on a vascular attention map, introduces the vascular structure into the model, and utilizes prior medical knowledge to improve the liver segment segmentation effect.

[0042] 3) The present invention dilutes the interference of irrelevant features of liver tissue. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flowchart of the method of the present invention;

[0044] Figure 2 is a detailed flowchart of the method in the embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0046] Embodiment

[0047] As Figure 1 shown, this embodiment provides a method for Couinaud segmentation of the liver based on a point cloud and voxel fusion strategy, which is applicable to the process of Couinaud segmentation of the liver using abdominal CT images before surgery. The method includes the following steps:

[0048] Step S1: Extract a liver mask from the CT image and generate a liver vascular attention map;

[0049] Step S2: Perform continuous point sampling of the liver in 3D space based on the liver vascular attention map;

[0050] Step S3: Voxelize the sampled 3D point cloud;

[0051] Step S4: Based on the topological relationship of coordinate points in 3D space and the semantic information of voxel grids, construct a multi-scale point cloud voxel fusion network and perform Couinaud segmentation prediction of the liver.

[0052] Next, in combination with Figure 2 , the method of this embodiment will be introduced in detail.

[0053] (1) Extract the liver mask from abdominal organ CT images using a pre-trained model

[0054] The extraction of the liver mask is a fundamental task for subsequent tasks. Since the liver is relatively large in volume and easy to identify among abdominal organs, here we use a pre-trained network model to segment the liver in abdominal CT. Among them, the backbone of the network model can be selected as the 3D-Unet network.

[0055] (2) Build and train a neural network to generate a vascular attention map for the liver region

[0056] In this embodiment, the backbone of the neural network for segmenting the liver vascular attention map can still be selected as 3D-Unet, and it can be trained based on the publicly available 3Dircadb dataset. Among them, the 3Dircadb dataset contains 20 liver CT images with voxel spacings ranging from 0.56 mm to 0.87 mm and slice thicknesses from 1 mm to 4 mm, and there are accurate annotations for liver and vascular segmentation.

[0057] The BCE loss and Dice loss can be used to supervise the learning process.

[0058] In the prediction stage after training, the 3DCT image (only containing the liver mask) can be input into the neural network, and the 3D-Unet model can output a binary vascular mask (M). Then, morphological dilation can be used to include as many vascular pixels as possible in the area covered by M, thereby generating a vascular attention map M'. The size of the dilation kernel can be (3, 3, 1).

[0059] (3) Perform continuous point sampling on the liver in three-dimensional space based on the vascular attention map

[0060] The purpose of this step is to obtain the input data for the point-branch CNN when training the neural network, and the point cloud data can reflect the topological relationship between Couinaud segments of the liver.

[0061] First, the image coordinates I = {i 1 , i 2 ,..., i t , i t ∈ R 3} of each voxel in the CT image are transformed into world coordinates P = {p 1 , p 2 ,..., p t , p t ∈ R 3} through the following formula

[0062] P = I * Spacing * Sirection + Origin

[0063] Among them, Spacing is the voxel spacing of the CT image, Direction is the scanning direction, and Origin is the coordinate of the origin of the image coordinate system in the world coordinate system.

[0064] Subsequently, point sampling is performed based on the vascular attention map. Specifically, in each training cycle, the model randomly samples T points (such as 20,000) in the CT image. Among them, T / 2 (10,000) points fall within the small area covered by the vascular attention map (this enables the model to have a greater possibility of accessing important points in this area during training). In addition, we adopt a random perturbation deviation Offset = (Δx, Δy, Δz) within the range of [-1, 1], so that each point p t =(x t , y t , z t ) in M' generates a new point p' t =(x t +Δx, y t +Δy, z t +Δz), and the image intensity of the corresponding coordinates is obtained by trilinear interpolation. In the training stage of the network, the label of the point p' t is obtained by the following formula:[[]]

[0065] O t = O t (R(x t +Δx), R(y t +Δy), R(z t +Δz))

[0066] Among them, R is the rounding function.

[0067] (4) Voxelize the sampled 3D point cloud

[0068] The purpose of this step is to obtain the input data of the voxel-based CNN when training the neural network, so as to extract the corresponding semantic information through the voxel-based convolutional network

[0069] Specifically, the point-related data {(p t , f t )} is transformed into the voxel grid {V u，v，w} through voxelization, where f t ∈R c is the feature of the point p t . Then, the point coordinates {p t} are normalized to the range of [0, 1], represented by , and the corresponding features {ft} is not changed during the normalization process. Then, we take the average value of the features of all points falling within the voxel grid (u, v, w) as the feature of this voxel grid, and in this way transform the normalized point cloud data into the voxel grid {V u,v,w}, and the specific operation is shown in the following formula:

[0070]

[0071] where r is the voxel resolution, I[·] is a binary indicator function, which takes the value of 1 when the point belongs to the voxel grid (u, v, w), and 0 otherwise, f t，c represents the feature of the point in the c-th channel, and N u，v，w is the number of points falling within the voxel grid (u, v, w). It is worth mentioning that due to the previously adopted point sampling strategy, the transformed voxel grid still retains the vascular structure information from the point data, while also diluting the unimportant information in the CT images.

[0072] (5) Construct a multi-scale point cloud voxel fusion network and train

[0073] First, build a multi-scale point cloud voxel fusion network. The multi-scale point voxel fusion network contains two branches: a point cloud-based CNN and a voxel-based CNN. The features extracted by these two branches at multiple scales will be fused to provide more accurate and robust Couinaud segmentation performance.

[0074] Second, for the training and test datasets, the 3Dircadb and LiTS public datasets can be used. The 3Dircadb dataset contains 20 liver CT images with voxel spacings ranging from 0.56 mm to 0.87 mm and slice thicknesses from 1 mm to 4 mm, and has accurate annotations for liver and vascular segmentation. The LiTS dataset contains 200 CT images of the liver with voxel spacings ranging from 0.45 mm to 0.98 mm and slice thicknesses from 0.45 mm to 5 mm, and has annotations for the liver and liver tumors.

[0075] In addition, for the processing of the datasets: According to the guidance of the liver vascular annotations in the 3Dircadb dataset, 20 subjects in the dataset are annotated using the Couinaud segmentation method and randomly divided into 10 subjects for training and 10 subjects for testing. For the LiTS dataset, according to the vascular structure in the CT images, the Couinaud segments of 100 subjects can be annotated and randomly divided into 50 training samples and 50 test samples.

[0076] It should be noted that the dataset mentioned here needs to be processed through the above (3) to (4).

[0077] In addition, some hyperparameters during the training process can be selected as a learning rate of 0.01, the optimizer is selected as SDG, the learning rate is reduced to 90% of the previous cycle every 50 epochs, and 400 epochs are trained, etc.

[0078] The specific functions of the network are as follows:

[0079] For the point cloud-based CNN, the input point data {(p t , f t )} will pass through a multi-layer perceptron, denoted as E P , and this branch can extract fine-grained features with topological information;

[0080] For the voxel-based CNN, the voxel grid V u,v,w converted from the point data will be passed as input into this branch, denoted as E v , and this branch can aggregate the features of surrounding points and learn the semantic information in the 3D space of the liver.

[0081] In addition, the features extracted from the voxel-based branch are re-converted into point data representation by trilinear interpolation, and combined with the fine-grained features extracted from the point-based branch to provide supplementary information, as follows:

[0082]

[0083] Among them, the superscript 1 of represents that is the fused point data and the corresponding features obtained after the first round of point-voxel operation Subsequently, this point data will be voxelized again and passed into the two CNN branches. After three iterations of point-voxel operations, we concatenate the point features

[0084] (6) Use the trained multi-scale point cloud voxel fusion network to predict the Couinaud segmentation of the liver

[0085] After the neural network is trained, it can be put into use, that is, the prediction stage.

[0086] Input the liver image into the network, and the output can be obtained:

[0087]

[0088] Among them, {0, 1, ..., 7} represents eight classifications of the Couinaud segments.

[0089] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0090] Multiple components in the device are connected to the I / O interface, including: an input unit, such as a keyboard, a mouse, etc.; an output unit, such as various types of displays, speakers, etc.; a storage unit, such as a magnetic disk, an optical disc, etc.; and a communication unit, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit allows the device to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0091] The processing unit executes the various methods and processes described above, such as methods S1 to S4. For example, in some embodiments, methods S1 to S4 can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device via the ROM and / or the communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S4 described above can be executed. Alternatively, in other embodiments, the CPU can be configured to execute methods S1 to S4 in any other suitable manner (e.g., by means of firmware).

[0092] The functions described above herein can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0093] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program codes can be executed entirely on the machine, partially on the machine, executed partially on the machine and partially on a remote machine as an independent software package, or executed entirely on a remote machine or server.

[0094] In the context of the present invention, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] As described above, only the specific embodiments of the present invention are provided, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A liver Couinaud segmentation method based on point cloud and voxel fusion strategy, characterized in that: The method comprises the following steps: Step S1, extracting a liver mask from a CT image and generating a liver vascular attention map; Step S2, performing continuous point sampling of the liver in 3D space based on the liver vascular attention map; Step S3, voxelizing the sampled 3D point cloud; Step S4: Based on the topological relationship of the coordinate points in the 3D space and the semantic information of the voxel grid, a multi-scale point cloud voxel fusion network is constructed, and the Couinaud segmentation of the liver is predicted.

2. A liver Couinaud segmentation method based on point cloud and voxel fusion strategy according to claim 1, characterized in that: In the step S1, a 3D-Unet network model is used to generate a liver vascular attention map, specifically: first, a pre-trained 3D-Unet model is used to output a vascular binary mask M, and then morphological expansion is used to encapsulate vascular pixels in the area covered by the vascular binary mask M to generate a liver vascular attention map M′.

3. The method for Couinaud segmentation of liver based on point cloud and voxel fusion strategy according to claim 1, characterized in that: The step S2 comprises the following sub-steps: Step S21, the image coordinates of each voxel of the CT image are I={i1, i2, ..., i t ,i t ∈R 3 } is transformed into world coordinates P = {p1, p2, ..., p t , p t ∈R 3 }, the expression is: P=I*Spacing*Direction+Origin Among them, Spacing is the voxel spacing of the CT image, Direction is the scanning direction, and Origin is the coordinate of the origin of the image coordinate system in the world coordinate system; Step S22: performing continuous point sampling based on the liver vascular attention map.

4. The method for Couinaud segmentation of liver based on point cloud and voxel fusion strategy according to claim 3, characterized in that: The step S22 is specifically as follows: in each training cycle, randomly sample T points in the CT image, where T / 2 points fall within the first range area covered by the liver vascular attention map, and use a random perturbation offset Offset=(Δx, Δy, Δz) within the range of [-1, 1] so that each point p in the first range area t =(x t ,y t ,z t ) generate a corresponding new point p′ t =(x t +Δx,y t +Δy,z t +Δz), the image intensity at the corresponding coordinates is obtained by trilinear interpolation; During the training phase, the new point p′ t The labels are calculated based on the coordinate points processed by the normalization function.

5. The method for Couinaud segmentation of liver based on point cloud and voxel fusion strategy according to claim 4, characterized in that: The step S3 comprises: The point data (p t ,f t ) is converted to a voxel grid V by voxelization u,v,w , used for subsequent semantic information extraction; where f t ∈R c For point p t Features; The point coordinates p t Normalized to the range [0,1], we get Take the average of the feature values ​​of all points within the voxel grid (u, v, w) as the feature of the voxel grid, so as to normalize the point cloud data Transform to voxel grid V u,v,w .

6. The method for Couinaud segmentation of liver based on point cloud and voxel fusion strategy according to claim 5, characterized in that: The calculation expression of the voxel grid is: where r is the voxel resolution and I[·] is a binary indicator function. When it belongs to the voxel grid (u,v,w), its value is 1, otherwise it is 0, f t,c Representative Points The feature of the cth channel, N u,v,w is the number of points that fall within the bounds of the voxel grid (u,v,w).

7. The method for Couinaud segmentation of liver based on point cloud and voxel fusion strategy according to claim 1, characterized in that: The multi-scale point cloud voxel fusion network in step S4 includes a point cloud-based convolutional neural network branch and a voxel-based convolutional neural network branch. The feature fusion process is specifically as follows: The point data (p t ,f t ) is input to the point cloud-based convolutional neural network branch, and the multi-layer perceptron E P Extract fine-grained features with topological information; where f t ∈R c For point p t Features; The voxel grid V obtained by transforming the point data u,v,w Input to the voxel-based convolutional neural network branch E v , used to aggregate the features of surrounding points and learn the semantic information in the 3D space of the liver; The features extracted from the voxel-based convolutional neural network branch are converted back into point data representations by using trilinear interpolation, and the point data representations are combined with the fine-grained features extracted from the point cloud-based convolutional neural network branch to obtain fused point data and corresponding point features; The fused point data and corresponding features are voxelized again and passed to the two branches. After a set number of point voxel operation iterations, the obtained point features are concatenated and input into the point-by-point decoder for liver Couinaud segmentation prediction.

8. The method for Couinaud segmentation of liver based on point cloud and voxel fusion strategy according to claim 7, characterized in that: The point data representation is combined with the fine-grained features extracted from the point cloud-based convolutional neural network branch to obtain fused point data and corresponding point features, specifically: (p t ,f t 1 )=And P (P(p t ,f t ))+tri(E v (V)) t Among them, f t 1 The superscript 1 represents (p t ,f t 1 ) is the fusion point data obtained after the first round of point voxel operation and the corresponding feature f t 1 .

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images