Liver and skeleton three-dimensional reconstruction method for plain-scan CT (Computed Tomography) image based on semantic segmentation model

Abstract

The invention relates to the technical field of medical image processing and computer-aided diagnosis, in particular to a liver and skeleton three-dimensional reconstruction method of a plain-scan CT image based on a semantic segmentation model. The method comprises the following steps: 1) establishing a plain-scan CT database; 2) collecting plain-scan CT image data of a human body; 3) importing the CT image data into a three-dimensional reconstruction system; 4) performing image preprocessing on the CT plain-scan data according to a region of interest (ROI); 5) automatically labeling bones and livers in a plain-scan CT image by a semantic segmentation model embedded in a three-dimensional reconstruction system; (6) the system calibrates and fuses the marked CT images, and performs multi-view processing in the axial direction, the coronal direction and the sagittal direction; 7) generating a simulation three-dimensional model of the human body; according to the method, accurate three-dimensional modeling is carried out on the liver and the skeleton, so that a doctor can more intuitively know the relative position relation of a human anatomical structure, the safety and the treatment effect of an operation can be improved, and communication between the doctor and a patient is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing and computer-aided diagnosis, and in particular to a method for three-dimensional reconstruction of liver and bones based on plain scan CT images under a semantic segmentation model. Background Art

[0002] Three-dimensional reconstruction of the liver and bones is of great significance in medical diagnosis and surgical planning. Traditional plain scan CT images provide two-dimensional slice information. Doctors need to infer the three-dimensional anatomical structure through multiple slices, which is a complicated process and prone to errors. Existing three-dimensional reconstruction technology mainly relies on manual annotation, which is inefficient and has a great impact on the accuracy of the results. Therefore, a three-dimensional reconstruction method that can be automatically processed is needed to improve the modeling accuracy of the liver and bones and reduce the workload of doctors. Summary of the invention

[0003] In response to the challenges mentioned in the background technology section, the present invention provides an innovative method, a method for three-dimensional reconstruction of the liver and bones from plain scan CT images under a semantic segmentation model. By accurately modeling the liver and bones in three dimensions, doctors can intuitively understand the spatial position relationship between the liver and bones, thereby facilitating the location and evaluation of lesions, which is beneficial for surgical planning and treatment, and at the same time improves patients' understanding of their own condition. The method is practical and widely applicable.

[0004] The present invention provides a method for three-dimensional reconstruction of liver and bones based on plain scan CT images under a semantic segmentation model, which adopts the following technical solutions:

[0005] A method for three-dimensional reconstruction of liver and bones based on plain scan CT images under a semantic segmentation model comprises the following steps:

[0006] 1) Establish a plain scan CT database to form a document containing multiple sets of patient plain scan CT image data;

[0007] 2) Collect sufficient amount of human body plain scan CT image data and store the image data that meets the requirements in the patient file;

[0008] 3) Importing CT image data in the patient's file into the 3D reconstruction system;

[0009] 4) Image preprocessing of CT scan data according to the region of interest (ROI);

[0010] 5) The semantic segmentation model embedded in the 3D reconstruction system automatically annotates the bones and liver in the plain scan CT images;

[0011] 6) The system calibrates and fuses the annotated CT images and performs multi-view processing from three directions: axial, coronal, and sagittal;

[0012] 7) Generate three-dimensional models of the liver and bones and optimize the models.

[0013] Preferably, in step 4), the CT scan data is subjected to image preprocessing and image cropping operations according to the region of interest (ROI), namely:

[0014] cropped_image=original_image[x min :x max ,y min :y max ](1);

[0015] Preferably, in step 4), the CT scan data is subjected to image preprocessing and image resampling Resample operation according to the region of interest (ROI).

[0016] Right now:

[0017]

[0018] Preferably, in step 4), the CT scan data is subjected to image preprocessing and image normalization according to the region of interest (ROI), namely:

[0019]

[0020] Preferably, the loss function of the semantic segmentation model for automatically annotating the bones and liver in the plain scan CT image in step 5) is a combination of Dice loss and cross-entropy loss (CE) for segmenting the liver and bones in the CT image. The formula of the combined loss function can be expressed as:

[0021]

[0022] Substitute (4) (5) into (6), and (0<α<1)

[0023] L=α·L CE +(1-α)·L Dice (6);

[0024] Preferably, in step 5), the activation function of the semantic segmentation model for automatically annotating the bones and liver in the plain scan CT image is to use leaky ReLU as the activation function, replacing the commonly used ReLU activation function. The combination function of the formula of LeakyReLU is used to segment the liver and bones in the CT image, and the formula of the function can be expressed as:

[0025] f(x) = max(0.01x, x)(7);

[0026] Preferably, in step 5), the normalization method of the semantic segmentation model for automatically annotating the bones and liver in the plain scan CT image uses instance normalization instead of the more popular batch normalization. The formula of the normalization method Instance Normalization is:

[0027]

[0028] Preferably, the image calibration in step 6) adopts a registration method based on feature points to ensure the spatial alignment of data from each perspective. At the same time, the image fusion process integrates image information from different directions through a weighted average method, and the weights are dynamically adjusted according to actual needs to ensure that the final generated 3D model has higher accuracy and consistency.

[0029] Preferably, in step 7), for the reconstruction of the liver and bone models, the transition between adjacent CT images is processed by using an insert and isosurface extraction method to achieve a smooth connection of the model. According to each vertex of the three-dimensional data, it is determined whether its value is greater than or equal to a given isovalue. If so, it is marked as 1; otherwise, it is marked as 0; that is:

[0030]

[0031] Preferably, in step 7), the display of the three-dimensional reconstructed organ uses a cmap color mapping method. The generated three-dimensional model is visualized through a mapping and rendering module. The system uses the following method to determine the color (Color) and transparency (Opacity) of each model:

[0032]

[0033] Opacity(x,y,z)=calculateOpacity(f(x,y,z)) (11);

[0034] Preferably, in step 7), a multi-threaded scheduling method is used for loading the three-dimensional reconstructed organ. During the three-dimensional reconstruction process, the system will display the current progress to ensure that the user can understand the processing progress in real time. The progress update method is as follows:

[0035]

[0036] The beneficial effects of the present invention are:

[0037] 1. Accurate segmentation and enhanced display: The present invention uses advanced semantic segmentation algorithms to accurately identify and segment tissue structures such as the liver and bones, as well as lesions in the original CT images, especially in low-contrast conditions. This precise segmentation combined with 3D reconstruction technology allows doctors to clearly see the relative position and morphology of lesions and important anatomical structures, thereby providing more accurate information for surgical planning.

[0038] 2. Multi-level and multi-dimensional image fusion: The present invention can fuse multi-level CT images to generate three-dimensional images with multi-dimensional information. By adjusting the transparency and color mapping of each image layer, the characteristics of different anatomical structures can be highlighted, making it easier for doctors to comprehensively evaluate the location of the lesion and its relationship with surrounding structures. This multi-level image fusion technology can provide more sophisticated image analysis capabilities in complex cases.

[0039] 3. Fast processing and real-time interaction: The image processing method and three-dimensional reconstruction algorithm involved in the present invention are optimized to complete the reconstruction and processing of large amounts of CT images in a short time. Through the real-time interaction function, doctors can adjust the viewing angle, rotate the model or switch to different display modes at any time, making the image analysis process more efficient and greatly shortening the time for diagnosis and treatment planning.

[0040] 4. Personalized treatment plan formulation: The precise 3D reconstruction and image analysis technology provided by the present invention can help doctors formulate personalized treatment plans based on the individual anatomical characteristics of the patient. Through in-depth analysis of the 3D model, doctors can better judge the feasibility of the treatment plan and choose the best treatment path and method, thereby improving the treatment effect and reducing risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 It is a flow chart of a method for three-dimensional reconstruction of liver and bones based on plain scan CT images under a semantic segmentation model of the present invention;

[0043] Figure 2 It is the plain scan CT liver semantic segmentation map of the present invention;

[0044] Figure 3 It is the plain scan CT skeleton semantic segmentation map of the present invention;

[0045] Figure 4 It is the plain scan CT liver and bone semantic segmentation map of the present invention;

[0046] Figure 5 A schematic diagram of liver modeling according to the present invention;

[0047] Figure 6 A schematic diagram of the skeleton modeling of the present invention;

[0048] Figure 7 This is a schematic diagram of modeling after registration of the liver skeleton and CT image of the present invention; DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] An embodiment of a method for three-dimensional reconstruction of liver and bones based on plain scan CT images under a semantic segmentation model of the present invention is as follows: Figure 1 As shown, including:

[0051] See also Figure 1 The method for 3D reconstruction of liver and bones based on plain scan CT images under a semantic segmentation model includes the following steps:

[0052] 1) Establish a plain scan CT database to form a document containing multiple sets of patient plain scan CT image data;

[0053] 2) Collect sufficient amount of human body plain scan CT image data and store the image data that meets the requirements in the patient file;

[0054] 3) Importing CT image data in the patient's file into the 3D reconstruction system;

[0055] 4) Image preprocessing of CT scan data according to the region of interest (ROI);

[0056] 5) The semantic segmentation model embedded in the 3D reconstruction system automatically annotates the bones and liver in the plain scan CT images;

[0057] 6) The system calibrates and fuses the annotated CT images and performs multi-view processing from three directions: axial, coronal, and sagittal;

[0058] 7) Generate three-dimensional models of the liver and bones and optimize the models.

[0059] Furthermore, in step 4), the image preprocessing includes an image cropping operation, that is, cropping the CT image data according to the region of interest (ROI) to remove invalid areas. The formula is:

[0060] cropped_image=original_image[x min :x max ,y min :y max ](1);

[0061] where x min and x max are the minimum and maximum indices of the clipping area in the x direction, and y m i n and max are the minimum and maximum indices in the y direction.

[0062] The padding operation may need to use a function such as binary_fill_holes to fill small holes in the mask to ensure the accuracy of the bounding box.

[0063] If necessary, the cropped image can be resized to a specific size, using the nearest neighbor interpolation method. The formula is:

[0064] y(x)=y i ,in

[0065] x j is the location of the known data point, Indicates the selection makes |xx j |The smallest j, that is, the known data point x closest to the target point x j ,y i is the value corresponding to the most recent data point.

[0066] Furthermore, in step 4), image preprocessing includes a resampling operation, that is, resampling the CT image data to ensure a uniform resolution and coordinate system. The formula is:

[0067]

[0068] Where original_spacing is the original voxel distance, target_spacing is the target voxel distance, and original_dimension is the original image size

[0069] When resizing an image, you need to choose a suitable interpolation method to calculate the pixel value at the new position. The interpolation method used here is nearest neighbor interpolation.

[0070] Furthermore, in step 4), image preprocessing includes a normalization operation, that is, normalizing the CT image data to improve the accuracy of liver and bone region annotation and the rapid convergence of the model. The formula is:

[0071]

[0072] Where X represents the original pixel value, μ is the average value (mean) of the pixel value, σ is the standard deviation of the pixel value, and Z is the standardized pixel value.

[0073] Semantic segmentation: The semantic segmentation model in the 3D reconstruction system is used to automatically annotate the bones and liver in CT images.

[0074] Furthermore, in step 5), the semantic segmentation model uses deep learning technology, and its loss function is a combination of Dice loss and cross-entropy loss (CE) to optimize the segmentation result. The formula of the combined loss function is:

[0075]

[0076] Where X is the predicted segmentation result, Y is the actual segmentation result, |X∩Y| 是预测结果和真实结果的交集的大小 , X| and |Y| are the total areas of the predicted results and the true results, respectively.

[0077]

[0078] Where M is the number of categories, y o,c is the one-hot encoding of the true label, p o,c is the probability that the model predicts class c.

[0079] Substitute (1)(2) into (3)

[0080] L=α·L CE +(1-α)·L Dice (7);

[0081] Among them, L is the total loss, L CE is the cross entropy loss, which measures the difference between the model’s predicted probability distribution and the true label, L Dice is the Dice loss, which measures the similarity between the predicted segmentation result and the true segmentation result. α is a weight parameter used to balance the two losses, usually between 0 and 1.

[0082] Furthermore, in step 5), the semantic segmentation model uses LeakyReLU as the activation function to replace the traditional ReLU activation function. The formula of LeakyReLU is:

[0083]

[0084] Among them, α is the leakage coefficient.

[0085] Furthermore, in step 5), the semantic segmentation model uses instance normalization as a normalization method to replace the traditional batch normalization method. The formula for Instance Normalization is:

[0086]

[0087] Where x is the input, μ is the mean of x, σ is the standard deviation of x, and γ and β are learnable parameters for scaling and translation operations.

[0088] Furthermore, the image calibration in step 6) uses a feature point-based registration method to ensure the spatial alignment of the data from each perspective. At the same time, the image fusion process integrates the image information from different directions through a weighted average method, and the weights are dynamically adjusted according to actual needs to ensure that the final generated 3D model has higher accuracy and consistency.

[0089] Furthermore, in step 7), for the reconstruction of the liver and bone models, the transition between adjacent CT images is processed by using the method of insert and isosurface extraction to achieve smooth connection of the model. According to each vertex of the three-dimensional data, it is determined whether its value is greater than or equal to the given isovalue. If so, it is marked as 1; otherwise, it is marked as 0; that is:

[0090]

[0091] Furthermore, in step 7), the cmap color mapping method is used for displaying the 3D reconstructed organs. The generated 3D model is visualized through the mapping and rendering modules. The system uses the following method to determine the color (Color) and transparency (Opacity) of each model:

[0092]

[0093] Opacity(x,y,z)=calculateOpacity(f(x,y,z)) (12);

[0094] Furthermore, in step 7), multi-threaded scheduling is used for loading the 3D reconstructed organs. During the 3D reconstruction process, the system will display the current progress to ensure that the user can understand the processing progress in real time. The progress update method is as follows:

[0095]

[0096] After executing the above process on the plain scan image, the following Figure 2-7 effect.

[0097] Advantages of the present invention:

[0098] 1) High-precision 3D reconstruction: The method of 3D reconstruction based on plain scan CT images can provide a high-precision 3D model. By carefully preprocessing and optimizing CT image data, the 3D structure of the liver and bones can be accurately reconstructed, which helps doctors make accurate diagnoses and treatment plans.

[0099] 2) Automated labeling and optimization: Deep learning technology is used to automatically segment CT images, semi-automatically label the liver and bone areas, and optimize the model. This automated processing reduces the tediousness of manual operations and subjective errors, and improves work efficiency and consistency of results.

[0100] 3) Comprehensive multi-view analysis: By processing the annotated CT images from multiple perspectives, the three-dimensional structure of the liver and bones can be comprehensively analyzed from different directions. This comprehensive perspective analysis helps to fully understand the spatial relationship and structural characteristics of the target area.

[0101] 4) Enhanced model quality: During the 3D model generation process, the present invention enhances the quality and smoothness of the model through insert technology and optimization algorithm. This process can eliminate noise and holes on the model surface, ensure the continuity and accuracy of the 3D model, and facilitate subsequent medical analysis and surgical planning.

[0102] 5) Optimization processing capability: When dealing with complex structures such as vascular bifurcations, the present invention combines fluid mechanics and biological principles to optimize the modeling of blood vessels. This method can truly reflect the biological characteristics of blood vessels and improve the biological rationality and practicality of the model.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope defined in the claims.

Claims

1. A method for three-dimensional reconstruction of liver and bones based on plain scan CT images, characterized in that: The following steps are involved: 1) Establish a plain scan CT database to form a document containing multiple sets of patient plain scan CT image data; 2) Collect sufficient amount of human body plain scan CT image data and store the image data that meets the requirements in the patient file; 3) Importing CT image data in the patient's file into the 3D reconstruction system; 4) Image preprocessing of CT scan data according to the region of interest (ROI); 5) The semantic segmentation model embedded in the 3D reconstruction system automatically annotates the bones and liver in the plain scan CT images; 6) The system calibrates and fuses the annotated CT images and performs multi-view processing from three directions: axial, coronal, and sagittal; 7) Generate three-dimensional models of the liver and bones and optimize the models.

2. The method for three-dimensional reconstruction of liver and bones based on plain scan CT images according to claim 1, characterized in that: The image preprocessing in step 4) includes the image cropping operation, namely: cropped_image=original_image[x min :x max ,y min :y max ] (1)。 3. The method for three-dimensional reconstruction of liver and bones based on plain scan CT images according to claim 1, characterized in that: The image preprocessing in step 4) includes the resampling Resample operation, namely:

4. The method for three-dimensional reconstruction of liver and bones based on plain scan CT images according to claim 1, characterized in that: In step 4), the image preprocessing includes the normalization operation, namely:

5. The method for three-dimensional reconstruction of liver and bones based on plain scan CT images according to claim 1, characterized in that: In step 5), the semantic segmentation model uses deep learning technology, and its loss function is a combination of Dice loss and cross-entropy loss (CE) to segment the liver and bones in CT images. The formula of the combined loss function can be expressed as: Substitute (4) (5) into (6), and (0<α<1) L=α·L CE +(1-a)·L Dice (6)。 6. The method for three-dimensional reconstruction of liver and bones based on plain scan CT images according to claim 1, characterized in that: In step 5), the semantic segmentation model uses deep learning technology, and its activation function uses leakyReLU as the activation function, replacing the commonly used ReLU activation function. The combination function of the LeakyReLU formula is used to segment the liver and bones in the CT image. The function formula can be expressed as: f(x)=max(0.01x,x)(7).

7. The method for three-dimensional reconstruction of liver and bones based on plain scan CT images according to claim 1, characterized in that: In step 5), the semantic segmentation model uses deep learning technology, and its normalization method uses instance normalization instead of the more popular batch normalization. The formula of the normalization method Instance Normalization is:

8. The method for three-dimensional reconstruction of liver and bones based on plain scan CT images according to claim 1, characterized in that: The image calibration in step 6) uses a feature point-based registration method to ensure the spatial alignment of data from each perspective. At the same time, the image fusion process uses a weighted average method to integrate image information from different directions. The weights are dynamically adjusted according to actual needs to ensure that the final generated 3D model has higher accuracy and consistency.

9. The method for three-dimensional reconstruction of liver and bones based on plain scan CT images according to claim 1, characterized in that: In step 7), for the reconstruction of the liver and bone models, the transition between adjacent CT images is processed by using the method of insert and isosurface extraction to achieve smooth connection of the model. According to each vertex of the three-dimensional data, it is determined whether its value is greater than or equal to the given isovalue. If so, it is marked as 1; otherwise, it is marked as 0; that is:

10. The method for three-dimensional reconstruction of liver and bones based on plain scan CT images according to claim 1, characterized in that: In step 7), the cmap color mapping method is used to display the 3D reconstructed organs. The generated 3D model is visualized through the mapping and rendering modules. The system uses the following method to determine the color (Color) and transparency (Opacity) of each model: Opacity(x,y,z)=calculateOpacity(f(x,y,z)) (11).

11. The method for three-dimensional reconstruction of liver and bones based on plain scan CT images according to claim 1, characterized in that: In step 7), multi-threaded scheduling is used to load the 3D reconstructed organs. During the 3D reconstruction process, the system will display the current progress to ensure that the user can understand the processing progress in real time. The progress update method is as follows: 。

Images