A method, system, storage medium, and device for dynamic three-dimensional reconstruction of organ soft tissue

Abstract

This invention provides a method, system, storage medium, and device for dynamic three-dimensional reconstruction of organ soft tissue. The method includes: acquiring a static CT image sequence, segmenting the target organ, and extracting the organ contour to generate a CT point cloud based on the CT coordinate system; acquiring an ultrasound image sequence collected by a positioning sensor attached to an ultrasound probe, segmenting the target organ, and extracting the organ contour to generate an ultrasound point cloud based on the positioning coordinate system; matching the positions of the CT point cloud and the ultrasound point cloud through rigid registration to obtain a rigidly transformed CT point cloud; correcting the rigidly transformed CT point cloud through elastic registration to generate a deformed CT point cloud; and generating a three-dimensional surface of the organ from the deformed CT point cloud using a surface generation algorithm, thereby realizing dynamic three-dimensional reconstruction based on static CT images.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a method, system, storage medium, and device for dynamic three-dimensional reconstruction of organ soft tissue. Background Technology

[0002] In medical diagnosis and surgical planning, accurately acquiring dynamic three-dimensional morphological information of organs is crucial; real-time three-dimensional reconstruction technology of organ soft tissues is of great value for dynamically monitoring organ morphological changes and assisting in clinical diagnosis and treatment decisions. However, the practical application of this technology faces many challenges: First, the position and morphology of soft tissues are subject to significant dynamic changes due to factors such as respiratory movements, organ peristalsis, and changes in body position; second, traditional three-dimensional reconstruction algorithms have high computational complexity, making it difficult to meet the real-time requirements of clinical practice.

[0003] Among existing solutions, while continuous CT scanning can provide high-resolution images, it has two major limitations: first, the equipment operating cost is high, making continuous scanning difficult to routinely apply in clinical practice; second, continuous radiation exposure may harm patients. In contrast, although ultrasound imaging has the advantage of real-time performance, its spatial resolution is significantly lower than that of CT, and it has inherent limitations in displaying fine soft tissue structures.

[0004] Therefore, how to achieve high-precision, low-cost real-time dynamic reconstruction has become a key scientific problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the problem of how to achieve high-precision and low-cost dynamic three-dimensional organ reconstruction by proposing a method, system, storage medium, and device for dynamic three-dimensional reconstruction of organ soft tissue.

[0006] The technical solution of this invention is:

[0007] This invention provides a method for dynamic three-dimensional reconstruction of organ soft tissue, comprising:

[0008] S1. Obtain a static CT image sequence, segment the target organ, extract the organ contour, and generate a CT point cloud based on the CT coordinate system.

[0009] S2. Obtain the ultrasound image sequence acquired by the positioning sensor attached to the ultrasound probe, segment the target organ, extract the organ contour, and generate an ultrasound point cloud based on the positioning coordinate system.

[0010] S3. Match the positions of CT point cloud and ultrasound point cloud through rigid registration to obtain the rigidly transformed CT point cloud;

[0011] S4. Correct the rigidly transformed CT point cloud by elastic registration to generate the deformed CT point cloud.

[0012] S5. Generate the three-dimensional surface of the organ from the deformed CT point cloud using a surface generation algorithm, thereby realizing dynamic three-dimensional reconstruction based on static CT images.

[0013] Furthermore, S3 includes:

[0014] The Iterative Closest Point (ICP) algorithm is used to determine the rigid transformation relationship between the CT point cloud and the ultrasound point cloud, and to obtain the rotation matrix R and the translation vector t.

[0015] The CT point cloud is rigidly transformed by the rotation matrix R and the translation vector t to obtain the rigidly transformed CT point cloud.

[0016] Furthermore, S4 includes:

[0017] The rigidly transformed CT point cloud and the ultrasound point cloud are input into a pre-trained elastic registration network;

[0018] The displacement vector field is generated through the elastic registration network. ;

[0019] Predict the displacement of each point in the rigidly transformed CT point cloud based on the displacement vector field. The deformed CT point cloud is generated using the following formula;

[0020]

[0021] in, i This represents the number of each point in the source point cloud, i.e., the CT point cloud after rigid transformation. Indicates the first i The spatial location of each point.

[0022] Furthermore, the elastic registration network is constructed based on a point cloud backbone network, and the loss function of the elastic registration network includes matching error constraints, local rigidity constraints, and smoothness constraints.

[0023] Furthermore, the matching error constraint calculates the error of the overlapping portion between the rigidly transformed CT point cloud and the ultrasound point cloud. The calculation formula is:

[0024]

[0025] in, x This represents the spatial position of any point in the CT point cloud after rigid transformation. This represents the set of point clouds representing the overlapping portion of the CT point cloud and the ultrasound point cloud after rigid transformation. y This represents the spatial position of any point in the ultrasonic point cloud. Y This represents a set of ultrasonic point clouds.

[0026] Furthermore, the local rigid constraint maintains the local distance of the deformed CT point cloud, and the local rigid constraint The calculation formula is:

[0027]

[0028] in, X This represents the CT point cloud set after rigid transformation; express neighborhood points, j Indicates the neighboring point number; yes The set of neighborhood points; Indicates the first point in the source cloud i The spatial position of a point obtained by displacement vector field transformation Indicates the number is j The spatial positions of the neighborhood points are obtained after the displacement vector field transformation.

[0029] Furthermore, the smoothness constraint penalizes the gradient change of the displacement vector field, the smoothness constraint The calculation formula is:

[0030]

[0031] in, F Denotes the Frobenius norm; Indicates the first point in the source cloud i The gradient of the displacement vector field at each point.

[0032] Further, S5 includes:

[0033] The Marching Cube algorithm is used to process the deformed CT point cloud to generate organ surface mesh;

[0034] The organ's three-dimensional surface is constructed based on the organ's surface mesh, resulting in a dynamic three-dimensional reconstruction.

[0035] A system used in a method for dynamic three-dimensional reconstruction of organ soft tissue includes:

[0036] The CT image processing module is used to acquire static CT image sequences, segment target organs, extract organ contours, and generate CT point clouds based on the CT coordinate system.

[0037] An ultrasound image processing module is used to acquire ultrasound image sequences, segment target organs, extract organ contours to generate ultrasound point clouds based on a positioning coordinate system, wherein the ultrasound image sequences are acquired by a positioning sensor attached to an ultrasound probe.

[0038] The rigid registration module is used to match the positions of CT point clouds and ultrasound point clouds through rigid registration to obtain the rigidly transformed CT point cloud;

[0039] The elastic registration module is used to correct the rigidly transformed CT point cloud through elastic registration, and generate a deformed CT point cloud.

[0040] The 3D reconstruction module is used to generate the 3D surface of the organ from the deformed CT point cloud using a surface generation algorithm, thereby realizing dynamic 3D reconstruction based on static CT images.

[0041] A computer-readable storage medium having a computer program thereon, which, when executed, implements the method for dynamic three-dimensional reconstruction of organ soft tissue.

[0042] An ultrasound device includes: one or more processors;

[0043] The processor is used to store one or more programs;

[0044] When the one or more programs are executed by the one or more processors, the method for dynamic three-dimensional reconstruction of organ soft tissue is implemented.

[0045] The beneficial effects of this invention are:

[0046] This invention proposes a method, system, storage medium, and device for dynamic three-dimensional reconstruction of organ soft tissue. It is based on point cloud fitting technology using rigid / elastic hybrid model registration, which uses partial point cloud to fit the complete three-dimensional morphology of organ soft tissue.

[0047] The method of this invention aligns the point clouds from CT and ultrasound images in spatial position through rigid registration, and then aligns the deformed part of the point cloud (i.e. the overlapping part of the ultrasound point cloud and CT point cloud) through elastic registration; it uses a pre-trained deep neural network to dynamically generate organ contour point clouds, and then uses a marching cube algorithm to generate the organ surface from the point cloud, thereby realizing a dynamic real-time three-dimensional reconstruction method based on static CT images.

[0048] This invention is a dynamic real-time three-dimensional organ reconstruction method based on CT and ultrasound images, which solves the problem that traditional static CT images cannot reflect real-time organ deformation. The method acquires CT and ultrasound data and performs registration and reconstruction to achieve dynamic updates of organ morphology. It effectively integrates the high resolution of CT and the real-time performance of ultrasound, and can accurately reflect the real-time deformation of organs, providing important support for precise surgical navigation.

[0049] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0050] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the invention.

[0051] Figure 1 A flowchart of a method for dynamic three-dimensional reconstruction of organ soft tissue according to an embodiment of the present invention is shown.

[0052] Figure 2 A schematic diagram illustrating the implementation process of a dynamic three-dimensional reconstruction system for soft tissue of an organ according to an embodiment of the present invention is shown. Detailed Implementation

[0053] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0054] Example 1

[0055] like Figure 1 , 2 As shown, this invention provides a method for dynamic three-dimensional reconstruction of organ soft tissue, which includes the following steps:

[0056] S1. Obtain a static CT image sequence, segment the target organ, extract the organ contour, and generate a CT point cloud based on the CT coordinate system.

[0057] In this step, human CT data is imported, and the target soft tissue is segmented in the CT sequence images. Traditional segmentation methods or neural networks can be used for segmentation. The contours of the organ segments are extracted, and uniform sampling is performed to obtain the organ contour point cloud under CT.

[0058] S2. Obtain the ultrasound image sequence acquired by the positioning sensor attached to the ultrasound probe, segment the target organ, extract the organ contour to generate an ultrasound point cloud based on the positioning coordinate system; the positioning sensor includes a magnetic navigation sensor or an optical navigation sensor.

[0059] In this step, a magnetic or optical navigation sensor is attached to an ultrasound probe, which is placed on the body surface (scanning organs such as the liver and kidneys) or through natural cavities (such as the prostate) to scan soft tissues. The probe is rotated to ensure that the ultrasound covers as much of the organ as possible. However, due to limitations in the ultrasound imaging range or bone obstruction, some organs (such as the kidneys) may be difficult to scan completely. Based on the ultrasound, the organ is segmented, and the segmented contour is extracted and uniformly sampled to obtain the ultrasound-guided organ contour point cloud (based on electromagnetic coordinates). Since some organs cannot be scanned completely, the ultrasound-guided point cloud can be considered a partial point cloud.

[0060] S3. Match the positions of CT point cloud and ultrasound point cloud through rigid registration to obtain the rigidly transformed CT point cloud;

[0061] Specifically, the Iterative Closest Point (ICP) algorithm is used to determine the rigid transformation relationship between the CT point cloud and the ultrasound point cloud, and the rotation matrix R and translation vector t are obtained; the CT point cloud is then subjected to a rigid transformation based on the rotation matrix R and translation vector t to obtain the rigidly transformed CT point cloud.

[0062] In this step, rigid registration based on ICP is employed to match the CT point cloud and the ultrasound point cloud in terms of location. The rotation matrix R and translation vector t are obtained by solving for the optimal rigid transformation; the rigidly transformed CT point cloud is then obtained through the rigid transformation relationship.

[0063] S4. Correct the rigidly transformed CT point cloud by elastic registration to generate the deformed CT point cloud.

[0064] S5. Generate the three-dimensional surface of the organ from the deformed CT point cloud using a surface generation algorithm, thereby realizing dynamic three-dimensional reconstruction based on static CT images.

[0065] In this step, the Marching Cube algorithm is used to process the deformed CT point cloud and generate an organ surface mesh. Based on the organ surface mesh, the three-dimensional surface of the organ is constructed to obtain a dynamic three-dimensional reconstruction result.

[0066] If the dynamic 3D reconstruction result contains surface discontinuities, the organ surface mesh is optimized through mesh smoothing to obtain a smooth 3D surface. Based on the smooth 3D surface, combined with the texture information of the static CT image, the final dynamic 3D reconstruction result is generated.

[0067] In one example, S4 includes inputting the rigidly transformed CT point cloud and the ultrasound point cloud into a pre-trained elastic registration network; and generating a displacement vector field through the elastic registration network. Predict the displacement of each point in the rigidly transformed CT point cloud based on the displacement vector field. The deformed CT point cloud is generated using the following formula;

[0068]

[0069] in, i This represents the number of each point in the source point cloud, i.e., the CT point cloud after rigid transformation. Indicates the first i The spatial location of each point.

[0070] The displacement vector field is initialized as a zero vector field, meaning the displacement of each point in the source point cloud is 0. It is then optimized using gradient descent (optimized based on the error during training). During training, the CT point cloud is used as the source point cloud input, the ultrasound point cloud as the target point cloud input, and the displacement vector field as the output. The entire elastic registration network predicts the displacement vector field so that the error between the deformed CT point cloud obtained after elastic registration and the ultrasound point cloud after rigid transformation meets the training requirements. The magnitude of the vector field is the same as that of the CT point cloud, and each value represents the displacement of each point in the CT point cloud. The optimization equation is solved using gradient descent of the neural network to determine the predicted displacement of the point cloud.

[0071] Specifically, the elastic registration network is constructed based on a point cloud backbone network, and the loss function of the elastic registration network includes matching error constraints. Local rigid constraints and smoothness constraints ;

[0072] The matching error constraint calculates the error in the overlapping portion of the rigidly transformed CT point cloud and the ultrasound point cloud. The calculation formula is:

[0073]

[0074] in, x This represents the spatial position of any point in the CT point cloud after rigid transformation. This represents the set of point clouds representing the overlapping portion of the CT point cloud and the ultrasound point cloud after rigid transformation. y This represents the spatial position of any point in the ultrasonic point cloud. Y This represents a set of ultrasound point clouds. Due to the incompleteness of ultrasound point cloud acquisition (e.g., a complete point cloud covering the entire organ cannot be acquired for the kidney due to obstruction by ribs), directly performing non-rigid registration between the incomplete ultrasound point cloud and the rigidly transformed CT point cloud will introduce significant errors. This invention designs the matching error as the overlapping portion between the source point cloud (rigidly transformed CT point cloud) and the target point cloud (ultrasound point cloud). Only the overlapping portion is considered for non-rigid transformation; the non-overlapping portion is fitted using local rigid constraints and smoothness constraints.

[0075] Wherein, the local rigid constraint maintains the local distance of the deformed CT point cloud, and the local rigid constraint The calculation formula is:

[0076]

[0077] in, i This represents the number of each point in the source point cloud, i.e., the CT point cloud after rigid transformation. Indicates the first i The spatial location of each point X This represents the CT point cloud set after rigid transformation; express neighborhood points, j Indicates the neighboring point number; yes The set of neighborhood points; Indicates the first point in the source cloud i The spatial position of a point obtained by displacement vector field transformation Indicates the number is j The spatial positions of the neighboring points are obtained after transformation by the displacement vector field. Local rigid constraints are used to ensure the smoothness of the point cloud positions, that is, after the positions of some overlapping point clouds change due to non-rigid registration, the remaining point clouds are constrained to smoothly change their positions.

[0078] The smoothness constraint penalizes the gradient change of the displacement vector field. The calculation formula is:

[0079]

[0080] in, i This represents the number of each point in the source point cloud, i.e., the CT point cloud after rigid transformation. Indicates the first i The spatial location of each point; F Denotes the Frobenius norm; Indicates the first point in the source cloud i The gradient of the displacement vector field at each point. The smoothing constraint penalty for the gradient change of the displacement vector field is to ensure the smoothness of the point cloud displacement and avoid drastic changes in displacement.

[0081] In this example, the rigidly transformed CT point cloud data and ultrasound point cloud data are obtained as the initial source point cloud. X and target point cloud Y Spatial location. Based on the initial source point cloud. X Generate displacement vector field Predict each point displacement ,in Through the displacement vector field Obtain the location of the CT point cloud after deformation .

[0082] Determine the point cloud after deformation The overlapping part with the target point cloud Y If there is overlap, then extract. The point set in the [reference] is calculated using matching error constraints. Each point in Y Nearest point y The matching error is obtained by summing the squared distances. Local rigidity constraints are used to emphasize the rigid transformation of the local neighborhood of each point, resulting in a local rigidity loss. The gradient change of the displacement vector field is calculated based on smoothness constraints, yielding the smoothness loss. Finally, the matching error constraints are fused. Local rigid constraints and smoothness constraints A comprehensive loss function for the elastic registration network is constructed to optimize the displacement vector field. Based on the optimized displacement vector field, the deformed CT point cloud is updated to determine the final organ contour point cloud.

[0083] Example 2:

[0084] This application also provides a dynamic three-dimensional reconstruction system for organ soft tissue. It should be noted that this system can be used to execute the dynamic three-dimensional reconstruction method for organ soft tissue provided in this application, including:

[0085] The CT image processing module is used to acquire static CT image sequences, segment target organs, extract organ contours, and generate CT point clouds based on the CT coordinate system.

[0086] An ultrasound image processing module is used to acquire ultrasound image sequences, segment target organs, extract organ contours to generate ultrasound point clouds based on a positioning coordinate system, wherein the ultrasound image sequences are acquired by a positioning sensor attached to an ultrasound probe.

[0087] The rigid registration module is used to match the positions of CT point clouds and ultrasound point clouds through rigid registration to obtain the rigidly transformed CT point cloud;

[0088] The elastic registration module is used to correct the rigidly transformed CT point cloud through elastic registration, and generate a deformed CT point cloud.

[0089] The 3D reconstruction module is used to generate the 3D surface of the organ from the deformed CT point cloud using a surface generation algorithm, thereby realizing dynamic 3D reconstruction based on static CT images.

[0090] Example 3:

[0091] This invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method for dynamic three-dimensional reconstruction of organ soft tissue.

[0092] Example 4

[0093] This invention provides an ultrasonic device, including: one or more processors;

[0094] The processor is used to store one or more programs;

[0095] When the one or more programs are executed by the one or more processors, the method for dynamic three-dimensional reconstruction of organ soft tissue is implemented.

[0096] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A method for dynamic three-dimensional reconstruction of organ soft tissue, characterized in that, include: S1. Obtain a static CT image sequence, segment the target organ, extract the organ contour, and generate a CT point cloud based on the CT coordinate system. S2. Obtain the ultrasound image sequence acquired by the positioning sensor attached to the ultrasound probe, segment the target organ, extract the organ contour, and generate an ultrasound point cloud based on the positioning coordinate system. S3. Match the positions of CT point cloud and ultrasound point cloud through rigid registration to obtain the rigidly transformed CT point cloud; S4. Correct the rigidly transformed CT point cloud by elastic registration to generate the deformed CT point cloud. S5. Generate the three-dimensional surface of the organ from the deformed CT point cloud using a surface generation algorithm to achieve dynamic three-dimensional reconstruction based on static CT images. S4 includes: The rigidly transformed CT point cloud and the ultrasound point cloud are input into a pre-trained elastic registration network; a displacement vector field is generated through the elastic registration network. Predict the displacement of each point in the rigidly transformed CT point cloud based on the displacement vector field. The deformed CT point cloud is generated using the following formula; ； Where i represents the number of each point in the source point cloud, i.e., the CT point cloud after rigid transformation. This represents the spatial location of the i-th point; The elastic registration network is constructed based on a point cloud backbone network. The loss function of the elastic registration network includes matching error constraints, local rigidity constraints, and smoothness constraints. The matching error constraints calculate the error in the overlapping portion between the rigidly transformed CT point cloud and the ultrasound point cloud. The calculation formula is: ； Where x represents the spatial position of any point in the CT point cloud after rigid transformation. y represents the set of point clouds in the overlapping area of ​​the CT point cloud and the ultrasound point cloud after rigid transformation; y represents the spatial position of any point in the ultrasound point cloud; and Y represents the set of ultrasound point clouds.

2. The method for dynamic three-dimensional reconstruction of organ soft tissue as described in claim 1, characterized in that S3 include: The Iterative Closest Point (ICP) algorithm is used to determine the rigid transformation relationship between the CT point cloud and the ultrasound point cloud, and to obtain the rotation matrix R and the translation vector t. The CT point cloud is rigidly transformed by the rotation matrix R and the translation vector t to obtain the rigidly transformed CT point cloud.

3. The method for dynamic three-dimensional reconstruction of organ soft tissue as described in claim 1, characterized in that, The local rigid constraint maintains the local distance of the deformed CT point cloud. The calculation formula is: ； Where X represents the CT point cloud set after rigid transformation; express The neighboring points, j represents the neighboring point number; yes The set of neighborhood points; This represents the spatial position of the i-th point in the source point cloud after transformation by the displacement vector field. This represents the spatial position of the neighboring point j after the displacement vector field transformation.

4. The method for dynamic three-dimensional reconstruction of organ soft tissue as described in claim 1, characterized in that, The smoothness constraint penalizes the gradient change of the displacement vector field, the smoothness constraint The calculation formula is: ； Where F represents the Frobenius norm; X represents the gradient of the displacement vector field of the i-th point in the source point cloud; X represents the CT point cloud set after rigid transformation.

5. The method for dynamic three-dimensional reconstruction of organ soft tissue as described in claim 1, characterized in that... S5 includes: The Marching Cube algorithm is used to process the deformed CT point cloud to generate organ surface mesh; The organ's three-dimensional surface is constructed based on the organ's surface mesh, resulting in a dynamic three-dimensional reconstruction.

6. A system used in the dynamic three-dimensional reconstruction method for organ soft tissue according to any one of claims 1 to 5, characterized in that, include: The CT image processing module is used to acquire static CT image sequences, segment target organs, extract organ contours, and generate CT point clouds based on the CT coordinate system. An ultrasound image processing module is used to acquire ultrasound image sequences, segment target organs, extract organ contours to generate ultrasound point clouds based on a positioning coordinate system, wherein the ultrasound image sequences are acquired by a positioning sensor attached to an ultrasound probe. The rigid registration module is used to match the positions of CT point clouds and ultrasound point clouds through rigid registration to obtain the rigidly transformed CT point cloud; The elastic registration module is used to correct the rigidly transformed CT point cloud through elastic registration, and generate a deformed CT point cloud. The 3D reconstruction module is used to generate the 3D surface of the organ from the deformed CT point cloud using a surface generation algorithm, thereby realizing dynamic 3D reconstruction based on static CT images.

7. A computer-readable storage medium, characterized in that, It contains a computer program, which, when executed, implements the dynamic three-dimensional reconstruction method for organ soft tissue as described in any one of claims 1 to 5.

8. An ultrasonic device, characterized in that, include: One or more processors; The processor is used to store one or more programs; When the one or more programs are executed by the one or more processors, the method for dynamic three-dimensional reconstruction of organ soft tissue as described in any one of claims 1 to 5 is implemented.

Images