Guidance method based on 3D camera and CT image and puncture surgical robot system

Abstract

The application relates to the technical field of medical equipment, in particular to a guiding method based on a 3D camera and a CT image and a puncture surgical robot system, which comprises the following steps: S1. preoperative CT three-dimensional reconstruction and point cloud extraction, to obtain a CT point cloud coordinate system; S2. hand-eye calibration of a mechanical arm and a 3D camera; S3. transformation of the CT point cloud coordinate system to a robot reference coordinate system through registration of the CT point cloud and the camera point cloud; and S4. puncture guiding through the robot reference coordinate system, which is based on multi-view point cloud reconstruction technology, does not need special calibration devices to dynamically calibrate the mechanical arm-3D camera pose relationship, and solves the problems of high-precision tools and complex operation in traditional methods.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and more specifically, to a guidance method and a puncture surgery robot system based on 3D cameras and CT images. Background Technology

[0002] In surgical robot systems, hand-eye calibration is a key step in achieving precise robot operation. Its core is to determine the spatial pose relationship between the sensor fixed at the end of the robotic arm and the end effector of the robotic arm. This relationship directly affects the accuracy of converting the spatial information collected by the sensor into the robot coordinate system, and thus determines the precision of surgical positioning and operation.

[0003] Existing hand-eye calibration techniques are mostly based on the classic kinematic loop model AX=XB. While this model can achieve calibration, it is highly dependent on the calibration device. For 3D sensors, most solutions require precise calibration devices of known dimensions, such as standard spheres, precision-machined calibration plates, or freeform surfaces with pre-set CAD models. Coordinate mapping relationships are established by identifying feature points of these devices, and then the hand-eye matrix is ​​solved. These dedicated calibration devices are expensive to manufacture and prone to accuracy degradation due to collisions, wear, and other factors, requiring regular maintenance or replacement, increasing the system's operating costs and maintenance complexity.

[0004] Some existing technologies attempt to circumvent dedicated calibration devices, such as using planar features in the scene for calibration. However, these methods often require additional tools to pre-measure the coordinates of at least three points on the plane. This process not only increases the complexity of the operation and prolongs preoperative preparation time, but may also introduce errors due to manual operation, affecting calibration accuracy. At the same time, such methods have strong limitations on application scenarios. If the surgical environment lacks regular planar structures, calibration cannot be completed, making it difficult to adapt to diverse clinical surgical scenarios.

[0005] In minimally invasive surgical scenarios such as puncture surgery, the reliance on dedicated calibration devices for hand-eye calibration not only increases the overall cost of the system, but also makes it difficult to meet the requirements of surgery for efficiency, precision and flexibility due to problems such as complex operation and poor adaptability to different scenarios. This has become a major bottleneck restricting the clinical popularization and application of surgical robot systems. Summary of the Invention

[0006] The purpose of this invention is to provide a guidance method based on 3D camera and CT images. It is based on multi-view point cloud reconstruction technology and can dynamically calibrate the pose relationship between the robotic arm and the 3D camera without the need for a dedicated calibration device, thus solving the problems of traditional methods that rely on high-precision tools and are complicated to operate.

[0007] Another objective of this invention is to provide a guided puncture surgical robot system based on a 3D camera and CT images. The 3D camera replaces high-end optical / MRI navigation equipment, and the 3D camera is fused with the preoperative CT reconstructed point cloud, which significantly reduces system cost and improves operational efficiency and accuracy.

[0008] The technical solution of the present invention:

[0009] On one hand, the present invention provides a guidance method based on a 3D camera and CT images, comprising the following steps:

[0010] S1. Preoperative CT 3D reconstruction and point cloud extraction to obtain the CT point cloud coordinate system;

[0011] S2. Hand-eye calibration of the robotic arm and 3D camera, including the following steps:

[0012] S2.1. Define four types of coordinate systems as calibration references, including the robot reference coordinate system, flange coordinate system, 3D camera coordinate system, and CT point cloud coordinate system;

[0013] S2.2. Based on the coordinate system, the robotic arm is controlled to move the 3D camera to 9 different poses, and the pose information and point cloud data captured by the 3D camera are recorded simultaneously.

[0014] S2.3. Using pose information and point cloud data, the BO-IA algorithm is used to model the hand-eye relationship in the 3D motion space and search for the globally optimal initial estimate. The AA-ICPv algorithm is used to register the multi-view point cloud, and the corresponding point matching and transformation are iteratively optimized to obtain the hand-eye matrix. The matrix with the smallest average error is selected as the hand-eye matrix. ;

[0015] S3. Transform the CT point cloud coordinate system to the 3D camera coordinate system by registering the CT point cloud with the camera point cloud;

[0016] S4. Through the hand-eye matrix Flange coordinates and pose matrix Transform the 3D camera coordinate system to the robot's reference coordinate system and perform puncture guidance.

[0017] Furthermore, S1 includes the following steps:

[0018] S1.1. Use a multi-energy spectral CT scanner to scan the target and obtain two-dimensional slice data;

[0019] S1.2. Convert the format of the two-dimensional slice data, stack them along the Z-axis, separate the target tissue through grayscale and binarization processing, and extract the target contour pixels;

[0020] S1.3. Combine the coordinates of the contour pixels with the Z-axis index to form an initial point cloud. Separate the external contour through connected component analysis and obtain the CT point cloud coordinate system after downsampling.

[0021] Furthermore, the scanning specification in S1.1 is 373p resolution, and the reconstructed voxel size is 0.5mm.

[0022] Furthermore, the transformation from the 3D camera coordinate system to the robot's reference coordinate system in S4 is obtained by the following formula:

[0023]

[0024]

[0025] In the formula, This serves as the robot's reference coordinate system. 3D camera coordinate system; Hand-eye matrix; For the flange coordinates and pose matrix; This is the transformation matrix for the reference coordinates of the 3D camera and the robot.

[0026] Furthermore, S3 includes the following steps:

[0027] S3.1. Transformation matrix based on 3D camera and robot reference coordinates The point cloud captured by the 3D camera is converted to the robot's reference coordinate system, and the part with Z-axis < 0 is removed to obtain a point cloud model that does not include the optical platform.

[0028] S3.2. The target contour pixels extracted in S1.2 are coarsely registered with the point cloud model without an optical platform by centroid alignment and rotation alignment, providing an initial position reference for fine registration;

[0029] S3.3. Based on the coarse registration results in S3.2, the ICP algorithm is used to determine the minimum registration error using a point-to-surface registration method, and then to achieve fine registration by secondary optimization of surface matching using a face-to-face registration method, resulting in the registration matrix. The final coordinate transformation matrix is ​​calculated, and the CT point cloud is transformed into the robot's reference coordinate system using the final coordinate transformation matrix.

[0030] Furthermore, the final coordinate transformation matrix in S3.3 is obtained by the following formula:

[0031]

[0032]

[0033] In the formula, This serves as the robot's reference coordinate system. The coordinate system is the CT point cloud coordinate system; This is the final coordinate transformation matrix; For the registration matrix; This is the transformation matrix for the reference coordinates of the 3D camera and the robot.

[0034] Furthermore, S4 includes the following steps:

[0035] S4.1. The puncture start and end points are selected through CT point cloud, the robotic arm obtains the corresponding robot reference coordinate system, and the joint angle path is generated through the inverse kinematics calculation of the robotic arm to realize motion planning;

[0036] S4.2. Adjust the actuator axis to be consistent with the puncture path direction, and perform the puncture operation perpendicular to the actuator axis direction.

[0037] On the other hand, the present invention provides a guided puncture surgery robot system based on 3D camera and CT images, including a six-degree-of-freedom robot motion module, a tool module disposed at the end of the six-degree-of-freedom robot motion module, an end effector module disposed at the tool module, a 3D camera module rigidly connected to the end effector module, and a CT module signal-connected to the six-degree-of-freedom robot motion module.

[0038] Furthermore, the six-degree-of-freedom robot motion module is equipped with a real-time data exchange (RTDE) interface for connecting external applications and the UR controller.

[0039] Furthermore, the tool module is equipped with multiple tool placement units, which are used to install the tools required for puncture.

[0040] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:

[0041] 1. By using industrial-grade 3D cameras to replace expensive optical navigation equipment, combined with markerless hand-eye calibration technology, the hardware investment and maintenance costs are significantly reduced, providing a cost-performance advantage for the widespread adoption of the system.

[0042] 2. The system achieves millimeter-level positioning accuracy through precise registration of preoperative CT 3D reconstruction point cloud with intraoperative 3D camera point cloud, markerless hand-eye calibration, and high-precision robot control, meeting the clinical accuracy requirements of puncture surgery. The fully automated process reduces error fluctuations introduced by manual operation.

[0043] 3. The fully automated design process eliminates the tedious steps of marker implantation and manual contour drawing in traditional methods. The 3D camera's second-level point cloud acquisition and optimized registration algorithm shorten the preoperative preparation time, meeting the needs of efficient clinical surgery.

[0044] 4. By removing interfering point clouds and using label-free hand-eye calibration technology to resist featureless scenes, the system can adapt to complex environments. The modular design of the end effector supports a variety of puncture operations, providing a flexible solution for minimally invasive surgery.

[0045] 5. From preoperative CT data to real-time intraoperative point cloud, and then to precise robot operation, each module forms a closed-loop system through coordinate transformation and data communication, ensuring the accurate mapping of medical imaging information, real-time spatial information and robot operating space, and providing reliable protection for surgical safety. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0048] Figure 2 (a) is a schematic diagram of the items used in the calibration process; (b) is a display diagram of the registration results of 9 sets of point clouds after the item combination calibration.

[0049] Figure 3 (a) is a schematic diagram of the point cloud extraction model; (b) is the processed PNG image; (c) is the complete point cloud map of the model; (d) is the external point cloud map of the model.

[0050] Figure 4 (a) is a schematic diagram of point cloud information of the model captured by the robot; (b) is a single point cloud image obtained by the camera.

[0051] Figure 5 (a) shows the point cloud Y after excision; (b) shows the result after ICP registration.

[0052] Figure 6 (a) is a schematic diagram of three high-precision mark points set on the robot base coordinate system and optical platform; (b) is a magnified schematic diagram of the mark point positions and labels; (c) is a schematic diagram of the selected mark points in the point cloud captured by the camera.

[0053] Figure 7 (a) shows a diagram of the measuring tool used to measure the size of the block using a vernier caliper; (b) shows a diagram of measuring the size and orientation of the block using CloudCompare software.

[0054] Figure 8(a) shows the selection of corner points K in point cloud Y; (b) shows the selection of corner points K in point cloud Z.

[0055] Figure 9 (a) shows the selected corner points of the point cloud in the final test; (b) shows the actual corner point positions of the test model; (c) shows the diagram from the tip of the robot's mobile end effector to the actual corner point; (d) shows the error measurement diagram between the tip and the actual corner point.

[0056] Figure 10 (a) is a schematic diagram of the entire point cloud of the model; (b) is a schematic diagram of the set insertion starting point position; (c) is a schematic diagram of the final insertion effect. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Example

[0058] Please refer to Figure 1 Based on the basic theoretical concept of this invention, a guidance method based on 3D camera and CT images is proposed, including the following steps:

[0059] S1. Preoperative CT 3D reconstruction and point cloud extraction to obtain the CT point cloud coordinate system;

[0060] S2. Hand-eye calibration of the robotic arm and 3D camera, including the following steps:

[0061] S2.1. Define four types of coordinate systems as calibration references, including the robot reference coordinate system, flange coordinate system, 3D camera coordinate system, and CT point cloud coordinate system;

[0062] S2.2. Based on the coordinate system, the robotic arm is controlled to move the 3D camera to 9 different poses, and the pose information and point cloud data captured by the 3D camera are recorded simultaneously.

[0063] S2.3. Using pose information and point cloud data, the BO-IA algorithm is used to model the hand-eye relationship in the 3D motion space and search for the globally optimal initial estimate. The AA-ICPv algorithm is used to register the multi-view point cloud, and the corresponding point matching and transformation are iteratively optimized to obtain the hand-eye matrix. The matrix with the smallest average error is selected as the hand-eye matrix. ;

[0064] S3. Transform the CT point cloud coordinate system to the 3D camera coordinate system by registering the CT point cloud with the camera point cloud;

[0065] S4. Through the hand-eye matrix Flange coordinates and pose matrix Transform the 3D camera coordinate system to the robot's reference coordinate system and perform puncture guidance.

[0066] It should be noted that in hand-eye calibration, the 3D camera is regarded as the 'eye' and the end edge of the robotic arm is regarded as the 'hand'. In order to obtain the accurate positional relationship matrix between the camera and the robot, this invention first models the hand-eye relationship in the three-dimensional motion space SE (3) by Bayesian optimized initial alignment (BO-IA), and uses the improved Gaussian process covariance function and expectation improvement (EI) acquisition function to efficiently search for the global optimal initial estimate; then, the Anderson-accelerated ICP variant (AA-ICPv) is used for fine registration, and historical iteration gradient information is introduced to predict the convergence direction, which accelerates the linear convergence of traditional ICP to superlinear convergence, reducing the computation time by more than 40%.

[0067] This method uses alternating matching and transformation optimization of corresponding points to uniformly register multi-view point clouds to the robot base or flange coordinate system, while simultaneously calibrating the hand-eye relationship.

[0068] In this invention, the robot is manipulated to move to nine different positions and its corresponding pose information is recorded. Registration is then performed based on nine sets of point cloud data captured by a point cloud camera at each position to ultimately obtain a hand-eye matrix. For example... Figure 2 (a) and Figure 2 As shown in (b), multiple sets of hand-eye calibrations were performed using different objects during the process. A total of 15 sets of hand-eye matrices were obtained by performing hand-eye calibrations on single objects and combinations of multiple objects. The hand-eye matrices describe the positional relationship between the 3D camera installed at the end of the robot and the center of the end flange of the robot.

[0069] The point cloud images captured under the same robot pose were converted into 15 sets of corresponding hand-eye matrices. A feature point of the object was selected, and the error distance between the real coordinates of the point and the coordinates of the point under the converted point cloud information was compared. This process was repeated multiple times, and the matrix with the smallest error among the 15 sets of hand-eye matrices was selected. Its average error was 1.2mm. The matrix with the smallest error was used as the hand-eye matrix in this embodiment.

[0070] Furthermore, S1 includes the following steps:

[0071] S1.1. Use a multi-energy spectral CT scanner to scan the target and obtain two-dimensional slice data;

[0072] S1.2. Convert the format of the two-dimensional slice data, stack them along the Z-axis, separate the target tissue through grayscale and binarization processing, and extract the target contour pixels;

[0073] S1.3. Combine the coordinates of the contour pixels with the Z-axis index to form an initial point cloud. Separate the external contour through connected component analysis and obtain the CT point cloud coordinate system after downsampling.

[0074] It should be noted that in S1, 3D reconstruction and point cloud extraction are performed by stacking a series of 2D images along the Z-axis. The edge pixels of each image are extracted and assigned Z-axis coordinates, thereby expanding the 2D edge information into a 3D point cloud. This achieves the reconstruction from 2D slices to 3D structures. Finally, the point cloud is optimized to obtain point cloud data that can be used for analysis.

[0075] This embodiment uses the MARS Microlab multi-energy spectral CT scanner to scan the model object, such as Figure 3 As shown in (a), taking the model in the figure as an example, the reconstructed CT image file is in DICOM format and needs to be converted to the more easily editable PNG format, such as... Figure 3 As shown in (b), after grayscale and binarization, the image is converted to black and white. Edge detection is then performed to extract the edge information of each image, which is then mapped to three-dimensional point cloud data. X and Y are pixel coordinates, and Z is the image number. This allows the extraction of the entire point cloud of the object. Figure 3 As shown in (c).

[0076] To obtain the outer contour of an object, the outer contour point cloud is separated by selecting the edge pixels of the largest connected region in the image, such as... Figure 3 As shown in (d), this is used for subsequent registration with the camera point cloud. The generated point cloud model is then downsampled and saved as a PLY file. The accuracy of the extracted point cloud is less than 0.5 mm from that of the real object. The acquired point cloud model effectively digitizes the internal and external information of the target model, thereby aiding the system in identifying and planning the location of lesions.

[0077] Furthermore, the scanning specification in S1.1 is 373p resolution, and the reconstructed voxel size is 0.5mm.

[0078] Furthermore, the transformation from the 3D camera coordinate system to the robot's reference coordinate system in S4 is obtained by the following formula:

[0079]

[0080]

[0081] In the formula, This serves as the robot's reference coordinate system. 3D camera coordinate system; Hand-eye matrix; For the flange coordinates and pose matrix; This is the transformation matrix for the reference coordinates of the 3D camera and the robot.

[0082] It should be noted that this system is based on four coordinate systems. Since the robotic arm remains fixed, the object used for calibration remains constant in the robotic arm's reference coordinate system. The point cloud coordinates obtained by the 3D camera are all based on the camera coordinate system, so the above relationships can be derived, including the flange coordinates and pose matrix. This can be obtained from a robotic arm teaching pendant. During the conversion process, the registration process between the CT-extracted point cloud and the 3D camera-captured point cloud can establish the CT point cloud coordinate system. Transform to 3D camera coordinate system In the middle; the hand-eye calibration process can Transform to flange coordinate system In the middle; by reading the control panel built into the robotic arm, it can... Transform to robot reference coordinate system In this way, the point cloud extracted by CT and the point cloud in the surgical environment can be converted into the reference coordinate system of the robotic arm, so that the robotic arm can be manipulated to perform puncture at the lesion site in the surgical space.

[0083] Furthermore, S3 includes the following steps:

[0084] S3.1. Transformation matrix based on 3D camera and robot reference coordinates The point cloud captured by the 3D camera is converted to the robot's reference coordinate system, and the part with Z-axis < 0 is removed to obtain a point cloud model that does not include the optical platform.

[0085] S3.2. The target contour pixels extracted in S1.2 are coarsely registered with the point cloud model without an optical platform by centroid alignment and rotation alignment, providing an initial position reference for fine registration;

[0086] S3.3. Based on the coarse registration results in S3.2, the ICP algorithm is used to determine the minimum registration error using a point-to-surface registration method, and then to achieve fine registration by secondary optimization of surface matching using a face-to-face registration method, resulting in the registration matrix. The final coordinate transformation matrix is ​​calculated, and the CT point cloud is transformed into the robot's reference coordinate system using the final coordinate transformation matrix.

[0087] It should be noted that, in order to convert the coordinates of a single point cloud image captured by the camera to the robot's reference coordinate system, the robotic arm is first manipulated to photograph the model on the optical plane and record the robot's current pose information, such as... Figure 4 As shown in (a), the captured point cloud is as follows Figure 4 As shown in (b), the point cloud coordinate system can be transformed using the hand-eye matrix obtained above, based on the calculation formula mentioned above.

[0088] By removing the portions of the point cloud with z-axis values ​​less than 0, a point cloud model excluding the optical platform can be obtained, such as... Figure 5 As shown in (a), this is done to reduce interference with subsequent registration with CT point clouds. Because the point clouds reconstructed and extracted in CT are in their corresponding generated coordinate system, and the subsequent fine registration algorithm depends on the initial position, the camera point cloud and the object's external contour point cloud extracted by CT are first coarsely registered, and then the centroids are aligned and rotated for alignment.

[0089] The fine registration process uses the Iterative Closest Point (ICP) algorithm. MATLAB offers three basic ICP registration methods: point-to-point, point-to-surface, and face-to-face. The first step is to find the optimal ICP registration method. This is achieved by trying different methods to find the one that minimizes the registration error (RMSE).

[0090] This embodiment selects point-to-area registration and uses the pcregistericp function for initial registration. Then, fine registration is performed on the initial registration result, specifying face-to-face registration. By calculating and printing the registration error and visualizing the registration result, the two point clouds are finally registered in one coordinate system, as shown in the following figure. Figure 5 As shown in (b), the coordinate system transformation is achieved. The corner coordinate comparison shows a registration error of about 2mm, which is mainly caused by the CT point cloud reconstruction error (1mm) and the camera point cloud noise (±0.15mm).

[0091] Furthermore, the final coordinate transformation matrix in S3.3 is obtained by the following formula:

[0092]

[0093]

[0094] In the formula, This serves as the robot's reference coordinate system. The coordinate system is the CT point cloud coordinate system; This is the final coordinate transformation matrix; For the registration matrix; This is the transformation matrix for the reference coordinates of the 3D camera and the robot.

[0095] Furthermore, S4 includes the following steps:

[0096] S4.1. The puncture start and end points are selected through CT point cloud, the robotic arm obtains the corresponding robot reference coordinate system, and the joint angle path is generated through the inverse kinematics calculation of the robotic arm to realize motion planning;

[0097] S4.2. Adjust the actuator axis to be consistent with the puncture path direction, and perform the puncture operation perpendicular to the actuator axis direction.

[0098] To better understand the present invention, the present invention provides a guided puncture surgical robot system based on 3D camera and CT images, including a six-degree-of-freedom robot motion module, a tool module disposed at the end of the six-degree-of-freedom robot motion module, an end effector module disposed at the tool module, a 3D camera module rigidly connected to the end effector module, and a CT module signal-connected to the six-degree-of-freedom robot motion module.

[0099] It should be noted that the 3D camera uses the Zhixiang Optoelectronics HD50 binocular structured light point cloud industrial-grade camera, with an optimal working distance of 500±250mm, an FOV of H55°* V36°, a point pitch @ working distance (mm) of 0.273@500, and a repeatability of ±0.15mm.

[0100] CT images were acquired using a small animal multi-energy spectral CT scanner (MARS Microlab, New Zealand). The scanning method was helical scanning, with a scan length of 10-300 mm, a nominal voxel size of 90-500 μm, an energy range of 30-120 keV, and a spatial resolution of 50-200 μm. High-resolution CT images could be obtained by adjusting the scanning parameters.

[0101] The six-degree-of-freedom robot (Universal Robots UR5e, Denmark) is the system's motion module, with a maximum load of 5 kg, a maximum working radius of 850 mm, and repeatability of ±0.03 mm.

[0102] The system's required software interfaces and modules were developed using MATLAB and Python, with the graphical interface implemented in MATLAB. This interface enables CT image processing and point cloud extraction, 3D camera point cloud imaging, point cloud registration visualization, target point selection, and robot control.

[0103] Furthermore, the six-degree-of-freedom robot motion module is equipped with a real-time data exchange (RTDE) interface for connecting external applications and the UR controller.

[0104] It should be noted that the UR5e robotic arm's Real-Time Data Exchange (RTDE) interface provides a method for synchronizing external applications with the UR controller via a standard TCP / IP connection without disrupting any of the UR controller's real-time properties. This yields the following advantages for robot motion control:

[0105] Real-time synchronization: RTDE typically generates output messages at 125 Hz; Input information: Updates to variables in the controller can be broken down into multiple messages. A constant update rate is not required; inputs retain their last received values. Runtime environment: The RTDE client can run on a UR Control Box PC or any external PC.

[0106] Furthermore, the tool module is equipped with multiple tool placement units, which are used to install the tools required for puncture.

[0107] It should be noted that the tools in this embodiment are a miniature drill bit and a suction pipe, used for drilling holes and suctioning liquid from the model.

[0108] To better understand this invention, five tests are provided below to verify the feasibility of the system. It should be noted that the error measurement test sample is a prism made of silicone material with a circular hole with a diameter of 1.1 mm and a depth of 0.7 mm on the upper plane. The fluid extraction test sample is also made of silicone material, with a protruding wooden block simulating hard tissue. A cylindrical hole with a diameter of 1.1 mm and a depth of 35 mm is set as an insertion channel, and a ping-pong ball with the same opening is attached to the bottom to simulate a fluid accumulation mass.

[0109] Test 1

[0110] Test 1 is an error experiment for hand-eye calibration and point cloud calculation, aiming to obtain the accuracy of the initial point cloud conversion after hand-eye calibration. Given the precise dimensions of the screw holes on the optical platform, three Mark points with known accurate coordinates are fixed on the optical plane, such as... Figure 6 As shown in (a) and (b), the robot's base coordinate system and the positions of three Mark points are labeled. The robotic arm moves to five different poses, and a frame of point cloud data is captured at each pose using a 3D camera. This data is then converted to the actual robot reference coordinate system, and the coordinates of the corresponding three points in the converted point cloud are selected using the CloudCompare software, as shown below. Figure 6 As shown in (c). This allows us to measure the error between the coordinates of the three points in the point cloud and the actual coordinates. The test errors are shown in Table 1 below. Ultimately, the errors in both hand-eye calibration and point cloud calculation are less than 3mm.

[0111]

[0112] Test 2

[0113] Test 2 is an error experiment on the point cloud model converted from CT images. To measure the error between the point cloud model converted from CT images and the actual object, a self-made testing tool was used, such as... Figure 7As shown in (a), the tool consists of four cut marble blocks, which are glued to the same cardboard to fix their positions. The horizontal and vertical lengths of each block are measured with calipers, as shown. Figure 7 As shown in (a), in this measurement, the horizontal direction is recorded as x-axis and the vertical direction as y-axis. The data is scanned using CT, and the size of each square in the point cloud is measured and recorded using the line segment tool in CloudCompare, as shown below. Figure 7 As shown in Figure (b), each stone in the point cloud is labeled with its measurement direction and side length. The final results are shown in Table 2, with an error of approximately 1 mm compared to the actual dimensions.

[0114]

[0115] Test 3

[0116] Test 3 is an experiment on the relative error after registration of CT point clouds and camera point clouds. Based on the above test, the point cloud model captured by the camera and transformed to the coordinate system is cropped and denoted as point cloud Y, as shown. Figure 8 As shown in (a), the entire point cloud after reconstruction from the CT image, extraction, and coordinate transformation is denoted as point cloud Z, as follows: Figure 8 As shown in (b).

[0117] To measure the error estimation of this process, select the same corner point K of the point cloud Y and Z axes in CloudCompare. Figure 8 The coordinates of the two selected corner points are marked and compared. The robotic arm acquires 3D camera point clouds in a certain pose and registers them with the original CT point clouds. After registration, the coordinates of point K in the two sets of point clouds are recorded as one set of data. The robotic arm repeats the experiment 5 times in 5 different poses, and the results are shown in Table 3. The coordinate error distance of the same corner point in the two point clouds is measured to be approximately 2 mm.

[0118]

[0119] Test 4

[0120] Test 4 is a comprehensive error experiment of the entire system. The accuracy of this system is mainly affected by the registration accuracy of the two point clouds in the system and the accuracy of hand-eye calibration. To measure the accuracy of the final surgical actions performed by the system, we selected a corner point from the overall point cloud that has been transformed into the robot's reference coordinate system, such as... Figure 9 As shown in (a), Figure 9 (b) shows the actual location of the corner point. Based on the coordinates of the selected point, the robot controls the tip of the end effector to move to this corner point, as shown in Figure (b). Figure 9 As shown in (c). The overall system error is represented by the error in the robot's movement to the actual corner point. The measurement method is as follows: Figure 9In the middle (d), the measurement results are shown in Table 4. Three different corner points were selected, and the robotic arm moved to the corner points five times. The measurement error was about 3 mm.

[0121]

[0122] Test 5

[0123] Based on the above tests, this test 5 is a fully automated robot extraction experiment to verify the overall system's stability and feasibility throughout the entire process.

[0124] A simulation experiment of fluid extraction was conducted using model A. A wooden rod was inserted into the tool assembly at the end of the robot and fixed in place. The wooden rod was pre-set to be inserted into the hole in model A.

[0125] The overall point cloud model A extracted from the CT image, such as Figure 10 As shown in (a), the predetermined insertion starting point is selected as 10mm directly above the approximate center of the cavity, as follows. Figure 10 As shown in (b).

[0126] The insertion endpoint is 10mm above the bottom of the ping-pong ball. Ensure the robot inserts the wooden stick along the two points without touching model A throughout the process. The final insertion result should look like this. Figure 10 As shown in (c), this demonstrates that the system can autonomously complete the insertion action.

[0127] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A guidance method based on 3D camera and CT images, characterized in that, Includes the following steps: S1. Preoperative CT 3D reconstruction and point cloud extraction to obtain the CT point cloud coordinate system, including the following steps: S1.

1. Use a multi-energy spectral CT scanner to scan the target and obtain two-dimensional slice data; S1.

2. Convert the format of the two-dimensional slice data, stack them along the Z-axis, separate the target tissue through grayscale and binarization processing, and extract the target contour pixels; S1.

3. Combine the coordinates of the contour pixels with the Z-axis index to form an initial point cloud. Separate the external contour through connected component analysis and obtain the CT point cloud coordinate system after downsampling. S2. Hand-eye calibration of the robotic arm and 3D camera, including the following steps: S2.

1. Define four types of coordinate systems as calibration references, including the robot reference coordinate system, flange coordinate system, 3D camera coordinate system, and CT point cloud coordinate system; S2.

2. Based on the coordinate system, manipulate the robotic arm to move the 3D camera to 9 different poses, and simultaneously record the pose information and point cloud data captured by the 3D camera; S2.

3. Using the pose information and the point cloud data, the hand-eye relationship is modeled in the three-dimensional motion space using the BO-IA algorithm, and the globally optimal initial estimate is searched. The multi-view point cloud is registered using the AA-ICPv algorithm, and the corresponding point matching and transformation are iteratively optimized to obtain the hand-eye matrix. The matrix with the smallest average error is selected as the hand-eye matrix. ; S3. Transform the CT point cloud coordinate system to the 3D camera coordinate system through registration of the CT point cloud and the camera point cloud, including the following steps: S3.

1. Transformation matrix based on the 3D camera and robot reference coordinates The point cloud captured by the 3D camera is converted to the robot's reference coordinate system, and the part with Z-axis < 0 is removed to obtain a point cloud model that does not include the optical platform. S3.

2. The target contour pixels extracted in S1.2 are coarsely registered with the point cloud model without an optical platform by centroid alignment and rotation alignment, providing an initial position reference for fine registration; S3.

3. Based on the coarse registration results in S3.2, the ICP algorithm is used to determine the minimum registration error using a point-to-surface registration method, and then to achieve fine registration by secondary optimization of surface matching using a face-to-face registration method, resulting in the registration matrix. The final coordinate transformation matrix is ​​calculated, and the CT point cloud is transformed into the robot's reference coordinate system using the final coordinate transformation matrix; S4. Through the hand-eye matrix Flange coordinates and pose matrix The 3D camera coordinate system is transformed to the robot's reference coordinate system for puncture guidance; the transformation from the 3D camera coordinate system to the robot's reference coordinate system is obtained by the following formula: ； ； In the formula, This serves as the robot's reference coordinate system. 3D camera coordinate system; Hand-eye matrix; For the flange coordinates and pose matrix; This is the transformation matrix for the reference coordinates of the 3D camera and the robot.

2. The method according to claim 1, characterized in that, The scanning specification in S1.1 is 373p resolution, and the reconstructed voxel size is 0.5mm.

3. The method according to claim 1, characterized in that, The final coordinate transformation matrix in S3.3 is obtained by the following formula: ； ； In the formula, This serves as the robot's reference coordinate system. The coordinate system is the CT point cloud coordinate system; This is the final coordinate transformation matrix; For the registration matrix; This is the transformation matrix for the reference coordinates of the 3D camera and the robot.

4. The method according to claim 1, characterized in that, S4 includes the following steps: S4.

1. The puncture start and end points are selected through CT point cloud, the robotic arm obtains the corresponding robot reference coordinate system, and the joint angle path is generated through the inverse kinematics calculation of the robotic arm to realize motion planning; S4.

2. Adjust the actuator axis to be consistent with the puncture path direction, and perform the puncture operation perpendicular to the actuator axis direction.

5. A guided puncture surgical robot system based on the 3D camera and CT image-based guidance method according to any one of claims 1-4, characterized in that, It includes a six-degree-of-freedom robot motion module, a tool module disposed at the end of the six-degree-of-freedom robot motion module, an end effector module disposed at the tool module, a 3D camera module rigidly connected to the end effector module, and a CT module that is signal-connected to the six-degree-of-freedom robot motion module.

6. The system according to claim 5, characterized in that, The six-degree-of-freedom robot motion module is equipped with a real-time data exchange (RTDE) interface for connecting external applications and the UR controller.

7. The system according to claim 5, characterized in that, The tool module is provided with multiple tool placement units, which are used to install the tools required for puncture.

Images