A calibration method for a spinal endoscopic surgery robot

Abstract

The present invention relates to a spinal endoscopic surgical robot calibration method, comprising the following steps: using a calibration device to calibrate a binocular camera, determining the internal parameters, external parameters and distortion parameters of the binocular camera; connecting a tracking device to a rigid body, the rigid body including the patient's spine, the robot end and the spinal endoscope, using a binocular camera to track the tracking device and obtain the rigid body posture change information; performing image fusion and registration on the preoperative CT image and the spinal endoscopic image, obtaining the corresponding relationship between the lesion part image and the actual physical lesion area; locating the local lesion target in the surgical area according to the rigid body posture information and image fusion registration data obtained by the binocular camera to complete the calibration. It can effectively realize hand-eye calibration and rigid body tracking, improve the automation level of the spinal endoscopic surgical robot, effectively improve the accuracy and stability of the operation, reduce the intraoperative risk and the occurrence of postoperative complications, and greatly reduce the radiation damage to medical staff caused by CT fluoroscopic guidance.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical robots, and in particular to a spinal endoscopic surgical robot calibration method. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Compared with traditional open surgery, minimally invasive endoscopic spinal surgery has the characteristics of less trauma, faster postoperative recovery, and reliable surgical results. Currently, most of the minimally invasive endoscopic spinal surgeries are performed directly by doctors or partially by using spinal endoscopic surgical robots.

[0004] Before performing an operation, the spinal endoscopic surgical robot needs to be calibrated to help the robot obtain the coordinate system reference position of the end effector. The existing technology relies on optical instruments and auxiliary facilities to achieve calibration, which has problems such as complex operation, low surgical efficiency, low precision, and poor stability. In addition, multiple CT scans are required during the operation to obtain the accurate position of the spine, which generates a large amount of radiation for patients and medical staff. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a spinal endoscopic surgical robot calibration method, which can effectively realize hand-eye calibration and rigid body tracking, improve the degree of automation of the spinal endoscopic surgical robot, effectively improve the accuracy and stability of the operation, reduce intraoperative risks and the occurrence of postoperative complications, and greatly reduce the radiation damage to medical staff caused by CT fluoroscopy guidance.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention provides a spinal endoscopic surgery robot calibration method, comprising the following steps:

[0008] Step 1: Use the calibration device to calibrate the binocular camera to determine the internal parameters, external parameters and distortion parameters of the binocular camera;

[0009] Step 2: The tracking device is connected to the rigid body, which includes the patient's spine, the end of the robot, the spinal endoscope, the flange connecting the robot and the spinal endoscope, the tooling, fixture or surgical instrument connecting the robot and the spinal endoscope, and the binocular camera is used to track the tracking device and obtain the position change information of the rigid body;

[0010] Step 3: Perform image fusion and registration of the preoperative CT image and the spinal endoscopy image to obtain the correspondence between the image of the lesion site and the actual physical lesion area;

[0011] Step 4: Based on the rigid body posture information obtained by the binocular camera and the image fusion registration data in step 3, the local lesion target in the surgical area is located to complete the calibration.

[0012] In step 1, the two cameras of the binocular camera send infrared light to the calibration device respectively, and the left and right cameras receive the infrared light reflected by the calibration device to obtain a binary image, and obtain the internal parameters, external parameters and distortion parameters of the binocular camera.

[0013] In step 1, the calibration device is four small balls fixed together, located in the same plane and not collinear, each of which has a retroreflective coating to reflect the infrared light emitted by the binocular camera back to the binocular camera to obtain a binary image.

[0014] In step 2, the tracking device is four small balls fixed together, located in the same plane and not collinear, each of which has a retroreflective coating and is connected to the rigid body, reflecting the infrared light emitted by the binocular camera back to the binocular camera to obtain a binary image of the rigid body.

[0015] The diameter of each ball in the tracking device and the calibration device is the same or different.

[0016] In step 2, three-dimensional reconstruction is performed based on the binary image obtained by the binocular camera and the tracking device is calibrated to obtain the position and posture information of the tracking device, and the position and posture of the rigid body are obtained based on the position and posture information of the tracking device.

[0017] In step 2, in the process of tracking the changes in the rigid body posture, a tracking device is connected to the bone tissue in the patient's surgical area and the end of the spinal endoscope, and the tracking device is fixed on the bone tissue and the spinal endoscope respectively to obtain two-dimensional image information of the position and posture of the spinal endoscope in the patient's body during the operation.

[0018] In step 3, the tomographic images of the patient's organs and target tissues are reconstructed based on the patient's preoperative CT images to obtain a three-dimensional visualization image model; the positional correspondence between the three-dimensional visualization image model and the two-dimensional image information is obtained based on the spinal endoscopy image to achieve image fusion and registration.

[0019] In steps 1 and 2, based on the internal and external parameters of the binocular camera, the spatial transformation relationship between the camera coordinate system and the pixel coordinate system, and the spatial transformation relationship between the world coordinate system and the camera coordinate system are obtained, and the spatial transformation relationship between the world coordinate system and the pixel coordinate system is obtained, and the position information of the rigid body in the image in the actual physical space is obtained.

[0020] In step 4, the local lesion target data of the surgical area is obtained based on the image fusion and registration data obtained in step 3. The spatial transformation relationship matrix between the world coordinate system and the pixel coordinate system obtained in steps 1-2 is used to inversely solve the local lesion target position information of the surgical area. The known patient spine part and the fixed coupling relationship matrix of the tracking device are combined to obtain the required position state of the patient spine part tracking device.

[0021] Compared with the prior art, one or more of the above technical solutions have the following beneficial effects:

[0022] 1. A binocular camera optical positioning system is used for tracking and positioning, and optical feedback is used to achieve non-contact positioning. The binocular camera receives the optical information reflected by the reflective ball, and the system calculates the position and posture based on the feedback optical information. It is simple and easy to use, and can effectively realize hand-eye calibration and rigid body tracking, thereby improving the automation and accuracy of the spinal endoscopic surgical robot.

[0023] 2. It overcomes the problems of complex and unstable operation in the traditional calibration process, can effectively improve the accuracy and stability of the operation, and reduce the intraoperative risks and postoperative complications.

[0024] 3. Utilize preoperative CT images and binocular cameras to guide endoscopic tracking and positioning through visual images, which greatly reduces the number of intraoperative CT scans, reduces the amount of radiation exposure to patients and medical staff during surgery, and can greatly reduce the radiation damage to medical staff caused by CT fluoroscopic guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0026] Figure 1 is a schematic diagram of a spinal endoscopic surgery robot system calibration process provided by one or more embodiments of the present invention;

[0027] Figure 2 is a schematic diagram of the principle of tracking the posture change of a rigid body with a binocular camera provided by one or more embodiments of the present invention;

[0028] Figure 3 It is a schematic diagram of the structure of a calibration device or a tracking device provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0031] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0032] As described in the background technology, the spinal endoscopic surgical robot needs to be calibrated before performing surgery to help the robot obtain the coordinate system reference position of the end effector. The existing technology relies on optical instruments and auxiliary facilities to achieve calibration, which has problems such as complex operation, low surgical efficiency, low precision, and poor stability. In addition, multiple CT scans are required during the operation to obtain the accurate position of the spine, which generates a large amount of radiation for patients and medical staff.

[0033] Therefore, the following embodiment provides a spinal endoscopic surgical robot calibration method, which can effectively realize hand-eye calibration and rigid body tracking, improve the degree of automation of the spinal endoscopic surgical robot, effectively improve the accuracy and stability of the operation, reduce intraoperative risks and postoperative complications, and greatly reduce the radiation damage to medical staff caused by CT fluoroscopy guidance.

[0034] Embodiment 1:

[0035] like Figure 1-3 As shown, a spinal endoscopic surgery robot calibration method includes the following steps:

[0036] Step 1: Use a calibration device to calibrate the binocular camera to determine the internal parameters, external parameters, distortion parameters and other information of the binocular camera;

[0037] In the process of binocular camera calibration, a passive binocular camera is used. The left and right cameras send infrared light to the calibration device respectively. The left and right cameras receive the infrared light reflected by the calibration device and take pictures to obtain binary images. After solving, the internal parameters, external parameters, distortion parameters and other information of the binocular camera can be obtained.

[0038] During the binocular camera calibration process, the determined internal parameters of the binocular camera are used to transform the coordinates from the camera coordinate system to the pixel coordinate system to solve the spatial transformation relationship between the camera coordinate system and the pixel coordinate system.

[0039] During the binocular camera calibration process, the determined binocular camera external parameters are used to transform the coordinates from the world coordinate system to the camera coordinate system to solve the spatial transformation relationship between the world coordinate system and the camera coordinate system.

[0040] During the binocular camera calibration process, the determined binocular camera distortion parameters are used to obtain the image distortion introduced by the manufacturing accuracy and assembly process deviation and the degree of distortion of the original image, so as to make corresponding compensation and obtain high-precision pose information of the rigid body.

[0041] In the process of binocular camera calibration, the spatial transformation relationship between the world coordinate system and the pixel coordinate system can be obtained by combining the spatial transformation relationship between the camera coordinate system and the pixel coordinate system, and the spatial transformation relationship between the world coordinate system and the camera coordinate system, so as to further obtain the pose information of the rigid body in the image in the actual physical space. In addition, by introducing distortion parameters, the pose information can be compensated to improve accuracy.

[0042] During the binocular camera calibration process, the calibration device used is four reflective balls that are fixed together, located in the same plane and not collinear. The balls have a retroreflective coating, which can achieve reliable tracking.

[0043] In this embodiment, the structure of the calibration device is as follows: Figure 3 As shown, it includes four reflective balls that are fixed together, located in the same plane and not collinear. The relative position relationship of the four balls is not restricted. The direction in which the binocular camera emits the light source is forward. The balls are passive reflective balls with a retroreflective coating on the surface. The direction opposite to the light source emission direction is backward, that is, the reflected light is emitted in the direction of the binocular camera and is received by the binocular camera.

[0044] Step 2: The tracking device is fixedly connected to a rigid body, which includes the patient's spine, the end of the robot, the spinal endoscope, the flange connecting the robot and the spinal endoscope, the tooling or fixture connecting the robot and the spinal endoscope, and surgical instruments. The binocular camera tracks the tracking device in real time to obtain the position change information of the rigid body.

[0045] In the process of tracking the changes in the posture of the rigid body, the binocular camera is used to perform three-dimensional reconstruction based on the acquired binary image, and then the tracking device is calibrated to calculate the posture information of the tracking device. The tracking device is fixedly connected to the rigid body, and the posture of the rigid body can be obtained in real time based on the posture of the tracking device.

[0046] In the process of tracking the change of the rigid body posture, the spatial transformation relationship and distortion compensation relationship between the world coordinate system and the pixel coordinate system can be obtained through the internal parameters, external parameters and distortion parameters of the binocular camera obtained in step 1. The precise posture relationship of the tracking device in the image is obtained through spatial changes, that is, the tracking device is calibrated.

[0047] In the process of tracking the changes in the posture of a rigid body, the tracking device used is four reflective balls that are fixed together, located in the same plane and not collinear. The balls have a retroreflective coating, which can achieve reliable tracking.

[0048] In this embodiment, the tracking device and the calibration device are both rigid bodies with four reflective balls. The structures can be the same or different, and the sizes and dimensions can be the same or different, as long as they are located in the same plane and are not collinear.

[0049] In the process of tracking the changes in the rigid body posture, a tracking device is installed on the bone tissue in the patient's surgical area and the end of the spinal endoscope. The tracking device is fixed on the bone tissue and the spinal endoscope respectively to obtain real-time two-dimensional video images of the position and posture of the spinal endoscope in the patient's body during the operation.

[0050] Step 3: Perform image fusion and registration of the preoperative CT image and the real-time image of the spinal endoscope to obtain the correspondence between the image of the lesion site and the actual physical lesion area;

[0051] During the image fusion and registration process, the tomographic images of the patient's organs and target tissues are reconstructed based on the patient's preoperative CT images to obtain a three-dimensional visualization image model. At the same time, combined with the real-time endoscopic image, the positional correspondence between the three-dimensional image and the two-dimensional video image is established to achieve image fusion and registration.

[0052] In this embodiment, the 3D reconstruction of CT images goes through three stages: denoising of CT raw data based on Gaussian filtering, reconstruction of intermediate slice images based on inverse distance weighted interpolation, and 3D reconstruction of CT based on ray casting.

[0053] Based on Gaussian filtering, CT raw data denoising takes into account the influence of neighborhood pixel distance on weight and adopts distance weighting function to assign weight, namely:

[0054]

[0055] In which, it is assumed that the original image v(i,j) is an N×N matrix, M represents the set of pixel coordinates of the CT image, n represents the total number of coordinate points in M, u(i,j) represents the weighted value and its cumulative sum is 1, and f(x,y) is the image after filtering and denoising.

[0056] The intermediate slice image reconstruction based on the inverse distance weighted interpolation method is used to reconstruct the intermediate pixel points of the obtained CT image slice, so as to obtain a smooth reconstructed surface and improve the three-dimensional reconstruction effect and accuracy. Among them, the opacity at the adjacent data points and sampling points can be expressed as:

[0057]

[0058] And satisfy:

[0059]

[0060] Among them, M i (x i ,y i ,z i ) represents other neighboring data points in the voxel cube grid where the sampling point M(x, y, z) is located, f(M i ) is the color value of the neighboring data points and sampling points, μ i is the weight of the distance from each neighboring point to the sampling point, l i is the Euclidean distance from the neighboring point to the sampling point, i=1,2,...,n.

[0061] CT 3D reconstruction based on the ray casting method is to synthesize and calculate the opacity and color values ​​of all sampling points on the ray to obtain the final color of the corresponding pixel point, which can be expressed as:

[0062]

[0063] Among them, S i 、T i Respectively represent the color and opacity before the incident sampling point, S o 、T o Respectively represent the color and opacity after the projected light passes through the sampling point, S n 、T n Respectively represent the chromaticity and opacity of the current sampling point.

[0064] In the process of image fusion and registration, an iterative nearest point algorithm based on point cloud data is used for fusion and registration, that is, the three-dimensional point cloud reconstructed from the preoperative CT image and the surface point cloud obtained from the intraoperative scan are continuously moved, and the sum of the least squares distances between points is used as the measurement standard to ultimately obtain the best overlapping effect, thereby obtaining the coordinate transformation relationship of the above point cloud data.

[0065] Assume that the registration point cloud is U = {u i ∈R 3 ,i=1,2,...,m} and V={v i ∈R 3 ,i=1,2,...,n}, any point u in U i and a point v in V i The closest point pairs are formed, and the sum of the Euclidean distances of the closest points in the registered point cloud dataset can be expressed as:

[0066]

[0067] Where R is the rotation matrix and T is the translation vector. Through continuous iteration, R and T are solved to minimize D(R,T).

[0068] Step 4: Comprehensively analyze the rigid body posture information obtained by the binocular camera and the image fusion registration data to locate the local lesion target in the surgical area.

[0069] Based on the image fusion and registration data obtained in step 3, the local lesion target data in the surgical area can be obtained. The spatial transformation relationship matrix between the world coordinate system and the pixel coordinate system obtained in step 1 can be reversely solved to obtain the local lesion target posture information in the surgical area. By further combining the known patient spine part and the fixed coupling relationship matrix of the tracking device, the desired posture state of the patient spine part tracking device can be obtained.

[0070] According to the position information of the tracking device fixed to the rigid body during the operation obtained by the binocular camera in step 2, the rigid body includes the patient's spine, the end of the robot, the spinal endoscope, the connection flange between the robot and the spinal endoscope, the tooling or fixture connecting the robot and the spinal endoscope, surgical instruments, etc., which is continuously compared with the obtained expected value, and the minimum value is obtained based on the principle of least squares method, that is, it is considered that the spinal endoscope surgical robot has positioned the target part of the local lesion in the surgical area during the operation.

[0071] A binocular camera optical positioning system is used for tracking and positioning, and optical feedback is used to achieve non-contact positioning. The binocular camera receives the optical information reflected by the reflective ball, and the system calculates the position and posture based on the feedback optical information. It is simple and easy to use, and can effectively realize hand-eye calibration and rigid body tracking, thereby improving the automation level of the spinal endoscopic surgical robot. It overcomes the problems of complex and unstable traditional operations, can effectively improve the accuracy and stability of the operation, and reduce the risk during surgery and the occurrence of postoperative complications.

[0072] Utilizing preoperative CT images and binocular cameras to guide endoscopic tracking and positioning through visual images can greatly reduce the number of intraoperative CT scans, reduce the amount of radiation exposure to patients and medical staff during surgery, and greatly reduce the radiation damage to medical staff caused by CT fluoroscopic guidance.

[0073] The principle of binocular camera tracking rigid body posture changes is as follows Figure 2 As shown:

[0074] Step 1: Use a passive binocular camera. The left and right cameras send infrared light to the calibration device respectively. The left and right cameras receive the infrared light reflected by the calibration device and take pictures to obtain binary images. After solving, the internal parameters, external parameters, distortion parameters and other information of the binocular camera can be obtained;

[0075] Step 2: Perform three-dimensional reconstruction of the binocular camera based on the acquired binary image;

[0076] Step 3: Calibrate the tracking device;

[0077] Step 4: Calculate the position and posture information of the tracking device;

[0078] Step 5: Fix the tracking device to the rigid body;

[0079] Step 6: Calculate the position and posture of the tracking device in real time to obtain the position and posture of the rigid body.

[0080] Among them, a tracking device is installed on the bone tissue in the patient's surgical area and the end of the spinal endoscope. The tracking device is fixed to the bone tissue and the spinal endoscope respectively by bolts, so as to obtain real-time two-dimensional video images of the position and posture of the spinal endoscope in the patient's body during the operation.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A spinal endoscopic surgery robot calibration system, characterized in that: Includes the following modules: Parameter determination module: calibrate the binocular camera using a calibration device to determine the internal parameters, external parameters and distortion parameters of the binocular camera; Information acquisition module: The tracking device is connected to a rigid body, which includes the patient's spine, the robot end, a spinal endoscope, a flange connecting the robot and the spinal endoscope, a tooling, a fixture or a surgical instrument connecting the robot and the spinal endoscope, and a binocular camera is used to track the tracking device and obtain information on the position change of the rigid body; In the information acquisition module, the tracking device is four small balls that are fixed together and located in the same plane and not collinear. Each small ball has a retroreflective coating and is connected to the rigid body, and reflects the infrared light emitted by the binocular camera back to the binocular camera to obtain a binary image of the rigid body. Fusion and registration module: Fusion and registration of preoperative CT images and spinal endoscopy images to obtain the correspondence between the images of the lesion site and the actual physical lesion area; Target positioning module: Based on the rigid body posture information obtained by the binocular camera and the image fusion and registration data in the fusion registration, the local lesion target in the surgical area is located and calibrated; In the parameter determination module, the calibration device is four small balls fixed together and located in the same plane and not collinear, each small ball has a retroreflective coating, which reflects the infrared light emitted by the binocular camera back to the binocular camera to obtain a binary image; In the fusion registration module, the tomographic images of the patient's organs and target tissues are reconstructed based on the patient's preoperative CT images to obtain a three-dimensional visualization image model; the position correspondence between the three-dimensional visualization image model and the two-dimensional image information is obtained based on the spinal endoscopy image to achieve image fusion and registration; The three-dimensional reconstruction of preoperative CT images goes through three stages: denoising of CT raw data based on Gaussian filtering, reconstruction of intermediate slice images based on inverse distance weighted interpolation, and CT three-dimensional reconstruction drawing based on ray casting method. In the information acquisition module, in the process of tracking the changes in the rigid body posture, a tracking device is connected to the bone tissue in the patient's surgical area and the end of the spinal endoscope, and the tracking device is fixed on the bone tissue and the spinal endoscope respectively to obtain two-dimensional image information of the position and posture of the spinal endoscope in the patient's body during the operation.

2. A spinal endoscopic surgery robot calibration system as claimed in claim 1, characterized in that: In the parameter determination module, the two cameras of the binocular camera respectively send infrared light to the calibration device, and the left and right cameras receive the infrared light reflected by the calibration device to obtain a binary image, thereby obtaining the internal parameters, external parameters and distortion parameters of the binocular camera.

3. The spinal endoscopic surgery robot calibration system according to claim 1, characterized in that: The diameter of each ball in the tracking device and the calibration device is the same or different.

4. The spinal endoscopic surgery robot calibration system according to claim 1, characterized in that: In the information acquisition module, three-dimensional reconstruction is performed based on the binary image acquired by the binocular camera and the tracking device is calibrated to obtain the position and posture information of the tracking device, and the position and posture of the rigid body is obtained based on the position and posture information of the tracking device.

5. The spinal endoscopic surgery robot calibration system according to claim 1, characterized in that: In the parameter determination module and the information acquisition module, based on the internal and external parameters of the binocular camera, the spatial transformation relationship between the camera coordinate system and the pixel coordinate system, and the spatial transformation relationship between the world coordinate system and the camera coordinate system are obtained, the spatial transformation relationship between the world coordinate system and the pixel coordinate system is obtained, and the position information of the rigid body in the image in the actual physical space is obtained.

6. The spinal endoscopic surgery robot calibration system according to claim 1, characterized in that: In the target positioning module, the local lesion target data of the surgical area is obtained based on the image fusion and registration data obtained by the fusion and registration module. The spatial transformation relationship matrix between the world coordinate system and the pixel coordinate system obtained by the parameter determination module and the information acquisition module is inversely solved to obtain the local lesion target posture information of the surgical area. The known patient spine part and the fixed coupling relationship matrix of the tracking device are combined to obtain the required posture state of the patient spine part tracking device.

Images