Surgical robot integrated with a vision module navigation positioning system and method

Abstract

The application relates to a navigation positioning system and method integrating a surgical robot and a vision module, a connecting rod assembly is arranged to connect a binocular camera and a base of a mechanical arm, the position of the binocular camera can be flexibly changed, the risk of losing target tracking due to shielding in a surgical process can be effectively avoided, and an encoder is arranged on a hinge point of the connecting rod assembly to assist in calculating the relative position of a free end of the mechanical arm and the binocular camera at the end of the connecting rod assembly, after the position of the binocular camera is changed, the external parameter matrix of the current position of the camera can be accurately obtained without camera calibration again, the relative position and posture relationship of the binocular camera and the mechanical arm assembly can be quickly updated, and the working efficiency of the surgical robot is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of surgical robots, in particular to a navigation positioning system and method integrating a surgical robot and a vision module. BACKGROUND

[0002] The surgical robot navigation positioning system is an advanced technology for assisting surgical operations, aiming to improve the accuracy and precision of operations. The system combines robot technology, image processing and positioning technology to provide precise positioning and navigation guidance for doctors during operations.

[0003] The surgical robot navigation positioning system also uses positioning technology to achieve precise navigation. During the operation, the system uses tracking sensors or markers to obtain the relative position relationship between the robot arm and the patient's body surface and internal anatomical structure. By collecting and processing position data in real time, the system can calculate the pose and accurate positioning between the robot arm and the surgical target, enabling the doctor to accurately perform surgical operations.

[0004] However, in the existing surgical navigation positioning system, in order to facilitate position calibration, the observation camera is generally fixed, and once the camera is blocked or the position and attitude information of the tracked target is lost, the position of the camera cannot be changed and the information of the tracked target cannot be updated in time, which affects the work of the surgical robot. SUMMARY

[0005] To solve the above problems, the present application proposes a more optimized navigation positioning system and method integrating a surgical robot and a vision module.

[0006] The present application adopts the following technical solutions:

[0007] The present application proposes a navigation positioning system integrating a surgical robot and a vision module, comprising a binocular camera, a mechanical arm assembly, a connecting rod assembly, a calibration board and a plane target, the mechanical arm assembly comprising a fixedly arranged base and a controllably movable mechanical arm, one end of the connecting rod assembly being rotatably connected to the base of the mechanical arm assembly, the other end of the connecting rod assembly being rotatably connected with the binocular camera, the mechanical arm having at least an active telescopic stroke relatively close to or away from the base of the mechanical arm assembly, the calibration board being installed at the free end of the mechanical arm assembly, and the plane target being fixedly arranged in front of the binocular camera.

[0008] The connecting rod assembly comprises at least N connecting rods which are sequentially hingedly connected, N+1 hinge points are formed at both ends of the connecting rod assembly and between each two adjacent connecting rods, N is greater than or equal to 2, a rotation damping structure is arranged at each hinge point, and an encoder is installed at each hinge point, the encoding information of the encoder is used to calculate the relative position of the two ends of the connecting rod assembly when the posture of the connecting rod assembly changes, and the relative position of the binocular camera at the free end and the end of the connecting rod assembly is calculated by means of the relative position of the two ends of the connecting rod assembly.

[0009] The application further provides a navigation positioning method of a surgical robot and a vision module, which is used for calculating the relative position of the binocular camera at the free end of the mechanical arm assembly and the end of the connecting rod assembly in the navigation positioning system of the surgical robot and the vision module.

[0010] Step S1, a rotation matrix T=T1×T2×···×T n to calculate the relative position of the two ends of the connecting rod assembly, T n represents a rotation matrix constructed according to the position and posture of the next hinge point relative to the previous hinge point, the rotation matrix T n The calculation formula of the rotation matrix T

[0011]

[0012] wherein a is the distance between the previous hinge point and the current hinge point along the X direction, d is the distance between the previous hinge point and the current hinge point along the Z direction, α is the included angle between the Z axis of the previous hinge point and the current hinge point along the X direction, and θ is the rotation angle of the current hinge point around the Z axis. The parameter θ is obtained by real-time calculation of the encoder, and the other three parameters are all certain parameter values in the design of the connecting rod assembly, and the X direction and the Z direction are the horizontal direction and the vertical direction respectively.

[0013] Step S2, calculating the relative position and posture relationship of the binocular camera at the free end of the mechanical arm assembly and the end of the connecting rod assembly, comprising:

[0014] S21, the binocular camera shoots pictures of the calibration board at different positions to obtain at least 15 complete pictures of the calibration board;

[0015] S22, corner points are recognized and extracted from the pictures of the calibration board taken by the binocular camera through Zhang's calibration algorithm "A Flexible New Technique for Camera Calibration", and then the internal and external parameters of each camera module are solved, the coordinate conversion relationship between the two camera modules of the binocular camera is solved through the geometric constraint of the epipolar line, and the values of the internal parameters of the two camera modules and the values of the radial and tangential distortion coefficients are obtained, wherein the internal parameters include the camera focal length and the image center coordinates, and the internal parameters can be represented by a 3x4 matrix A;

[0016] S23, the corner point position information on the new calibration board picture is extracted by using the corner detection algorithm, and the three-dimensional space position of the corner point relative to the camera module coordinate system is calculated through the following formula, wherein and represents the horizontal and vertical coordinate values of the corner pixel, A represents the internal memory matrix of the camera module, represents the rotation and translation vectors of the camera module coordinate system and the calibration board coordinate system, P is the three-dimensional space coordinate value of the corner point to be solved, Q represents the product result of the internal parameter matrix and the external parameter matrix of the camera module, and the subscripts and respectively represent the related parameters of the left camera module and the right camera module, and the subscript w represents that the variable is referenced to the world coordinate system:

[0017]

[0018] Then the above formula has the cross product of the left and right sides equal to zero vector:

[0019]

[0020] Then the i-th row of the matrix Q of the above formula is recorded as , and the above formula can be obtained:

[0021]

[0022] S24, singular value decomposition is performed on the matrix A T A, and the eigenvector of the smallest eigenvalue obtained is the three-dimensional space coordinate value of the corner point in the world coordinate system, including:

[0023] S241, the three-dimensional space coordinate values of all corner points on the calibration board are obtained through the calculation method of S21-S23, and the position and attitude of the calibration board relative to the camera module coordinate system are optimized through the PnP algorithm.

[0024] S242, in the hand-eye calibration unit, the six-axis encoding information of the corresponding mechanical arm assembly of the calibration board in the above brand-new calibration board picture is calculated by the rotation matrix T to solve the position and posture transformation relationship of the calibration board relative to the base of the mechanical arm assembly, wherein, through the position and posture relationship of the calibration board relative to the camera module coordinate system and the position and posture transformation relationship of the calibration board relative to the base of the mechanical arm assembly, the relative coordinate information of the link assembly end and the binocular camera is obtained by using the calibration algorithm "Robot sensor calibration:solving AX=XB on the Euclidean group", the position and posture of the calibration board relative to the base of the mechanical arm assembly and the position and posture of the binocular camera relative to the base of the mechanical arm assembly are obtained by matrix inverse arithmetic operation, and the result is that all coordinate systems are converted into the coordinate system relative to the base of the mechanical arm assembly.

[0025] The application has the following beneficial effects: the application can flexibly change the position of the binocular camera through the link assembly, effectively avoids the risk of losing target tracking due to occlusion in the surgical process, and accurately obtains the external parameter matrix of the current position of the camera without re-calibrating the camera when changing the position of the binocular camera, which can quickly update the relative position and posture relationship of the binocular camera and the mechanical arm assembly, and greatly improves the working efficiency of the surgical robot. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a schematic diagram of a navigation positioning system integrating the surgical robot and the vision module in the embodiment. DETAILED DESCRIPTION

[0027] To further illustrate the embodiments, the application provides accompanying drawings. These drawings are part of the disclosure of the application, mainly used to illustrate the embodiments, and can be used to explain the operating principle of the embodiments in conjunction with the related description of the specification. Those skilled in the art should understand other possible implementations and advantages of the application by referring to these contents. The components in the drawings are not drawn to scale, and similar component symbols are usually used to represent similar components.

[0028] The application will be further described in conjunction with the drawings and specific embodiments.

[0029] Reference Figure 1As shown, as a preferred embodiment of the present application, a surgical robot and visual module integrated navigation positioning system is provided, comprising: a binocular camera 1, a mechanical arm assembly 2, a connecting rod assembly 3, a calibration plate 4 and a plane target 5. The mechanical arm assembly 2 comprises a fixed base 20 and a free end 21 capable of changing position through multi-axis movement of the mechanical arm, one end of the connecting rod assembly 3 is rotatably connected to the base 20, and the other end of the connecting rod assembly 3 is rotatably connected with the binocular camera 1. The calibration plate 4 is installed on the free end 21, so that when the free end 21 is controlled to move and change position, the distance between the binocular camera 1 and the calibration plate 4 changes, and the binocular camera 1 can take pictures of the calibration plate 4 at different distances. The plane target 5 is fixedly arranged in front of the binocular camera 1 as a simulated tracking target to estimate the relative position and attitude relationship between the patient and the mechanical arm assembly 2.

[0030] The connecting rod assembly 3 comprises at least N segments of connecting rods connected in sequence, so that N+1 hinge points are formed between the two ends of the connecting rod assembly 3 and between each two adjacent connecting rods, and N is greater than or equal to 2. For example, in the present embodiment, the connecting rod assembly 3 comprises two connecting rods, and the three hinge points are formed by the two ends of the two connecting rods away from each other (i.e. the two ends of the entire connecting rod assembly 3) and the hinge point where the two connecting rods are connected to each other. A rotating damping structure is arranged at each hinge point, such as an oil damping, magnetic damping or the like, so that when an external force is applied to any connecting rod of the connecting rod assembly 3 or the binocular camera 1, any connecting rod of the connecting rod assembly 3 or the binocular camera 1 can rotate around the hinge point by overcoming the damping force of the rotating damping structure, and after the external force is removed, each connecting rod or the binocular camera 1 can be locked at the adjusted position and remain substantially stationary due to the influence of the rotating damping structure.

[0031] Through the arrangement of the connecting rod assembly 3, the position of the binocular camera 1 can be adjusted by deflecting the angle of the binocular camera 1 and the connecting rods in the connecting rod assembly 3, thereby avoiding the binocular camera 1 being blocked.

[0032] Once the position of the binocular camera 1 as a visual module changes, it usually means that the spatial positioning of the surgical robot navigation system needs to be recalibrated, which has a great impact on the working efficiency of the surgical robot navigation system. The surgical robot and visual module integrated navigation positioning method used in the present embodiment further solves this problem.

[0033] [Step S1] Construct a rotation matrix to calculate the relative position of the two ends of the linkage assembly 3. In this embodiment, an encoder 31 is installed at each hinge point of the linkage assembly 3, so that when the posture of any segment of the linkage assembly 3 changes, the relative position of the two ends of the linkage assembly 3 can be calculated in real time by reading the encoding information of the encoder 31 and combining the length of each segment of the linkage assembly 3. Further, in this embodiment, the relative position of the two ends of the linkage assembly 3 is calculated by constructing a rotation matrix. Taking the linkage assembly 3 with two segments of linkages in this embodiment as an example, the hinge point connecting the linkage assembly 3 with the base of the mechanical arm assembly 2 is the first hinge point, the hinge point between the two segments of linkages in the linkage assembly 3 is the second hinge point, and the hinge point connecting the linkage assembly 3 with the binocular camera 1 is the third hinge point:

[0034] The four parameters are used to construct a rotation matrix to calculate the relative position and posture of the next hinge point relative to the previous hinge point (the direction closer to the base of the mechanical arm assembly 2 is the front). The rotation matrix T n is calculated as follows:

[0035]

[0036] wherein parameter a is the distance between the coordinate origin of the previous hinge point and the coordinate origin of the current hinge point along the X-axis direction of the triangular coordinate system of the previous hinge point; parameter d is the distance between the coordinate origin of the previous hinge point and the coordinate origin of the current hinge point along the Z-axis direction of the triangular coordinate system of the previous hinge point; parameter a is the included angle between the Z-axis of the previous hinge point and the current hinge point along the X-axis direction of the triangular coordinate system of the previous hinge point; and parameter θ is the included angle between the X-axis of the previous hinge point and the current hinge point along the Z-axis direction of the triangular coordinate system of the previous hinge point. Parameter θ is obtained in real time by the encoder 31, and the other three parameters are determined when the linkage assembly 3 is designed. It should be noted that the X-axis direction and the Z-axis direction of the triangular coordinate system at the hinge point are understood by those skilled in the art without any doubt.

[0037] In this embodiment, the position and posture of the second hinge point relative to the first hinge point and the position and posture of the third hinge point relative to the second hinge point can be constructed by two rotation matrices, which are connected in series by matrix multiplication, i.e. T=T1×T2, wherein the lower right subscript number is used to distinguish different rotation matrices. It can be understood that if the number of hinge points of the linkage assembly 3 is greater, more rotation matrices T=T1×T2×···×T n may be constructed in series.

[0038] [Step S2] Calculate the relative position and attitude relationship between the free end 21 and the end of the link assembly 3. Since the link assembly 3 is installed on the mechanical arm assembly 2, the relative position and attitude relationship between the free end 21 (or the calibration board 4 installed on the free end 21) and the end of the link assembly 3 (or the binocular camera 1 installed on the end of the link assembly 3) can be further quickly obtained through the size of the link assembly 3 and the position installed on the mechanical arm assembly 2, combined with the above-mentioned rotation matrix method. The method is as follows:

[0039] S21, the binocular camera 1 shoots pictures of the calibration board 4 at different positions, and at least 15 groups of complete pictures of the calibration board are obtained.

[0040] S22, the corner points of the collected pictures are recognized and extracted by Zhang's calibration algorithm "A Flexible New Technique for Camera Calibration", and then the internal and external parameters of each camera are solved. Through the geometric constraint of the epipolar line, the coordinate conversion relationship between the two camera modules of the binocular camera 1 (the binocular camera 1 includes two camera modules respectively arranged on the left and right sides) is solved. The result is that the values of the internal parameters of the two camera modules and the values of the radial distortion coefficient and the tangential distortion coefficient can be quickly obtained through Zhang's calibration algorithm and the epipolar line constraint, wherein the internal parameters include the camera focal length and the image center coordinates. The internal parameters can be represented by a 3x4 matrix A.

[0041] S23, the 15 groups of pictures are corrected for distortion by using the camera distortion coefficients obtained after calibration to obtain 15 new groups of pictures, and then the position information of the corner points on the new 15 groups of pictures is extracted by using the corner detection algorithm. The three-dimensional space position of the corner point relative to the camera module coordinate system is calculated by the following formula, wherein u and v represent the horizontal and vertical coordinate values of the corner point pixel, A represents the camera module internal matrix, represents the rotation and translation vectors of the camera module coordinate system and the calibration board 4 coordinate system, P is the three-dimensional space coordinate value of the corner point to be solved, Q represents the product result of the camera module internal parameter matrix and the external parameter matrix, and the subscript and respectively represent the related parameters of the left camera module and the right camera module, and the subscript w represents that the variable is referenced to the world coordinate system:

[0042]

[0043] Then the above formula has a left and right cross product equal to zero vector:

[0044]

[0045] Then the i-th row of the matrix Q of the above formula is recorded as Then the above formula can be obtained:

[0046]

[0047] S24, by singular value decomposition of matrix A T A, the eigenvector of the minimum eigenvalue is the three-dimensional spatial coordinate value of the corner point in the world coordinate system, wherein:

[0048] S241, by the above calculation method, the three-dimensional spatial coordinate values of all corner points on the calibration plate 4 are obtained, and the position and attitude of the calibration plate 4 relative to the camera module coordinate system are optimized by PnP algorithm, such as SRPnP algorithm“A simple, robust and fast method for the perspective-n-point problem”.

[0049] S242, in the hand-eye calibration unit, the six-axis encoding information of the corresponding mechanical arm assembly 2 of the calibration plate 4 in the above-mentioned 15 new groups of pictures is calculated by the rotation matrix T to solve the position and attitude transformation relationship of the calibration plate 4 relative to the base of the mechanical arm assembly 2. Specifically, by using the position and attitude relationship of the calibration plate 4 relative to the camera module coordinate system and the position and attitude transformation relationship of the calibration plate 4 relative to the base of the mechanical arm assembly 2, the relative coordinate information of the linkage assembly 3 end and the binocular camera 1 is obtained by using the calibration algorithm“Robot sensor calibration: solving AX=XB on the Euclidean group”. Specifically, by using the matrix inverse arithmetic operation, the position and attitude of the calibration plate 4 relative to the base of the mechanical arm assembly 2 and the position and attitude of the binocular camera 1 relative to the base of the mechanical arm assembly 2 are obtained. The result is to convert all coordinate systems into the coordinate system relative to the base of the mechanical arm assembly 2.

[0050] In the implementation process of the embodiment, the position of the binocular camera 1 can be flexibly changed through the linkage assembly 3, effectively avoiding the risk of losing the tracking of the planar target 5 due to occlusion during the operation process, and changing the position of the binocular camera 1 can accurately obtain the external parameter matrix of the current position of the camera without the need for camera calibration again, which can quickly update the relative position and attitude relationship between the binocular camera 1 and the mechanical arm assembly 2, greatly improving the working efficiency of the surgical robot. The calculation formula for calculating the coordinate transformation relationship between the planar target 5 and the binocular camera 1 can be obtained by S23 step.

[0051] The navigation positioning method of the surgical robot and the vision module integrated in the embodiment adopts matrix operation, the formula is more concise, the calculation speed is faster, and the accuracy can also be ensured.

[0052] The Zhang's calibration algorithm "A Flexible New Technique for Camera Calibration", the SRPnP algorithm "A simple, robust and fast method for the perspective-n-point problem", the calibration algorithm "Robot sensor calibration: solving AX=XB on the Euclidean group" mentioned in the present embodiment are all mature existing algorithms in the art.

[0053] In addition, although the present application is specifically shown and described in connection with preferred embodiments, those skilled in the art will appreciate that various modifications in form and detail can be made to the present application without departing from the spirit and scope of the application as defined in the appended claims.

Claims

1. A navigation and positioning system integrating a surgical robot and a visual module, characterized in that: include: A binocular camera, a robotic arm assembly, a connecting rod assembly, a calibration plate, and a plane target. The robotic arm assembly includes a fixed base and a free end that can change position through multi-axis motion of the robotic arm. One end of the connecting rod assembly is rotatably connected to the base, and the other end of the connecting rod assembly is rotatably connected to the binocular camera. The calibration plate is installed at the free end, and the plane target is fixedly set in front of the binocular camera. The connecting rod assembly includes at least N connecting rods hinged in sequence, with a total of N+1 hinges formed at both ends of the connecting rod assembly and between every two adjacent connecting rods, where N is greater than or equal to 2. A rotation damping structure is provided at each hinge point, and an encoder is also installed at each hinge point. The encoder's encoding information is used to calculate the relative positions of the head and tail ends of the connecting rod assembly when the connecting rod assembly undergoes a rotational posture change, and the relative positions of the free end and the binocular camera at the end of the connecting rod assembly are calculated with the help of the relative positions of the head and tail ends of the connecting rod assembly.

2. A navigation and positioning method integrating a surgical robot and a visual module, characterized in that: The method for calculating the relative position of the binocular camera at the free end of the manipulator assembly and the end of the connecting rod assembly in the navigation and positioning system integrating the surgical robot and the visual module of claim 1 comprises: Step S1, construct the rotation matrix T = T1×T2×···×T n To calculate the relative position of the connecting rod assembly at both ends, T n Represents the rotation matrix constructed by the position and posture of the latter hinge point compared to the previous hinge point. The rotation matrix T n The calculation formula is: Among them, parameter a is the distance between the coordinate origin of the previous hinge point and the coordinate origin of the current hinge point along the X-axis direction of the previous hinge point triangular coordinate system; parameter d is the distance between the coordinate origin of the previous hinge point and the coordinate origin of the current hinge point along the Z-axis direction of the previous hinge point triangular coordinate system; parameter α is the angle between the previous hinge point and the Z-axis of the current hinge point along the X-axis direction of the previous hinge point triangular coordinate system; parameter θ is the angle between the previous hinge point and the X-axis of the current hinge point along the Z-axis direction of the previous hinge point triangular coordinate system; parameter θ is obtained by real-time calculation of the encoder, and the other three parameters are all confirmed when the connecting rod assembly is designed; Step S2, calculating the relative position and posture relationship between the free end of the robotic arm assembly and the binocular camera at the end of the connecting rod assembly, including: S21, the binocular camera takes pictures of the calibration plate at different positions, and obtains at least 15 sets of complete pictures of the calibration plate; S22, using Zhang's calibration algorithm ("A Flexible New Technique for Camera Calibration") to identify and extract corner points from an image of the calibration plate taken by the binocular camera. Then, the intrinsic and extrinsic parameters of each camera module are calculated. Using epipolar geometric constraints, the coordinate transformation relationship between the two camera modules of the binocular camera is solved to obtain the intrinsic parameters of the two camera modules, as well as the radial and tangential distortion coefficients. The intrinsic parameters include the camera focal length and the image center coordinates. The internal parameters can be represented by a 3×4 matrix A. S23, using the camera distortion coefficients obtained after calibration to correct the distortion of the calibration plate image to obtain a new calibration plate image, and then using a corner detection algorithm to extract the position information of the corner points on the new calibration plate image. The three-dimensional spatial position of the corner points relative to the camera module coordinate system is calculated using the following formula, where u and v represent the horizontal and vertical coordinate values ​​of the corner point pixels, and A represents the camera module memory matrix. Represents the rotation and translation vectors of the camera module coordinate system and the calibration plate coordinate system, P is the three-dimensional space coordinate value of the corner point, Q represents the product of the internal parameter matrix and the external parameter matrix of the camera module, and the subscript and Represent the relevant parameters of the left camera module and the right camera module respectively. The subscript w indicates that the variable is based on the world coordinate system: Then the above formula exists where the cross product of the left and right terms is equal to the zero vector: Then the i-th row of the matrix Q of the above formula is recorded as , then the above formula can be obtained: S24, by the matrix A T A is subjected to singular value decomposition, and the eigenvector of the minimum eigenvalue obtained is the three-dimensional space coordinate value of the corner point in the world coordinate system, including: S241, obtaining the three-dimensional spatial coordinate values ​​of all corner points on the calibration plate through the calculation method of S21-S23, and optimizing the position and posture of the calibration plate relative to the camera module coordinate system through the PnP algorithm; S242, in the hand-eye calibration unit, the six-axis encoding information of the robotic arm assembly corresponding to the calibration plate in the above-mentioned new calibration plate image is solved by calculating the position and posture transformation relationship of the calibration plate relative to the base of the robotic arm assembly through the rotation matrix T, wherein, through the position and posture relationship of the calibration plate relative to the camera module coordinate system and the position and posture transformation relationship of the calibration plate relative to the base of the robotic arm assembly, the calibration algorithm "Robot sensor calibration: solving AX=XB on the Euclidean group" is used to obtain the relative coordinate information of the end of the connecting rod assembly and the binocular camera, and the position and posture of the calibration plate relative to the base of the robotic arm assembly and the position and posture of the binocular camera relative to the base of the robotic arm assembly are obtained through the arithmetic operation of matrix inversion. As a result, all coordinate systems are converted into coordinate systems relative to the base of the robotic arm assembly.

Images