Probe-based intraoperative real-time robot hand-eye calibration method and system

Abstract

The method comprises the following steps: acquiring a relative pose A of a mechanical arm end relative to a mechanical arm base and a relative pose B of a probe of the mechanical arm end relative to a camera; establishing a pose relationship of the probe relative to the mechanical arm base through a preset calibration equation; calculating and solving the calibration equation to obtain a relative pose X of the probe relative to the mechanical arm end and a relative pose Z of the camera relative to the mechanical arm base; and determining real-time 5-degree-of-freedom pose information of the probe based on the calculated relative pose X of the probe relative to the mechanical arm end and the relative pose Z of the camera relative to the mechanical arm base. Through the processing scheme, the operation accuracy of the surgical robot is improved.

Description

Technical Field

[0001] This disclosure relates to the field of surgical robot data processing technology, and in particular to a probe-based method and system for intraoperative real-time robotic hand-eye calibration. Background Technology

[0002] Hand-eye calibration is fundamental to vision-guided robot control methods. It maps sensor-identified objects to the robotic arm's coordinate space, enabling precise grasping and manipulation of targets. However, current hand-eye calibration technologies have several shortcomings in medical surgery, primarily the following:

[0003] 1) In actual surgery, for hygiene and safety reasons, doctors usually change surgical instruments or move cameras multiple times.

[0004] Each time instruments are changed or cameras are adjusted, it is difficult to guarantee that the size and installation position of the instruments are precisely consistent, thus requiring recalibration of the surgical robot. Currently, common hand-eye calibration methods require specialized reference materials, typically using a checkerboard crosshair calibration board; however, these methods are unusable during surgery. In clinical practice, the most convenient method is calibration using surgical instruments, with needle-like instruments (such as probes, puncture needles, non-deformable catheters, cotton swabs, etc.) being the most common. Using these tools as reference materials is extremely convenient in clinical practice.

[0005] 2) Since the size, length, and placement of the probe are unknown, only the position of its axis (5 degrees of freedom) can be reliably and accurately measured from the sensor. Existing calibration algorithms require measuring the target's complete attitude (6 degrees of freedom) from the sensor.

[0006] To address this issue, this invention develops a novel calibration algorithm that transforms the complete matrix problem into an optimization problem on a Lie group subspace with only 5 degrees of freedom, thereby realizing this calibration method.

[0007] Based on existing technology, this invention discloses a method for real-time intraoperative hand-eye calibration using a probe. By replacing the checkerboard calibration board with a probe, and calculating and solving for five degrees of freedom, the probe size, length, relative pose of the probe to the end effector of the robotic arm, and relative pose of the camera to the base of the robotic arm are obtained. This allows the surgical robot to adjust in real-time during surgery, maintaining high precision. Summary of the Invention

[0008] In view of this, the present disclosure provides a probe-based intraoperative real-time robotic hand-eye calibration method and system to at least partially solve the problems existing in the prior art.

[0009] In a first aspect, embodiments of this disclosure provide a probe-based method for intraoperative real-time robotic hand-eye calibration, including:

[0010] Obtain the relative pose A of the robotic arm end effector relative to the robotic arm base and the relative pose B of the probe of the robotic arm end effector relative to the camera. A is a known 6-DOF pose of the robotic arm end effector relative to the robotic arm base and B is a known 5-DOF pose of the probe of the robotic arm relative to the camera.

[0011] The pose relationship of the probe relative to the robotic arm base is established by using a preset calibration equation;

[0012] The calibration equations are calculated and solved to obtain the relative pose X of the probe relative to the end of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm. X is the unknown 5-DOF pose of the probe relative to the end of the robotic arm; Z is the unknown 6-DOF pose of the camera relative to the base of the robotic arm.

[0013] Based on the calculated relative pose X of the probe to the end of the robotic arm and the relative pose Z of the camera to the base of the robotic arm, the real-time 5-DOF pose information of the probe is determined.

[0014] According to a specific implementation of this disclosure, before obtaining the relative pose A of the robotic arm end effector relative to the robotic arm base and the relative pose of the probe at the robotic arm end effector relative to the camera, the method further includes:

[0015] Four parameters A, B, X, and Z are preset, where A is the known 6-DOF pose of the robotic arm end effector relative to the robotic arm base, B is the known 5-DOF pose of the probe relative to the camera, X is the unknown 5-DOF pose of the probe relative to the robotic arm end effector, and Z is the unknown 6-DOF pose of the camera relative to the robotic arm base.

[0016] According to a specific implementation of this disclosure, obtaining the relative pose A of the robotic arm end effector relative to the robotic arm base and the relative pose of the probe at the robotic arm end effector relative to the camera includes:

[0017] The relative pose B of the probe with respect to the camera is obtained through data acquisition from the camera.

[0018] According to a specific implementation of this disclosure, establishing the pose relationship of the probe relative to the robotic arm base through a preset calibration equation includes:

[0019] Based on the parameter settings, the pose of the probe relative to the robotic arm base was calculated using both AX and ZB methods, resulting in...

[0020] Equation [1]: AX = ZB.

[0021] According to a specific implementation of this disclosure, the step of calculating and solving the calibration equation to obtain the relative pose X of the probe relative to the end effector of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm includes:

[0022] Assuming that all 6 degrees of freedom of variables A and B are known, AX = ZB can be expressed in the following form:

[0023]

[0024] This leads to equation [2]: R A R X =R Z R B And equation [3] R A t X +t A =R z t B +t Z .

[0025] According to a specific implementation of this disclosure, the step of calculating and solving the calibration equation to obtain the relative pose X of the probe relative to the end effector of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm further includes:

[0026] The attitude components of X and B on the Z-axis are ignored, and the calibration method is expressed in the form of an incomplete Lie algebra se(3). The R in equations [2] and [3] is then used. X and R B Replace it with a 3D vector, denoted as r X ,r B ,

[0027] In fact

[0028] Thus, we obtain a new equation form:

[0029] Equation [4]: ​​R A r X =R Z r B And equation [5]: R A t X +t A =R Z t B +t Z .

[0030] According to a specific implementation of this disclosure, the step of calculating and solving the calibration equation to obtain the relative pose X of the probe relative to the end effector of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm further includes:

[0031] Suppose Z is an unknown parameter with 6 degrees of freedom, and

[0032] Z = Exp(z) and z = (p z θ z Where p and θ represent relative displacement and relative angular displacement, respectively.

[0033] Let r l =R A r X r r =R Z r B Δr=r l -r r =R A r X -R Z r B ;

[0034] t l =R A t X +t A , t r =R Z t B +t Z Δ t =t l -t r =R A t X +t A -R Z t B -t Z

[0035] For sampling points A1, A2, A3, ... and their corresponding B1, B2, B3, ..., X and Z are obtained after optimization using formula [1], where,

[0036] Formula [1]:

[0037] According to a specific implementation of this disclosure, the step of calculating and solving the calibration equation to obtain the relative pose X of the probe relative to the end effector of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm further includes:

[0038] For formula [1], first give a set of initial values ​​X and Z. (0) Z (0) Let's set up a regression model.

[0039] Formula [2]:

[0040] Where J(Δr) and J(Δt) are the Jacobian matrices of Δr and Δt with respect to X and Z, respectively.

[0041] By iteratively applying formula [2], X can be obtained. (0) Z (0) X (1) Z (1) X (2) Z (2) ...and thus obtain

[0042]

[0043] According to a specific implementation of this disclosure, the step of calculating and solving the calibration equation to obtain the relative pose X of the probe relative to the end effector of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm further includes:

[0044] In the specific solution process, formula [2] is expanded to become

[0045]

[0046] in, The vector (Lie algebra) t represents the antisymmetric matrix; Log(R) is the Lie group R transformed into Lie algebra form; j l It is the left Jacobi of a 3-dimensional Lie algebra, and we have:

[0047]

[0048]

[0049] in a r It is a unit vector of r, i.e., a r =r / θ0 can be solved using the new formula, X (k+1) and Z (k+1) This allows us to obtain X and Z, which represent the real-time poses of the probe and the camera.

[0050] Secondly, embodiments of this disclosure provide a probe-based intraoperative real-time robotic hand-eye calibration system, comprising:

[0051] The acquisition module is used to acquire the relative pose A of the robotic arm end effector relative to the robotic arm base and the relative pose B of the probe of the robotic arm end effector relative to the camera. A is a known 6-DOF pose of the robotic arm end effector relative to the robotic arm base and B is a known 5-DOF pose of the probe of the robotic arm relative to the camera.

[0052] A module is established to establish the pose relationship between the probe and the robotic arm base using a preset calibration equation.

[0053] The calculation module is used to calculate and solve the calibration equation to obtain the relative pose X of the probe relative to the end of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm. X is the unknown 5-DOF pose of the probe relative to the end of the robotic arm; Z is the unknown 6-DOF pose of the camera relative to the base of the robotic arm.

[0054] The determination module is used to determine the real-time 5-DOF pose information of the probe based on the calculated relative pose X of the probe relative to the end of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm.

[0055] Thirdly, embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the probe-based intraoperative real-time robotic hand-eye calibration method in the first aspect or any implementation thereof.

[0056] Fourthly, embodiments of this disclosure also provide a computer program product, which includes a computing program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the probe-based intraoperative real-time robotic hand-eye calibration method in the first aspect or any implementation thereof.

[0057] The probe-based intraoperative real-time robotic hand-eye calibration scheme in this embodiment includes acquiring the relative pose A of the robotic arm end effector relative to the robotic arm base and the relative pose B of the probe at the robotic arm end effector relative to the camera, where A is a known 6-DOF pose of the robotic arm end effector relative to the robotic arm base and B is a known 5-DOF pose of the probe relative to the camera; establishing the pose relationship of the probe relative to the robotic arm base through a preset calibration equation; calculating and solving the calibration equation to obtain the relative pose X of the probe relative to the robotic arm end effector and the relative pose Z of the camera relative to the robotic arm base, where X is an unknown 5-DOF pose of the probe relative to the robotic arm end effector and Z is an unknown 6-DOF pose of the camera relative to the robotic arm base; and determining the real-time 5-DOF pose information of the probe based on the calculated relative pose X of the probe relative to the robotic arm end effector and the relative pose Z of the camera relative to the robotic arm base. The present invention provides a novel calibration algorithm that transforms the complete matrix problem into an optimization problem in a Lie group subspace with only 5 degrees of freedom, thereby realizing this calibration method. Simultaneously, a probe is used instead of the checkerboard calibration board. By calculating and solving the 5 degrees of freedom, the probe size, length, relative pose of the probe to the end effector of the robotic arm, and relative pose of the camera to the base of the robotic arm are obtained. This allows the surgical robot to adjust in real time during surgery, maintaining high precision. Attached Figure Description

[0058] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A schematic diagram of probe degrees of freedom provided in an embodiment of this disclosure;

[0060] Figure 2 This is a schematic diagram of the probe robot system structure provided in an embodiment of the present disclosure;

[0061] Figure 3 A flowchart illustrating robot hand-eye calibration provided in this embodiment of the disclosure. Detailed Implementation

[0062] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0063] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0064] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0065] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0066] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0067] This disclosure provides a probe-based intraoperative real-time robotic hand-eye calibration method. The probe-based intraoperative real-time robotic hand-eye calibration method provided in this embodiment can be executed by a computing device, which can be implemented as software or a combination of software and hardware. This computing device can be integrated into a server, client, or similar device.

[0068] See Figure 1 , Figure 2 and Figure 3 The probe-based intraoperative real-time robotic hand-eye calibration method in this disclosure may include the following steps:

[0069] S101, obtain the relative pose A of the end effector of the robotic arm with respect to the base of the robotic arm and the relative pose B of the probe of the end effector of the robotic arm with respect to the camera. A is a known 6-DOF pose of the end effector of the robotic arm with respect to the base of the robotic arm and B is a known 5-DOF pose of the probe with respect to the camera.

[0070] S102, The pose relationship of the probe relative to the robotic arm base is established through a preset calibration equation;

[0071] S103, calculate and solve the calibration equation to obtain the relative pose X of the probe relative to the end of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm. X is the unknown 5-DOF pose of the probe relative to the end of the robotic arm; Z is the unknown 6-DOF pose of the camera relative to the base of the robotic arm.

[0072] S104, based on the calculated relative pose X of the probe relative to the end of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm, determine the real-time 5-DOF pose information of the probe.

[0073] Specifically, this application proposes a method for real-time intraoperative hand-eye alignment using a probe, which mainly includes the following aspects:

[0074] 1) Intraoperative probe real-time hand-eye calibration technology:

[0075] By using a probe instead of the existing checkerboard calibration board, the precise pose of the probe can be acquired in real time during surgery. The degrees of freedom of the needle in this invention are as follows: Figure 1 As shown. Figure 1 As shown, let (x,y,z) and (nx,n) y ,n z These represent the position and orientation of the probe relative to the end effector of the robotic arm (or camera), respectively. In the local coordinate system with the probe as the reference frame, the direction of the probe is the z-axis.

[0076] When using a probe during surgery, only the position of the probe tip (x, y, z) and the axis n of the probe can be measured; the spin angle of the probe on the z-axis cannot be measured. Therefore, during calculations, the probe's pose relative to the robotic arm's end effector (or camera) has only 5 degrees of freedom, including the spin angle around n. z The rotation angle is an unknown variable.

[0077] 2) After calibration at different locations using probes, the solution is obtained through calculations using 5 degrees of freedom:

[0078] The relative pose A of the robotic arm is changed multiple times, while the relative pose B of the probe relative to the camera is acquired through a camera, thereby obtaining a series of sampling points A1, A2, A3, ... and corresponding B1, B2, B3, ... . The equation AX = ZB (where X and B each have only 5 degrees of freedom) is established, and the 5 degrees of freedom are calculated and solved using the method proposed in this invention.

[0079] 3) After calculation and solution, the probe size, length, relative pose X of the probe relative to the end of the robotic arm, and relative pose Z of the camera relative to the base of the robotic arm are finally obtained.

[0080] See Figure 2 The entire system involved in this application consists of three parts:

[0081] 1. Robotic arm: High-performance multi-axis robotic arm used for computation and control processing;

[0082] 2. Probe: Fixed to the end of the robotic arm, its position changes with the movement of the robotic arm, used for real-time online calibration;

[0083] 3. Camera: Fixed in the space outside the robot arm body, the camera's pose remains unchanged relative to the world coordinate system, and is used to detect the probe's pose;

[0084] The flowchart of the calibration calculation method is as follows: Figure 3 As shown, it mainly includes the following steps:

[0085] 1. Set parameters

[0086] Set four parameters: A, B, X, and Z.

[0087] A: The known pose of the robotic arm end effector relative to the robotic arm base (6 degrees of freedom);

[0088] B: The known pose of the probe relative to the camera (only 5 degrees of freedom);

[0089] X: The pose of the unknown probe relative to the end effector of the robotic arm (only 5 degrees of freedom);

[0090] Z: Unknown camera pose relative to the robotic arm base (6 degrees of freedom);

[0091] 2. Data Collection

[0092] 1) Set the relative pose A of the robotic arm end effector with respect to the robotic arm base;

[0093] 2) Obtain the relative pose B of the probe with respect to the camera through data acquisition from the camera;

[0094] 3. Calibrate the equations

[0095] Similar to general hand-eye calibration methods, the probe's pose relative to the robotic arm base was calculated using both AX and ZB methods based on the parameter settings.

[0096] Equation [1]: AX = ZB

[0097] Of course, since the degree of freedom parameter of B is missing, equation [1] cannot be solved directly.

[0098] 4. Lie group subspace optimization solution method

[0099] By solving equation [1], the relative pose X of the probe relative to the end of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm are obtained. The specific solution process is as follows:

[0100] ① Transform equation [1] AX = ZB into a solvable equation form:

[0101] Generally, when all six degrees of freedom of variables A and B are known, AX = ZB will take the following form:

[0102]

[0103] This leads to equation [2]: R A R X =R Z R B And equation [3] R A t X +t A =R z t B +t Z;

[0104] ② Lie group molecular space of probe orientation

[0105] However, since the attitude components of X and B on the Z-axis are not considered in this invention, it must be represented by the (incomplete) Lie algebra se(3). The R in the above two equations... X and R B Replace it with a 3D vector, denoted as r X ,r B .

[0106] In fact

[0107] Thus, a new equation form can be obtained:

[0108] Equation [4]: ​​R A r X =R Z r B

[0109] and

[0110] Equation [5]: R A t X +t A =R Z t B + t Z

[0111] Next, we need to solve for r in the two equations above. X ,t X And Z.

[0112] ③ Lie group subspace optimization solution method

[0113] Before solving the equation, we first define some variables.

[0114] Z is an unknown parameter with 6 degrees of freedom, and

[0115] Z = Exp(z) and z = (p z θ z )

[0116] Where p and θ represent relative displacement and relative angular displacement, respectively.

[0117] Let r l =R A r X r r =R Z r B Δr=r l -r r =R A r X-R Z r B ;

[0118] t l =R A t X +t A , t r =R Z t B +t Z Δt=t l -t r =R A t X +t A -R Z t B -t Z

[0119] For sampling points A1, A2, A3, ... and corresponding B1, B2, B3, ..., X and Z are obtained after optimization using formula [1].

[0120] Formula [1]:

[0121] ④ Nonlinear iterative numerical solution method

[0122] The solution of formula [1] is not an analytical solution, but an estimate obtained by using the Gauss-Newton iteration method. The solution process is shown below.

[0123] First, give a set of initial values ​​for X and Z. (0) Z (0) Let's set up a regression model.

[0124] Formula [2]:

[0125] Where J(Δr) and J(Δt) are the Jacobian matrices of Δr and Δt with respect to X and Z, respectively.

[0126] Therefore, X can be obtained by iteratively applying formula [2]. (0) Z (0) X (1) Z (1) X (2) Z (2) ...and thus obtain

[0127]

[0128] ⑤ Methods for calculating the Jacobian of subspaces

[0129] In this problem, J(Δr) and J(Δt) are both very complex Jacobian matrices. In the specific solution process, formula [2] expands to become...

[0130]

[0131] in, The vector (Lie algebra) t represents the antisymmetric matrix; Log(R) is the Lie group R transformed into Lie algebra form; j l It is the left Jacobi of a 3-dimensional Lie algebra, and we have:

[0132]

[0133]

[0134] in a r It is a unit vector of r, i.e., a r With -r / θ0, we can then use the new formula to solve for X. (k+1) and Z (k+1) This leads to X and Z, which represent the poses of the probe and the camera.

[0135] According to a specific implementation of this disclosure, before obtaining the relative pose A of the robotic arm end effector relative to the robotic arm base and the relative pose of the probe at the robotic arm end effector relative to the camera, the method further includes:

[0136] Four parameters A, B, X, and Z are preset, where A is the known 6-DOF pose of the robotic arm end effector relative to the robotic arm base, B is the known 5-DOF pose of the probe relative to the camera, X is the unknown 5-DOF pose of the probe relative to the robotic arm end effector, and Z is the unknown 6-DOF pose of the camera relative to the robotic arm base.

[0137] According to a specific implementation of this disclosure, obtaining the relative pose A of the robotic arm end effector relative to the robotic arm base and the relative pose of the probe at the robotic arm end effector relative to the camera includes:

[0138] The relative pose B of the probe with respect to the camera is obtained through data acquisition from the camera.

[0139] According to a specific implementation of this disclosure, establishing the pose relationship of the probe relative to the robotic arm base through a preset calibration equation includes:

[0140] Based on the parameter settings, the pose of the probe relative to the robotic arm base was calculated using both AX and ZB methods, resulting in...

[0141] Equation [1]: AX = ZB.

[0142] According to a specific implementation of this disclosure, the step of calculating and solving the calibration equation to obtain the relative pose X of the probe relative to the end effector of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm includes:

[0143] Assuming that all 6 degrees of freedom of variables A and B are known, AX = ZB can be expressed in the following form:

[0144]

[0145] This leads to equation [2]: R A R X =R Z R B And equation [3] R A t X +t A =R z t B +t Z .

[0146] According to a specific implementation of this disclosure, the step of calculating and solving the calibration equation to obtain the relative pose X of the probe relative to the end effector of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm further includes:

[0147] The attitude components of X and B on the Z-axis are ignored, and the calibration method is expressed in the form of an incomplete Lie algebra se(3). The R in equations [2] and [3] is then used. X and R B Replace it with a 3D vector, denoted as r X ,r B ,

[0148] In fact

[0149] Thus, we obtain a new equation form:

[0150] Equation [4]: ​​R A r X =R Z r B And equation [5]: R A t X +t A =R Z t B +t Z .

[0151] According to a specific implementation of this disclosure, the step of calculating and solving the calibration equation to obtain the relative pose X of the probe relative to the end effector of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm further includes:

[0152] Suppose Z is an unknown parameter with 6 degrees of freedom, and

[0153] Z = Exp(z) and z = (p z θ z )

[0154] Where p and θ represent relative displacement and relative angular displacement, respectively.

[0155] Let r l =R A r X r r =R Z r B Δr=r l -r r =R A r X -R Z r B ;

[0156] t l =R A t X +t A , t r =R Z t B +t Z Δt=t l -t r =R A t X +t A -R Z t B -t Z

[0157] For sampling points A1, A2, A3, ... and their corresponding B1, B2, B3, ..., X and Z are obtained after optimization using formula [1], where,

[0158] Formula [1]:

[0159] According to a specific implementation of this disclosure, the step of calculating and solving the calibration equation to obtain the relative pose X of the probe relative to the end effector of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm further includes:

[0160] For formula [1], first give a set of initial values ​​X and Z. (0) Z (0 ), and then set up a regression model

[0161] Formula [2]:

[0162] Where J(Δr) and J(Δt) are the Jacobian matrices of Δt and Δt with respect to X and Z, respectively.

[0163] By iteratively applying formula [2], X can be obtained. (0) Z (0) X (1) Z (1) X (2) Z (2) ...and thus obtain

[0164]

[0165] According to a specific implementation of this disclosure, the step of calculating and solving the calibration equation to obtain the relative pose X of the probe relative to the end effector of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm further includes:

[0166] In the specific solution process, formula [2] is expanded to become

[0167]

[0168] in, The vector (Lie algebra) t represents the antisymmetric matrix; Log(R) is the Lie group R transformed into Lie algebra form; j l It is the left Jacobi of a 3-dimensional Lie algebra, and we have:

[0169]

[0170]

[0171] in a r It is a unit vector of r, i.e., a r =r / θ0

[0172] Solve using the new formula, X (k+1) and Z (k+1) This allows us to obtain X and Z, which represent the real-time poses of the probe and the camera.

[0173] Corresponding to the above method embodiments, this application also provides a probe-based intraoperative real-time robotic hand-eye calibration system, comprising:

[0174] The acquisition module is used to acquire the relative pose A of the robotic arm end effector relative to the robotic arm base and the relative pose B of the probe of the robotic arm end effector relative to the camera. A is a known 6-DOF pose of the robotic arm end effector relative to the robotic arm base and B is a known 5-DOF pose of the probe of the robotic arm relative to the camera.

[0175] A module is established to establish the pose relationship between the probe and the robotic arm base using a preset calibration equation.

[0176] The calculation module is used to calculate and solve the calibration equation to obtain the relative pose X of the probe relative to the end of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm. X is the unknown 5-DOF pose of the probe relative to the end of the robotic arm; Z is the unknown 6-DOF pose of the camera relative to the base of the robotic arm.

[0177] The determination module is used to determine the real-time 5-DOF pose information of the probe based on the calculated relative pose X of the probe relative to the end of the robotic arm and the relative pose Z of the camera relative to the base of the robotic arm.

[0178] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0179] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0180] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0181] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A probe-based intraoperative real-time robot hand-eye calibration method, characterized in that, The method comprises the following steps: obtaining a relative pose A of a mechanical arm end relative to a mechanical arm base and a relative pose B of a probe of the mechanical arm end relative to a camera, A being a known 6-DOF pose of the mechanical arm end relative to the mechanical arm base, and B being a known 5-DOF pose of the probe relative to the camera; establishing a pose relationship of the probe relative to the mechanical arm base through a preset calibration equation; solving the calibration equation to obtain a relative pose X of the probe relative to the mechanical arm end and a relative pose Z of the camera relative to the mechanical arm base, X being a 5-DOF pose of the probe relative to the mechanical arm end, and Z being a 6-DOF pose of the camera relative to the mechanical arm base; based on the calculated relative pose X of the probe relative to the mechanical arm end and the relative pose Z of the camera relative to the mechanical arm base, determining real-time 5-DOF pose information of the probe; wherein before the step of obtaining the relative pose A of the mechanical arm end relative to the mechanical arm base and the relative pose of the probe of the mechanical arm end relative to the camera, the method further comprises: pre-setting four parameters A, B, X and Z, wherein A is a known 6-DOF pose of the mechanical arm end relative to the mechanical arm base, B is a known 5-DOF pose of the probe relative to the camera, X is a 5-DOF pose of the probe relative to the mechanical arm end, and Z is a 6-DOF pose of the camera relative to the mechanical arm base.

2. The method of claim 1, wherein, The step of obtaining the relative pose A of the mechanical arm end relative to the mechanical arm base and the relative pose of the probe of the mechanical arm end relative to the camera comprises: obtaining the relative pose B of the probe relative to the camera through data acquisition of the camera.

3. The method of claim 2, wherein, The step of establishing the pose relationship of the probe relative to the mechanical arm base through the preset calibration equation comprises: according to the parameter setting, calculating the pose of the probe relative to the mechanical arm base by using two methods of AX and ZB, to obtain equation【1】: AX=ZB.

4. The method of claim 1, wherein, The step of solving the calibration equation to obtain the relative pose X of the probe relative to the mechanical arm end and the relative pose Z of the camera relative to the mechanical arm base comprises: assuming that the 6-DOF variables of A and B are known, the expression of AX=ZB is as follows: Further, equation 【2】: R A R X = R Z R B and equation 【3】 R A t X + t A = R z t B + t Z .

5. The method of claim 4, wherein, The step of solving the calibration equation to obtain the relative pose X of the probe relative to the mechanical arm end and the relative pose Z of the camera relative to the mechanical arm base further comprises: The attitude components of X and B on the Z axis are not considered, and the calibration method is expressed in the form of an incomplete Lie algebra se(3), and R X and R B in equations 【2】 and 【3】 are replaced by a 3D vector, denoted as r X , r B , In practice thus, a new equation form is obtained: Equation [4]: R A r X = R Z r B and Equation [5]: R A t X + t A = R Z t B + t Z .

6. The method of claim 5, wherein, The step of solving the calibration equation to obtain the relative pose X of the probe relative to the mechanical arm end and the relative pose Z of the camera relative to the mechanical arm base further comprises: assuming that Z is a 6-DOF unknown parameter, and Z = Exp(z) and z = (p z , θ z ) wherein p and θ represent relative displacement and relative angular displacement respectively, Let r l = R A r X , r r = R Z r B , Δr = r l - r r = R A r X - R Z r B ; t l = R A t X + t A , Δt = t r = R Z t B + t Z , Δt = t l - t r = R A t X + t A - R Z t B - t Z For sampling points A1, A2, A3,..., and corresponding B1, B2, B3,..., using formula 【1】, after optimization processing, X, Z are obtained, wherein, Equation [1]:

7. The method of claim 6, wherein, The step of solving the calibration equation to obtain the relative pose X of the probe relative to the mechanical arm end and the relative pose Z of the camera relative to the mechanical arm base further comprises: A set of initial values of X and Z is given to equation 【1】 (0) , Z (0) , and a regression model equation 【2】 is set as follows: wherein J(Δr) and J(Δt) are Jacobian matrices of Δr and Δt relative to X and Z respectively, By iteratively applying formula [2], X can be obtained. (0) Z (0) ,X (1) , z (1) x (2) Z (2) ...and thus obtain 8. The method of claim 7, wherein, The calculating and solving of the calibration equation obtains a relative pose X of the probe relative to the end of the mechanical arm and a relative pose Z of the camera relative to the base of the mechanical arm, and further comprises: In the specific solving process, formula 2 is expanded into where denotes the anti-symmetric matrix corresponding to the vector (Lie algebra) t; Log(R) is the Lie group R transformed into Lie algebra form; j l is the left Jacobian of the 3-dimensional Lie algebra, and has: wherein a r is a unit vector of r, i.e. a r = r / θ0 Solve with new formula, X (k+1) and z (k+1) , and then get X and Z, that is, the real-time pose of the probe and the camera.

9. A probe-based intraoperative real-time robot hand-eye calibration system, characterized in that, Comprise: The acquisition module is configured to acquire a relative pose A of the end of the mechanical arm relative to the base of the mechanical arm and a relative pose B of a probe of the end of the mechanical arm relative to the camera, A is a known 6-DOF pose of the end of the mechanical arm relative to the base of the mechanical arm, and B is a known 5-DOF pose of the probe relative to the camera; The establishment module is configured to establish a pose relationship of the probe relative to the base of the mechanical arm through a preset calibration equation; The calculation module is configured to calculate and solve the calibration equation to obtain a relative pose X of the probe relative to the end of the mechanical arm and a relative pose Z of the camera relative to the base of the mechanical arm, X is a 5-DOF pose of the probe relative to the end of the mechanical arm, and Z is a 6-DOF pose of the camera relative to the base of the mechanical arm; The determination module is configured to determine 5-DOF pose information of the probe in real time based on the calculated relative pose X of the probe relative to the end of the mechanical arm and the relative pose Z of the camera relative to the base of the mechanical arm. Wherein Before the acquisition of the relative pose A of the end of the mechanical arm relative to the base of the mechanical arm and the relative pose of the probe of the end of the mechanical arm relative to the camera, the system is further configured to: Pre-set A, B, X and Z, wherein A is a known 6-DOF pose of the end of the mechanical arm relative to the base of the mechanical arm, B is a known 5-DOF pose of the probe relative to the camera, X is a 5-DOF pose of the probe relative to the end of the mechanical arm, and Z is a 6-DOF pose of the camera relative to the base of the mechanical arm.

Images