Multi-target calibration method for ultrasonic robot

Abstract

The invention discloses a multi-target calibration method of an ultrasonic robot, and relates to the technical field of automatic control, and the method comprises the following steps: constructing a forward kinematics model of a mechanical arm; a first transformation matrix between a base of the mechanical arm and the tail end of the mechanical arm is obtained through calculation according to the forward kinematics model; calibrating according to an image shot by the global camera to obtain a second transformation matrix between the global camera and the mark of the mechanical arm; a third transformation matrix between the tail end of the mechanical arm and the mark of the mechanical arm is determined according to the inherent characteristics of the ultrasonic probe and the needle head; and detecting the homogeneous transformation matrix of the two-dimensional code on the ultrasonic probe by using the global camera, and further calibrating a fourth transformation matrix between the mark of the mechanical arm and the ultrasonic probe and a fifth transformation matrix between the global camera and the base of the mechanical arm. According to the method, the ultrasonic robot is subjected to multi-target calibration, so that the accuracy of ultrasonic scanning can be improved.

Description

Technical Field

[0001] The present application relates to the field of automatic control technology, and in particular to a multi-target calibration method for an ultrasonic robot. Background Art

[0002] Ultrasound robots are a product of the integration of traditional medicine and modern technology, representing a continuation and development of ultrasound diagnosis and treatment. Ultrasound robots are generally composed of an ultrasonic manipulator, ultrasound equipment, force sensors, and a global camera, capable of autonomous ultrasound scanning. Therefore, system calibration is a crucial prerequisite for accurate ultrasound scanning. Ultrasound robot system calibration involves multi-target calibration, including calibration of the ultrasound probe and the global camera. Ultrasound probe calibration is a critical step in medical ultrasound imaging and related applications. Ultrasound technology is widely used for real-time imaging in medical imaging, but its two-dimensional images inherently lack spatial position information. For applications such as guided surgery, interventional therapy, and three-dimensional reconstruction, the ultrasound probe image plane must be mapped to the robot's workspace. Ultrasound probe calibration is fundamental to achieving high-precision image fusion and real-time spatial tracking. Global camera calibration is also a key technology for achieving high-precision visual perception and control of ultrasound robots. The global camera models the robot's working environment and provides high-precision visual information. This information is based on the global camera coordinate system, and calibration is required to obtain the global camera's extrinsic parameters and transform this information into the ultrasound robot's coordinate system. Accurate calibration of the ultrasonic robot system is one of the important factors affecting the accuracy of ultrasonic robot ultrasound scanning. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a multi-target calibration method for an ultrasonic robot to achieve accurate calibration of the ultrasonic robot.

[0004] To achieve the above objectives, an embodiment of the present application provides a multi-target calibration method for an ultrasonic robot, wherein the ultrasonic robot includes a multi-degree-of-freedom robotic arm, an ultrasonic probe, and a global camera; the method includes the following steps:

[0005] Constructing a forward kinematics model of the robotic arm;

[0006] Obtaining a first transformation matrix between the base of the robotic arm and the end of the robotic arm according to the forward kinematics model;

[0007] Obtaining a second transformation matrix from the global camera to the marker of the robotic arm according to image calibration obtained by the global camera;

[0008] Determine a third transformation matrix between the end of the robotic arm and the marker of the robotic arm by using inherent characteristics of the ultrasound probe and the needle;

[0009] The global camera is used to detect the homogeneous transformation matrix of the QR code on the ultrasound probe, and then the fourth transformation matrix between the marker of the robotic arm and the ultrasound probe and the fifth transformation matrix between the global camera and the base of the robotic arm are calibrated.

[0010] In some embodiments, obtaining a second transformation matrix from the global camera to the marker of the robotic arm based on the image calibration obtained by the global camera comprises the following steps:

[0011] Using the global camera to shoot the QR code;

[0012] The information of the four corner points on the two-dimensional code is extracted and solved using the PNP algorithm to obtain the second transformation matrix.

[0013] In some embodiments, determining a third transformation matrix between the end of the robotic arm and the marker of the robotic arm by using the inherent characteristics of the ultrasound probe and the needle comprises the following steps:

[0014] Define the needle coordinate system;

[0015] detecting different coordinates of the needle in the needle coordinate system and the ultrasound probe coordinate system to obtain two different sets of coordinate points;

[0016] A calibration equation is constructed according to the two different sets of coordinate points and solved to obtain the third transformation matrix.

[0017] In some embodiments, constructing a calibration equation based on two different sets of coordinate points and solving the equation to obtain the third transformation matrix comprises the following steps:

[0018] The first calibration equation defining the third transformation matrix is as follows:

[0019] pin p= pin T base base T end end T img img p;

[0020] in, pin p is the coordinate of the needle in the needle coordinate system; pin T base is the transformation matrix from the needle to the base of the robotic arm, base T end is the transformation matrix between the base of the manipulator and the end of the manipulator, end T imgis the transformation matrix from the end of the robotic arm to the ultrasound probe, img p is the coordinate of the needle in the ultrasound probe coordinate system;

[0021] The unconstrained nonlinear optimization problem is defined as follows:

[0022]

[0023] Among them, S(x) is the first objective function, L i (x) is the error function of the i-th group of measured posture relationships, x is the first optimization variable, (α x β y γ z ) respectively represent end T img The xyz three-axis Euler angle corresponding to the rotation matrix is: end t img express end T img The translation vector of pin t base express pin T base The translation vector, s x represents the internal parameter of the ultrasonic probe, x0 represents the initial value of the first optimization variable, represents the initial value of the optimization problem;

[0024] Obtaining a first optimal solution by iteratively solving the nonlinear optimization problem;

[0025] The closed-form solution of the first calibration equation is calibrated according to the first optimal solution.

[0026] In some embodiments, the step of calibrating the fifth transformation matrix between the global camera and the base of the robotic arm comprises the following steps:

[0027] defining a second calibration equation for the fifth transformation matrix;

[0028] Decoupling the second calibration equation to obtain a third calibration equation;

[0029] Formulate unconstrained optimization problems;

[0030] Obtaining the second optimal solution of the unconstrained optimization problem by solving the related lemma of the generalized inverse matrix;

[0031] A closed-form solution to the third calibration equation is obtained based on the second optimal solution.

[0032] In some embodiments, the second calibration equation defining the fifth transformation matrix is:

[0033] base T end end T tag = base T gc gc T tag ;

[0034] in, end T tag is the transformation matrix from the end of the manipulator to the marker of the manipulator, base T gc is the transformation matrix between the base of the manipulator and the global camera, gc T tag is the transformation matrix from the global camera to the marker of the robotic arm;

[0035] Let T A = base T end 、T X = end T tag 、T Y = base T gc 、T B = gc T tag , and use the global camera to obtain M groups of measurement data to obtain T Ai 、T Bi (i=1,2,...,M);

[0036] Then the second calibration equation is converted into:

[0037] T Ai T X =T Y T Bi ;

[0038] The converted second calibration equation is decoupled to obtain the third calibration equation as follows:

[0039] R Ai R X =R Y R Bi ;

[0040] R Ai t X +t Ai =R Y t Bi +t Y ;

[0041] Among them, R Ai and t AiRespectively represent T Ai The rotation matrix and translation vector, R Bi and t Bi Respectively represent T Bi The rotation matrix and translation vector, R X and t X Respectively represent T X The rotation matrix and translation vector, R Y and t Y Respectively represent T Y The rotation matrix and translation vector of ;

[0042] The unconstrained optimization problem is constructed as:

[0043]

[0044] Wherein, w is the second optimization variable, D is the second optimization coefficient, and d is the second optimization parameter;

[0045] According to the relevant lemma of the generalized inverse matrix, the second optimal solution of the unconstrained optimization problem is obtained as follows:

[0046] w * =(D T D) -1 D T d.

[0047] In some embodiments, the method further comprises the following steps:

[0048] Global positioning, trajectory planning and motion control are performed based on the calibrated ultrasonic robot.

[0049] To achieve the above objectives, another aspect of the present application provides a multi-target calibration device for an ultrasonic robot, the device comprising:

[0050] A model building unit, used for building a forward kinematics model of the robotic arm;

[0051] A first calibration unit is configured to obtain a first transformation matrix between a base of the manipulator and a tip of the manipulator by solving the forward kinematics model;

[0052] A second calibration unit, configured to obtain a second transformation matrix from the global camera to the marker of the robotic arm according to the image calibration obtained by the global camera;

[0053] a third calibration unit, configured to determine a third transformation matrix between the end of the robotic arm and the marker of the robotic arm by using inherent characteristics of the ultrasonic probe and the needle;

[0054] The fourth calibration unit is used to use the global camera to detect the homogeneous transformation matrix of the QR code on the ultrasound probe, and then calibrate the fourth transformation matrix between the marker of the robotic arm and the ultrasound probe and the fifth transformation matrix between the global camera and the base of the robotic arm.

[0055] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0056] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above-mentioned method when executed by a processor.

[0057] The embodiments of the present application include at least the following beneficial effects:

[0058] This application can construct a forward kinematic model of the robotic arm; solve the forward kinematic model to obtain a first transformation matrix between the robotic arm's base and the robotic arm's tip; calibrate the image captured by the global camera to obtain a second transformation matrix between the global camera and the robotic arm's marker; determine the third transformation matrix between the robotic arm's tip and the robotic arm's marker by using the inherent characteristics of the ultrasonic probe and the needle; use the global camera to detect the homogeneous transformation matrix of the QR code on the ultrasonic probe, and then calibrate the fourth transformation matrix between the robotic arm's marker and the ultrasonic probe, and the fifth transformation matrix between the global camera and the robotic arm's base. This application can improve the accuracy of ultrasonic scanning by performing multi-target calibration on the ultrasonic robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0060] Figure 1 A schematic diagram of a flow chart of a multi-target calibration method for an ultrasonic robot provided in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of the DH coordinate system of the ultrasonic manipulator provided in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of the coordinate system of the ultrasonic robot provided in an embodiment of the present application;

[0063] Figure 4Schematic diagram of an ultrasound probe calibration system provided in an embodiment of the present application;

[0064] Figure 5 A schematic structural diagram of a multi-target calibration device for an ultrasonic robot provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0067] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0068] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0070] Reference Figure 1 The embodiment of the present application provides a multi-target calibration method for an ultrasonic robot, wherein the ultrasonic robot includes a multi-degree-of-freedom robotic arm, an ultrasonic probe, and a global camera. The method may include, but is not limited to, steps S100 to S140, as follows:

[0071] S100: Constructing a forward kinematics model of the robotic arm;

[0072] S110: Obtaining a first transformation matrix between the base of the robotic arm and the end of the robotic arm according to the forward kinematics model;

[0073] S120: Obtaining a second transformation matrix from the global camera to the marker of the robotic arm according to the image calibration obtained by the global camera;

[0074] S130: Obtaining a third transformation matrix between the end of the robotic arm and the marker of the robotic arm by determining the inherent characteristics of the ultrasonic probe and the needle;

[0075] S140: Use the global camera to detect the homogeneous transformation matrix of the QR code on the ultrasound probe, and then calibrate the fourth transformation matrix between the marker of the robotic arm and the ultrasound probe and the fifth transformation matrix between the global camera and the base of the robotic arm.

[0076] Optionally, the obtaining of a second transformation matrix from the global camera to the marker of the robotic arm by calibrating the image captured by the global camera comprises the following steps:

[0077] Using the global camera to shoot the QR code;

[0078] The information of the four corner points on the two-dimensional code is extracted and solved using the PNP algorithm to obtain the second transformation matrix.

[0079] Optionally, determining a third transformation matrix between the end of the robotic arm and the marker of the robotic arm by using inherent characteristics of the ultrasound probe and the needle comprises the following steps:

[0080] Define the needle coordinate system;

[0081] detecting different coordinates of the needle in the needle coordinate system and the ultrasound probe coordinate system to obtain two different sets of coordinate points;

[0082] A calibration equation is constructed according to the two different sets of coordinate points and solved to obtain the third transformation matrix.

[0083] Optionally, constructing a calibration equation according to the two different sets of coordinate points and solving the equation to obtain the third transformation matrix comprises the following steps:

[0084] The first calibration equation defining the third transformation matrix is as follows:

[0085] pin p= pin T base base Tend end T img img p;

[0086] in, pin p is the coordinate of the needle in the needle coordinate system; pin T base is the transformation matrix from the needle to the base of the robotic arm, base T end is the transformation matrix between the base of the manipulator and the end of the manipulator, end T img is the transformation matrix from the end of the robotic arm to the ultrasound probe, img p is the coordinate of the needle in the ultrasound probe coordinate system;

[0087] The unconstrained nonlinear optimization problem is defined as follows:

[0088]

[0089] Among them, S(x) is the first objective function, L i (x) is the error function of the i-th group of measured posture relationships, x is the first optimization variable, (α x β y γ z ) respectively represent end T img The xyz three-axis Euler angle corresponding to the rotation matrix is: end t img express end T img The translation vector of pin t base express pin T base The translation vector, s x represents the internal parameter of the ultrasonic probe, x0 represents the initial value of the first optimization variable, represents the initial value of the optimization problem;

[0090] Obtaining a first optimal solution by iteratively solving the nonlinear optimization problem;

[0091] The closed-form solution of the first calibration equation is calibrated according to the first optimal solution.

[0092] Optionally, the step of calibrating the fifth transformation matrix between the global camera and the base of the robotic arm comprises the following steps:

[0093] defining a second calibration equation for the fifth transformation matrix;

[0094] Decoupling the second calibration equation to obtain a third calibration equation;

[0095] Formulate unconstrained optimization problems;

[0096] Obtaining the second optimal solution of the unconstrained optimization problem by solving the related lemma of the generalized inverse matrix;

[0097] A closed-form solution to the third calibration equation is obtained based on the second optimal solution.

[0098] Optionally, the second calibration equation defining the fifth transformation matrix is:

[0099] base T end end T tag = base T gc gc T tag ;

[0100] in, end T tag is the transformation matrix from the end of the manipulator to the marker of the manipulator, base T gc is the transformation matrix between the base of the manipulator and the global camera, gc T tag is the transformation matrix from the global camera to the marker of the robotic arm;

[0101] Let T A = base T end 、T X = end T tag 、T Y = base T gc 、T B = gc T tag , and use the global camera to obtain M groups of measurement data to obtain T Ai 、T Bi (i=1,2,...,M);

[0102] Then the second calibration equation is converted into:

[0103] T Ai T X =T Y T Bi ;

[0104] The converted second calibration equation is decoupled to obtain the third calibration equation as follows:

[0105] R Ai R X =R Y R Bi ;

[0106] R Ai t X +t Ai =R Y t Bi +t Y ;

[0107] Among them, R Ai and t Ai Represents T Ai The rotation matrix and translation vector, R Bi and t Bi Represents T Bi The rotation matrix and translation vector, R X and t X Represents T X The rotation matrix and translation vector, R Y and t Y Represents T Y The rotation matrix and translation vector of ;

[0108] The unconstrained optimization problem is constructed as:

[0109]

[0110] Wherein, w is the second optimization variable, D is the second optimization coefficient, and d is the second optimization parameter;

[0111] According to the relevant lemma of the generalized inverse matrix, the second optimal solution of the unconstrained optimization problem is obtained as follows:

[0112] w * =(D T D) -1 D T d.

[0113] Optionally, the method further comprises the following steps:

[0114] Global positioning, trajectory planning and motion control are performed based on the calibrated ultrasonic robot.

[0115] Next, the solution of the embodiment of the present application will be introduced and explained in detail with reference to specific application examples.

[0116] This embodiment discloses a multi-target calibration method for an ultrasonic robot. Ultrasonic robots can gradually replace ultrasound physicians to autonomously complete ultrasonic scanning operations, while processing collected ultrasonic images to assist physicians in diagnosis and treatment. The ultrasonic robot consists of an ultrasonic manipulator, an ultrasonic device, a force sensor, and a global camera. The parameters of the ultrasonic probe and the global camera need to be calibrated to improve the accuracy of the ultrasonic robot's three-dimensional reconstruction of ultrasonic images, trajectory planning, and control. The method of this embodiment is mainly based on the kinematic model of the ultrasonic robot. Through a two-step multi-target calibration method, the kinematic calibration of the ultrasonic robot is achieved. The first step is the calibration of the ultrasonic probe, and the second step is the calibration of the global camera.

[0117] Specifically, this embodiment includes the following technical solutions:

[0118] 1. Kinematic model of the robotic arm.

[0119] The kinematic model of the robotic arm describes the relationship between the position and joints of the end effector of the robotic arm. This embodiment uses the Denavit-Hartenberg (DH) method to derive the forward kinematic model of the ultrasonic robotic arm to describe the position and posture of the end effector of the robotic arm in space. The ultrasonic robotic arm used in this embodiment has 6 degrees of freedom, and 6 DH coordinate systems are established at the joints, such as Figure 2 As shown, where {0}:{6} represents the DH coordinate system of the 6-DOF manipulator, {d i}(i=1,4,6) is the joint offset, and {a3} is the connecting rod length.

[0120] Assume that the joint angles of the 6-DOF manipulator are θ = [θ1,θ2,...,θ6,] T , connecting rod torsion angle α=[α1,α2,...,α6,] T ,According to the chain rule, the forward kinematic model of the ultrasonic manipulator is as follows:

[0121] 0 T6(θ1,θ2,...,θ6)= 0 T1 1 T2L 5 T6 (1)

[0122] in, i-1 T i is the homogeneous transformation matrix between two adjacent coordinate systems, expressed as:

[0123]

[0124] Define J6 as the Jacobian matrix between the ultrasonic manipulator joint space and the operation space, P6, They are 0The displacement vector and rotation angle of T6, the velocity level kinematic model of the ultrasonic manipulator is as follows:

[0125]

[0126] in, It is the pseudo-reversal of J6.

[0127] 2. System kinematic model.

[0128] The ultrasonic robot system consists of a robotic arm, an ultrasonic probe, a probe connector, a six-dimensional force sensor, and a global camera. The robotic arm is a six-degree-of-freedom robotic arm. The ultrasonic probe is a linear probe used for imaging shallow tissues and blood vessels. At the same time, the imaging mode of the ultrasonic probe is B mode, which is the most common two-dimensional imaging mode. It converts the echo intensity of the ultrasonic wave emitted by the ultrasonic probe into a grayscale image. The probe connector is obtained through 3D printing. The global camera is an RGB-D camera, which is used to detect the position of the end actuator of the robotic arm and reconstruct the surface of the human tissue to be scanned.

[0129] The kinematic modeling of the entire ultrasonic robot system is carried out, such as Figure 3 As shown in the figure, {base} is the base coordinate system of the robot, {end} is the end coordinate system of the robot, {tag} is the tag coordinate system of the robot, {img} is the image coordinate system of the ultrasound probe, and {gc} is the coordinate system of the global camera. According to the forward kinematics model of the manipulator, the first transformation matrix can be solved in real time, that is, base T end , the QR code is photographed by a global camera, the information of the four corner points of the QR code is extracted and solved using the PNP algorithm, so that the second transformation matrix can be obtained in real time, that is, gc T tag ,and end T tag , tag T img , gc T base are the third transformation matrix, the fourth transformation matrix, and the fifth transformation matrix, respectively, and are the matrices to be calibrated. end T img = end T tag tag T img , end T img It needs to be obtained through ultrasonic probe calibration method. end T tag , gc T base It needs to be obtained through global camera calibration method.

[0130] 3. Multi-target calibration method.

[0131] 3.1. Ultrasonic probe calibration method.

[0132] In order to accurately obtain the pose information of each frame of ultrasound image, it is necessary to calibrate the transformation relationship between the image coordinate system {img} where the ultrasound image obtained by the ultrasound probe is located and the coordinate system {base} where the robot is located. In this way, the pose information of each frame of ultrasound image can be converted from the coordinate system {img} to the coordinate system {base}, and the absolute pose information of each frame of ultrasound image can be obtained, which provides the necessary input for subsequent three-dimensional reconstruction of the image, and then a more accurate human tissue structure can be reconstructed, providing necessary guidance for applications such as navigation surgery, interventional treatment and three-dimensional reconstruction.

[0133] This embodiment uses a needle detection method to calibrate the ultrasound probe. This calibration method is based on the inherent characteristics of the needle and ultrasound probe. First, the needle typically has clear geometric features, and its tip can be used as a single point feature, making it suitable for high-precision spatial positioning. Second, ultrasound probe imaging is based on the four processes of ultrasound transmission, propagation, reflection, and reception. Different tissue structures in the human body have different acoustic impedances and reflect ultrasound waves to varying degrees, producing different echo intensities. Ultimately, these appear as different grayscale values in the ultrasound image, allowing different tissue structures to be distinguished. The needle reflects ultrasound waves more strongly, appearing clearly and easily identifiable in the ultrasound image. The grayscale value of the needle portion in the grayscale ultrasound image is close to 255, facilitating the extraction of its pixel coordinates and achieving more accurate calibration. Finally, it should be noted that since ultrasound waves are completely reflected in air, the ultrasound probe cannot detect the needle in an air environment. Therefore, it is necessary to fix the needle in a water-filled calibration mold and reduce the amount of residual air in the water to allow the ultrasound probe to detect the needle in a water environment.

[0134] Ultrasonic probe calibration system Figure 4 As shown in the figure, {pin} is a custom needle coordinate system, and the origin of the {pin} coordinate system is set at the needle. By detecting the different coordinates of the needle in the {pin} coordinate system and the {img} coordinate system, two different sets of coordinate points are obtained, which can realize the solution of the subsequent calibration equation. Since the imaging mode of the ultrasound probe is a two-dimensional imaging mode, the needle coordinates obtained in the {pin} coordinate system are two-dimensional coordinates, and it is difficult to obtain a homogeneous transformation matrix, so it is difficult to solve the obtained img T pin ,This embodiment directly adopts the optimization method to solve the calibration equation.

[0135] The calibration equation is as follows:

[0136] pin p= pin T base base T end end T img img p (5)

[0137] in, pin p is the coordinate of the needle in the {pin} coordinate system. Since the origin of the {pin} coordinate system is located at the needle, the coordinate of the needle in the {pin} coordinate system is:

[0138] pin p=[0 0 0 1] T (6)

[0139] img p is the coordinate of the needle in the {img} coordinate system, expressed as:

[0140]

[0141] Because the ultrasound probe imaging mode is two-dimensional imaging, the ultrasound image does not have information on the z-axis. img p[2]=0; (uv) is the coordinate of the needle tip in the pixel coordinate system of the ultrasound probe. Since the image directly acquired by the ultrasound probe is based on the pixel coordinate system, it needs to be manually converted to the image coordinate system {img}. As the posture of the ultrasound manipulator changes, (uv) also changes accordingly; (s x s y ) is the internal parameter of the ultrasound probe, which can convert the pixel coordinate system of the ultrasound probe to the image coordinate system {img}. Because the imaging mode of the ultrasound probe is B mode, the ultrasound device can set the actual scanning depth d of the ultrasound image. img At the same time, the pixel height d of the ultrasound image is known pixel , then the internal reference s y It can be obtained, s y =d img / d pixel , and s x Need to be obtained through calibration.

[0142] In order to simplify the subsequent optimization and solution process and reduce the number of unknowns during optimization and solution, this embodiment sets the directions of the three axes x, y, and z of the {pin} coordinate system and the directions of the three axes of the {base} coordinate system to three pairs of identical vectors, namely but base T pin Expressed as:

[0143]

[0144] in,[ base t pinx , base t piny , base t pinz ] T The displacement vector corresponding to the {base} coordinate system to the {pin} coordinate system is an unknown quantity and needs to be calibrated and solved.

[0145] base T end is the homogeneous transformation matrix from the {base} coordinate system to the {end} coordinate system of the ultrasonic manipulator, which is obtained by solving the forward kinematics model of the manipulator. base T end It can be expressed as:

[0146]

[0147] in, base R end Corresponding to the rotation matrix from the {base} coordinate system to the {end} coordinate system, [ base t endx , base t endy , base t endz ] T The displacement vector corresponding to the {base} coordinate system to the {end} coordinate system.

[0148] end T img is the homogeneous transformation matrix from the ultrasonic manipulator coordinate system to the ultrasonic probe coordinate system, which needs to be calibrated and solved. end T img It can be expressed as:

[0149]

[0150] in, end R img Corresponding to the rotation matrix from the {end} coordinate system to the {img} coordinate system, [ end t imgx , end t imgy , end t imgz ] T The displacement vector corresponding to the {end} coordinate system to the {img} coordinate system.

[0151] According to equations (6)-(10), the original calibration equation (5) can be transformed into:

[0152]

[0153] in,base R end 、[ base t endx , base t endy , base t endz ] T 、s y , (uv) is a known quantity, [ base t pinx , base t piny , base t pinz ] T 、 end R img 、[ end t imgx , end t imgy , end t imgz ] T 、s x The quantity to be calibrated.

[0154] For the calibration equation (11), an iterative optimization algorithm is used to solve it. Equation (11) is transformed into:

[0155] Ay=b

[0156]

[0157] Among them, I 3×3 represents the third-order identity matrix, and N represents the number of images collected by the ultrasound probe during the calibration process, so we can get:

[0158] y0=A -1 b (13)

[0159] because end R img The norm of each column vector of is 1, so we can get s by solving the norm of y0[0] x ,and end R img The third column vector of end R img [:,0]× end R img [:,1] is obtained, × is the vector product, so we can get However, due to end R img is a rotation matrix, which needs to satisfy the three column vectors are orthogonal to each other, and end R img [:,0] and end R img[:,1] does not necessarily satisfy the orthogonal condition, so the obtained It can only be used as an initial value for subsequent iterative optimization solutions.

[0160] The rotation matrix to be calibrated end R img Converted into xyz three-axis Euler angle, that is end R img =eulerToRotation(α x β y γ z ), then the initial value Convert to as follows:

[0161]

[0162] Taking the first three rows of the matrix equation (11), we get L = 0. Taking L as the error function of the optimization problem, the optimization problem is defined as an unconstrained nonlinear optimization problem, that is:

[0163]

[0164] Since the F-norm of the matrix and the 2-norm of the vector are both strictly convex functions, problem (15) is a strictly convex optimization problem. At the same time, the error function L i (x) is differentiable, and the error function sum L = [L1(x) L2(x) LL N (x)] T .

[0165] Consider the first-order Taylor expansion of S(x):

[0166] S(x+Δx)≈S(x)+J T Δx(16)

[0167] Where J is the Jacobian matrix, expressed as:

[0168] J=[J1(x)J2(x)LJ N (x)] T (17)

[0169]

[0170] Find the extreme value of formula (16) We can get:

[0171] JJ T Δx=-JL (19)

[0172] For equation (19), introducing the damping factor λ, we get:

[0173] Δx=-(J T J+λI) -1 JL (20)

[0174] Assume k max Indicates the number of iterations, x k represents the optimization variable at time k, and the initial value of the iteration is x0. The update iteration formula of the target parameter x is:

[0175] x k+1 =x k +Δx (21)

[0176] Where k = 1, 2, ..., k max By k max By solving the optimization problem (15) by iterations, we can obtain the optimal solution x * , thus determining the closed-form solution of calibration equation (5) end T img and (s x s y ).

[0177] 3.2. Global camera calibration method.

[0178] In order to enable the global camera to accurately guide the ultrasonic robot to perform corresponding trajectory planning and control, it is necessary to calibrate the transformation relationship between the coordinate system {gc} where the global camera is located and the coordinate system {base} where the robot is located. This way, the target pose information obtained by the global camera can be converted to the coordinate system of the ultrasonic robot, guiding the ultrasonic robot to perform corresponding movements according to the target pose.

[0179] In this embodiment, a QR code is fixed on the ultrasonic probe fixture for global camera calibration. By detecting the homogeneous transformation matrix of the QR code in the {gc} coordinate system and the homogeneous transformation matrix of the end of the robotic arm in the {base} coordinate system, two different sets of transformation matrices are obtained, which can realize the solution of the subsequent calibration equation and obtain the target homogeneous transformation matrix. gc T base . Calibration system such as Figure 3 As shown, base T end The forward kinematics model of the ultrasonic manipulator can be solved. gc T tag The image based on the global camera can be solved, and gc T base 、 end T tag is the matrix to be calibrated, and the calibration equation is as follows:

[0180]

[0181] For the convenience of expression, let: T A = base T end 、T X = end T tag 、T Y = base T gc 、T B = gc T tag At the same time, the calibration of the global camera requires obtaining M sets of measurement data, namely T Ai 、T Bi (i=1,2,...,M). Therefore, the calibration equation (22) can be converted to:

[0182] T Ai T X =T Y T Bi (23) Because the homogeneous transformation matrix ref T cur It can be expressed as:

[0183]

[0184] Therefore, formula (23) can be decoupled as:

[0185] R Ai R X =R Y R Bi (25)

[0186] R Ai t X +t Ai =R Y t Bi +t Y (26)

[0187] First, the rotation matrix of the calibration equation (23) can be solved based on matrix vectorization and Kronecker product. Equation (25) can be vectorized as:

[0188] vec(R Ai R X )=vec(R Y R Bi )(27)

[0189] Where vec(·) represents the matrix vectorization operator.

[0190] And formula (27) can be converted into:

[0191]

[0192] Among them, I 3×3 represents the 3rd-order identity matrix, Represents the Kronecker product symbol of a matrix.

[0193] The matrix form of formula (28) can be expressed as:

[0194]

[0195] For M groups of measurement data, formula (29) can be expanded to:

[0196]

[0197] Since the measurement data generally has errors, Equation (30) is usually not true. In this case, it is necessary to optimize the z variable to minimize the error of the second norm of Bz. In addition, since the rotation matrix is an orthogonal matrix and z is composed of two vectorized rotation matrices, the second norm of z has an equality constraint, thus establishing a constrained optimization problem about z, namely:

[0198]

[0199] Defining variables and Therefore, problem (31) can be transformed into:

[0200]

[0201] Since the matrix is a symmetric matrix, so its Rayleigh Quotient can be expressed as:

[0202]

[0203] Define λ min and q min Respectively The minimum eigenvalue and the corresponding unit eigenvector, according to the relevant theorem of Rayleigh quotient, The minimum value of λ min , if and only if Therefore, the optimal solution to the optimization problem (32) is

[0204] Similar to the closed-form solution based on quaternions, the closed-form solution based on Kronecker products also has the problem of sign ambiguity, that is, the equation R Ai (-R X )=(-R Y )R Biis still valid. The difference is that the sign ambiguity of the former is reflected in the conversion of the rotation matrix of the measurement data to the quaternion, so the number of ambiguous cases is related to the amount of measurement data and shows an exponential growth trend. However, the sign ambiguity of the latter is reflected in the rotation matrix to be calibrated, so there are only two ambiguous cases, which is independent of the amount of measurement data. Since the determinant of the rotation matrix is equal to 1, according to the related properties of the determinant of the square matrix, the sign ambiguity problem can be solved by judging the determinant of the matrix. If the determinant of the rotation matrix is equal to 1, the closed solution of the calibration equation (25) is R X and R Y ; If the determinant of the rotation matrix is equal to -1, then the closed-form solution of the calibration equation (25) is -R X and -R Y .

[0205] Secondly, solve the translation vector of the calibration equation (23). The matrix form of equation (26) can be expressed as:

[0206]

[0207] For M groups of data, formula (34) can be expanded to:

[0208]

[0209] Similarly, according to formula (35), an unconstrained optimization problem about w is established, namely:

[0210]

[0211] According to the relevant lemma of generalized inverse matrix, the optimal solution of problem (36) is w * =(D T D) -1 D T d, and thus solve the closed-form solution of calibration equation (26) to obtain t X and t Y .

[0212] Reference Figure 5 The present application also provides a multi-target calibration device for an ultrasonic robot, which can implement the multi-target calibration method for an ultrasonic robot. The device includes:

[0213] A model building unit, used for building a forward kinematics model of the robotic arm;

[0214] A first calibration unit is configured to obtain a first transformation matrix between a base of the manipulator and a tip of the manipulator by solving the forward kinematics model;

[0215] A second calibration unit, configured to obtain a second transformation matrix from the global camera to the marker of the robotic arm according to the image calibration obtained by the global camera;

[0216] a third calibration unit, configured to determine a third transformation matrix between the end of the robotic arm and the marker of the robotic arm by using inherent characteristics of the ultrasonic probe and the needle;

[0217] The fourth calibration unit is used to use the global camera to detect the homogeneous transformation matrix of the QR code on the ultrasound probe, and then calibrate the fourth transformation matrix between the marker of the robotic arm and the ultrasound probe and the fifth transformation matrix between the global camera and the base of the robotic arm.

[0218] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0219] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of the present application. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0220] It can be understood that the contents of the above method embodiments are all applicable to the embodiments of the present device, the functions specifically implemented by the embodiments of the present device are the same as those of the method of the present application, and the beneficial effects achieved are also the same as those achieved by the method of the present application.

[0221] See also Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0222] The processor 601 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0223] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called by the processor 601 to execute the methods of the embodiments of this application.

[0224] Input / output interface 603, used to implement information input and output;

[0225] Communication interface 604, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0226] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0227] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0228] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of the present application is implemented.

[0229] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0230] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0231] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0232] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0233] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0234] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0235] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0236] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0237] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0238] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0239] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0240] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0241] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A multi-target calibration method for an ultrasonic robot, characterized in that: The ultrasonic robot includes a multi-degree-of-freedom robotic arm, an ultrasonic probe, and a global camera; and the method includes the following steps: Constructing a forward kinematic model of the robotic arm; Obtaining a first transformation matrix between the base of the robotic arm and the end of the robotic arm according to the forward kinematics model; Obtaining a second transformation matrix from the global camera to the marker of the robotic arm according to image calibration obtained by the global camera; Determine a third transformation matrix between the end of the robotic arm and the marker of the robotic arm by using inherent characteristics of the ultrasound probe and the needle; The global camera is used to detect the homogeneous transformation matrix of the QR code on the ultrasound probe, and then the fourth transformation matrix between the marker of the robotic arm and the ultrasound probe and the fifth transformation matrix between the global camera and the base of the robotic arm are calibrated.

2. The multi-target calibration method of an ultrasonic robot according to claim 1, characterized in that: The step of obtaining a second transformation matrix between the global camera and the mark of the robotic arm by calibrating the image captured by the global camera comprises the following steps: Using the global camera to shoot the QR code; The information of the four corner points on the two-dimensional code is extracted and solved using the PNP algorithm to obtain the second transformation matrix.

3. The multi-target calibration method of an ultrasonic robot according to claim 1, characterized in that: The determining of a third transformation matrix between the end of the robotic arm and the marker of the robotic arm by using the inherent characteristics of the ultrasonic probe and the needle comprises the following steps: Define the needle coordinate system; detecting different coordinates of the needle in the needle coordinate system and the ultrasound probe coordinate system to obtain two different sets of coordinate points; A calibration equation is constructed according to the two different sets of coordinate points and solved to obtain the third transformation matrix.

4. The multi-target calibration method of an ultrasonic robot according to claim 3, characterized in that: The step of constructing a calibration equation based on the two different sets of coordinate points and solving the equation to obtain the third transformation matrix comprises the following steps: The first calibration equation defining the third transformation matrix is as follows: pin p= pin T base base T end end T img img p; in, pin p is the coordinate of the needle in the needle coordinate system; pin T base is the transformation matrix from the needle to the base of the robotic arm, base T end is the transformation matrix between the base of the manipulator and the end of the manipulator, end T img is the transformation matrix from the end of the robotic arm to the ultrasound probe, img p is the coordinate of the needle in the ultrasound probe coordinate system; The unconstrained nonlinear optimization problem is defined as follows: Among them, S(x) is the first objective function, L i (x) is the error function of the i-th group of measured posture relationships, x is the first optimization variable, (α x β y γ z ) respectively represent end T img The xyz three-axis Euler angle corresponding to the rotation matrix is: end t img express end T img The translation vector of pin t base express pin T base The translation vector, s x represents the internal parameter of the ultrasonic probe, x0 represents the initial value of the first optimization variable, represents the initial value of the optimization problem; Obtaining a first optimal solution by iteratively solving the nonlinear optimization problem; The closed-form solution of the first calibration equation is calibrated according to the first optimal solution.

5. The multi-target calibration method of an ultrasonic robot according to claim 1, characterized in that: The step of calibrating the fifth transformation matrix between the global camera and the base of the robotic arm comprises the following steps: defining a second calibration equation for the fifth transformation matrix; Decoupling the second calibration equation to obtain a third calibration equation; Formulate unconstrained optimization problems; Obtaining the second optimal solution of the unconstrained optimization problem by solving the related lemma of the generalized inverse matrix; A closed-form solution to the third calibration equation is obtained based on the second optimal solution.

6. The multi-target calibration method of an ultrasonic robot according to claim 5, characterized in that: The second calibration equation defining the fifth transformation matrix is: base T end end T tag = base T gc gc T tag ; in, end T tag is the transformation matrix from the end of the manipulator to the marker of the manipulator, base T gc is the transformation matrix between the base of the manipulator and the global camera, gc T tag is the transformation matrix from the global camera to the marker of the robotic arm; Let T A = base T end 、T X = end T tag 、T Y = base T gc 、T B = gc T tag , and use the global camera to obtain M groups of measurement data to obtain T Ai 、T Bi (i=1,2,...,M); Then the second calibration equation is converted into: T Ai T X =T Y T Bi ; The converted second calibration equation is decoupled to obtain the third calibration equation as follows: R Ai R X =R Y R Bi ; R Ai t X +t Ai =R Y t Bi +t Y ; Among them, R Ai and t Ai Represents T Ai The rotation matrix and translation vector, R Bi and t Bi Represents T Bi The rotation matrix and translation vector, R X and t X Represents T X The rotation matrix and translation vector, R Y and t Y Represents T Y The rotation matrix and translation vector of ; The unconstrained optimization problem is constructed as: Wherein, w is the second optimization variable, D is the second optimization coefficient, and d is the second optimization parameter; According to the relevant lemma of the generalized inverse matrix, the second optimal solution of the unconstrained optimization problem is obtained as follows: w * =(D T D) -1 D T d。 7. The multi-target calibration method of an ultrasonic robot according to any one of claims 1 to 6, characterized in that: The method further comprises the following steps: Global positioning, trajectory planning and motion control are performed based on the calibrated ultrasonic robot.

8. A multi-target calibration device for an ultrasonic robot, characterized in that: The device comprises: A model building unit, used for building a forward kinematics model of the robotic arm; A first calibration unit is configured to obtain a first transformation matrix between a base of the manipulator and a tip of the manipulator by solving the forward kinematics model; A second calibration unit, configured to obtain a second transformation matrix from the global camera to the marker of the robotic arm according to the image calibration obtained by the global camera; a third calibration unit, configured to determine a third transformation matrix between the end of the robotic arm and the marker of the robotic arm by using inherent characteristics of the ultrasonic probe and the needle; The fourth calibration unit is used to use the global camera to detect the homogeneous transformation matrix of the QR code on the ultrasound probe, and then calibrate the fourth transformation matrix between the marker of the robotic arm and the ultrasound probe and the fifth transformation matrix between the global camera and the base of the robotic arm.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images