A hand-eye calibration method, device and computer equipment

Abstract

This application relates to the field of hand-eye vision technology. Specifically, it relates to a hand-eye calibration method, device, and computer device, which can, to a certain extent, solve the problem of low accuracy of the rigid body transformation matrix obtained by solving the hand-eye calibration equation using a linear solution method. The hand-eye calibration method includes: determining the hand-eye calibration equation of the hand-eye vision system; performing a linear solution on the hand-eye calibration equation to obtain an initial rigid body transformation matrix between the camera coordinate system and the end coordinate system of the robot arm; establishing an initial objective function for non-linear iterative optimization of the initial rigid body transformation matrix; generating a target rigid body transformation matrix corresponding to the initial rigid body transformation matrix using the new objective function, where the target rigid body transformation matrix is used to transform the coordinates of the camera coordinate system into the end coordinate system of the robot arm, and the new objective function is obtained by performing non-linear iterative optimization of the initial objective function with the initial rigid body transformation matrix as the initial value.

Description

Technical Field

[0001] This application relates to the field of hand-eye vision technology, and in particular, to a hand-eye calibration method, apparatus, and computer device. Background Technique

[0002] In the field of machine vision, an intelligent manufacturing production line based on a hand-eye vision system can complete automated tasks such as picking, palletizing, welding, dispensing, and assembly. In order for the robot arm to operate according to the spatial pose recognized by the camera, it is necessary to know the pose relationship between the camera and the end of the robot arm. Hand-eye calibration is to determine the rigid body transformation matrix between the coordinate system of the end of the robot arm and the camera coordinate system fixed on the end of the robot arm.

[0003] During the hand-eye calibration process, a hand-eye calibration equation is first established, and then the hand-eye calibration equation is solved. When solving the hand-eye calibration equation, generally, a linear solution method is first used to solve the rotation matrix, and then the rotation matrix is substituted into the hand-eye calibration equation to solve the translation vector. The rotation matrix and the translation vector represent the rigid body transformation matrix between the coordinate system of the end of the robot arm and the camera coordinate system.

[0004] Since the linear solution method is vulnerable to noise, it must hold strictly under the condition of no noise interference. However, in the actual hand-eye calibration process, it is inevitable that there is noise in the observed data, such as the motion error of the robot itself, the imaging error of the camera, etc. This noise will cause the accuracy of the rotation matrix solved by the linearization method to decrease; at the same time, substituting the rotation matrix parameters with low accuracy into the hand-eye calibration equation to continue solving the translation vector will cause the estimation error of the rotation matrix parameters to be transmitted to the translation vector, further affecting the accuracy of the translation vector. Therefore, due to the vulnerability of the linear solution method to noise, when using the linear solution method to solve the hand-eye calibration equation, there is a problem of low accuracy of the solved rigid body transformation matrix. Summary of the Invention

[0005] In order to solve the problem of low accuracy of the solved rigid body transformation matrix when using the linear solution method to solve the hand-eye calibration equation, this application provides a hand-eye calibration method, apparatus, and computer device.

[0006] The embodiments of this application are implemented as follows:

[0007] The embodiments of this application provide a hand-eye calibration method, and the method includes:

[0008] Determine the hand-eye calibration equation of the hand-eye vision system, where the hand-eye vision system includes a camera and the end of the robot arm, the camera is installed at the end of the robot arm, and the hand-eye calibration equation is used to solve the rigid body transformation matrix between the camera coordinate system and the coordinate system of the end of the robot arm;

[0009] Perform a linear solution to the hand-eye calibration equation to obtain an initial rigid body transformation matrix between the camera coordinate system and the end coordinate system of the robot arm;

[0010] Establish an initial objective function for non-linear iterative optimization of the initial rigid body transformation matrix, where the initial objective function is used to characterize the relationship between the rigid body transformation matrix to be solved and the error of the rigid body transformation matrix to be solved;

[0011] Generate a target rigid body transformation matrix corresponding to the initial rigid body transformation matrix using the new objective function, where the target rigid body transformation matrix is used to transform the coordinates of the camera coordinate system into the end coordinate system of the robot arm, and the new objective function is obtained by non-linear iterative optimization of the initial objective function with the initial rigid body transformation matrix as the initial value.

[0012] In some embodiments, the generating a target rigid body transformation matrix corresponding to the initial rigid body transformation matrix using the new objective function further includes:

[0013] Initialize the parameters included in the initial objective function using the initial rigid body transformation matrix to obtain initialization parameters;

[0014] Substitute the initialization parameters into the initial objective function to obtain a first new objective function;

[0015] Determine a first optimized rigid body transformation matrix based on the first new objective function, and calculate a first error of the first optimized rigid body transformation matrix;

[0016] Update the parameters included in the first new objective function based on the first optimized rigid body transformation matrix to obtain updated parameters;

[0017] Substitute the updated parameters into the first new objective function to obtain a second new objective function;

[0018] Determine a second optimized rigid body transformation matrix based on the second new objective function, and calculate a second error of the second optimized rigid body transformation matrix;

[0019] When the difference between the first error and the second error is less than a threshold and the number of iterations is less than the maximum number of iterations, stop the iteration and output the second optimized rigid body transformation matrix, where the second optimized rigid body transformation matrix is the target transformation matrix.

[0020] In some embodiments, when the difference between the first error and the second error is greater than the threshold and the number of iterations is less than the maximum number of iterations, the parameters included in the second new objective function are updated based on the second optimized rigid body transformation matrix to obtain a corresponding new objective function, and then the corresponding optimized rigid body transformation matrix is determined based on the new objective function until the condition that the difference between the errors corresponding to the optimized rigid body transformation matrices obtained in two adjacent iterations is less than the threshold or the number of iterations reaches the maximum number of iterations is satisfied, and the optimized rigid body transformation matrix obtained in the last iteration is output and used as the target rigid body transformation matrix.

[0021] In some embodiments, the determining the hand-eye calibration equation of the hand-eye vision system further includes:

[0022] Establish a camera coordinate system, a robot arm end coordinate system, a robot base world reference coordinate system, and a calibration board coordinate system;

[0023] Determine the rigid body transformation matrix between the robot arm end coordinate system and the robot base world reference coordinate system, the rigid body transformation matrix between the camera coordinate system and the calibration board coordinate system, the rigid body transformation matrix between the calibration board coordinate system and the robot base world reference coordinate system, and the rigid body transformation matrix between the robot arm end coordinate system to be calibrated and the camera coordinate system;

[0024] Obtain the pose information of the robot during multiple movements, and determine the hand-eye calibration equation based on the pose information and the rigid body transformation matrix.

[0025] In some embodiments, the determining the rigid body transformation matrix between the robot arm end coordinate system and the robot base world reference coordinate system further includes:

[0026] Read the rigid body transformation matrix between the robot arm end coordinate system and the robot base world reference coordinate system from the robot control panel.

[0027] In some embodiments, the determining the rigid body transformation matrix between the camera coordinate system and the calibration board coordinate system further includes:

[0028] Obtain the standard point cloud data of the calibration board;

[0029] Collect the actual point cloud data of the calibration board through the camera;

[0030] Register the standard point cloud data and the actual point cloud data, and after registration, obtain the rigid body transformation matrix between the camera coordinate system and the calibration board coordinate system.

[0031] In some embodiments, the initial rigid body transformation matrix includes an initial rotation matrix equation and an initial translation vector. The linear solution of the hand-eye calibration equation to obtain the initial rigid body transformation matrix between the camera coordinate system and the end coordinate system of the robotic arm further includes:

[0032] Convert the hand-eye calibration equation into a homogeneous hand-eye calibration matrix equation;

[0033] Convert the homogeneous hand-eye calibration matrix into a rotation matrix equation and a translation vector equation;

[0034] Solve the rotation matrix equation using the least squares method to obtain the initial rotation matrix;

[0035] Substitute the initial rotation matrix into the translation vector equation and solve it using the least squares method to obtain the initial translation vector.

[0036] Another aspect of the present application provides a hand-eye calibration device, including:

[0037] A determination module for determining the hand-eye calibration equation of the hand-eye vision system. The hand-eye vision system includes a camera and the end of the robotic arm. The camera is installed at the end of the robotic arm. The hand-eye calibration equation is used to solve the rigid body transformation matrix between the camera coordinate system and the end coordinate system of the robotic arm;

[0038] A linear solution module for linearly solving the hand-eye calibration equation to obtain the initial rigid body transformation matrix between the camera coordinate system and the end coordinate system of the robotic arm;

[0039] An initial objective function establishment module for establishing an initial objective function for non-linear iterative optimization of the initial rigid body transformation matrix. The initial objective function is used to characterize the relationship between the rigid body transformation matrix to be solved and the error of the rigid body transformation matrix to be solved;

[0040] A non-linear iterative optimization module for generating a target rigid body transformation matrix corresponding to the initial rigid body transformation matrix using a new objective function. The target rigid body transformation matrix is used to transform the coordinates of the camera coordinate system into the end coordinate system of the robotic arm. The new objective function is obtained by non-linear iterative optimization of the initial objective function with the initial rigid body transformation matrix as the initial value.

[0041] Another aspect of the present application provides a computer device, including a memory and a processor. The memory stores a computer program. The processor, when executing the computer program, implements the steps of the above-mentioned hand-eye calibration method.

[0042] In another aspect of the present application, there is provided a computer-readable storage medium storing instructions which, when run on a computer, cause the computer to execute the above hand-eye calibration method.

[0043] Advantages of the present application: Determine the hand-eye calibration equation, solve the hand-eye calibration equation by the linear solution method to determine the initial rigid body transformation matrix, then use the initial objective function to perform non-linear iterative optimization on the initial rigid body transformation matrix, and finally use the globally optimal optimized rigid body transformation matrix found by non-linear iteration as the target rigid body transformation matrix. The target rigid body transformation matrix found by non-linear iteration is more accurate than the initial rigid body transformation matrix, achieving the purpose of improving the accuracy of solving the hand-eye calibration equation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 It shows a schematic flow chart of the hand-eye calibration method according to an embodiment of the present application;

[0046] Figure 2 It shows a schematic diagram of the positional relationship and coordinate system relationship between the components in the hand-eye vision system;

[0047] Figure 3 It shows a schematic flow chart of determining the hand-eye calibration equation of the hand-eye vision system according to another embodiment of the present application;

[0048] Figure 4 It shows a schematic flow chart of determining the rigid body transformation matrix between the camera coordinate system and the calibration plate coordinate system according to another embodiment of the present application;

[0049] Figure 5 It shows a schematic flow chart of linearly solving the hand-eye calibration equation to obtain the initial rigid body transformation matrix according to another embodiment of the present application;

[0050] Figure 6 It shows a schematic flow chart of generating the target rigid body transformation matrix corresponding to the initial rigid body transformation matrix using the new objective function;

[0051] Figure 7 It shows a structural block diagram of the hand-eye calibration device of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] To make the objectives, embodiments, and advantages of this application clearer, the following will clearly and completely describe the exemplary embodiments of this application with reference to the accompanying drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only a part of the embodiments of this application, rather than all of the embodiments.

[0053] It should be noted that the brief description of the terms in this application is only for the convenience of understanding the embodiments described next, rather than intending to limit the embodiments of this application. Unless otherwise specified, these terms should be understood in their ordinary and general meanings.

[0054] The terms "first", "second", "third", etc. in the description, claims, and the above accompanying drawings of this application are used to distinguish similar or homogeneous objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that such terms can be interchanged under appropriate circumstances.

[0055] The terms "comprising" and "having" and any variations thereof are intended to cover but not exclusively include. For example, a product or device comprising a series of components does not necessarily have to be limited to all the components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0056] The terms "installed" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected, or indirectly connected through an intermediate medium, and can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0057] The vision system of a robot is divided into a fixed-scene vision system and a moving hand-eye vision system. Among them, the hand-eye vision system is composed of a camera (or video camera) and the end of the robot arm.

[0058] According to the different relative positions between the camera (or video camera) and the robot, the hand-eye vision system is divided into the Eye-in-Hand system and the Eye-to-Hand system. In the Eye-in-Hand system, the camera (or video camera) is installed at the end of the robot arm (end-effector) and can move with the robot during the operation of the robot. While in the Eye-to-Hand system, the camera (or video camera) is installed at a fixed position outside the robot body and does not move with the robot during the operation of the robot.

[0059] In order for the end of the robotic arm to operate according to the spatial pose recognized by the camera (or video camera), it is necessary to know the pose relationship between the camera and the end of the robotic arm. Hand-eye calibration is to determine the rigid body transformation matrix between the camera (or video camera) and the end of the robotic arm.

[0060] The hand-eye calibration method of this application calibrates the rigid body transformation matrix between the coordinate system of the end of the robotic arm and the coordinate system of the camera installed at the end of the robotic arm in the Eye-in-Hand system. By calibrating the rigid body transformation matrix, the pixel coordinates of the camera can be transformed into the spatial coordinate system of the manipulator, and then the motion parameters of each motor can be calculated in the coordinate system of the end of the robotic arm. Thus, after the manipulator reaches the specified position based on the motion parameters, the camera is controlled to take a picture of the specified position.

[0061] The camera in this application is a 2D camera or a 3D camera. Since the 3D camera can collect information in one more dimension than the 2D camera, it can handle more complex scenes and has better anti-noise ability. Therefore, a 3D camera is preferably used to capture data information; at the same time, using the data information captured by the 3D camera for hand-eye calibration can also improve the accuracy of hand-eye calibration.

[0062] Figure 1 The flowchart of the hand-eye calibration method according to the embodiment of this application is shown. As Figure 1 shown, the hand-eye calibration method includes the following steps:

[0063] In step 110, the hand-eye calibration equation of the hand-eye vision system is determined. The hand-eye vision system includes a camera and the end of the robotic arm, and the camera is installed at the end of the robotic arm. The hand-eye calibration equation is used to solve the rigid body transformation matrix between the camera coordinate system and the coordinate system of the end of the robotic arm.

[0064] Figure 2 The schematic diagram of the position and coordinate system relationship between the components in the hand-eye vision system is shown. As Figure 2 , the hand-eye vision system includes: a robotic arm, a robotic base, a 3D camera, and a calibration board, corresponding to the coordinate system Coord A of the end of the robotic arm, the world reference coordinate system Coord B of the robotic base, the camera coordinate system Coord C , and the calibration board coordinate system Coord T .

[0065] Figure 3 The flowchart of determining the hand-eye calibration equation of the hand-eye vision system according to another embodiment of this application is shown. As Figure 3 shown, determining the hand-eye calibration equation of the hand-eye vision system includes the following steps:

[0066] In step 310, a camera coordinate system, a coordinate system at the end of the robotic arm, a world reference coordinate system of the robotic base, and a calibration board coordinate system are established;

[0067] In step 320, a rigid body transformation matrix between the coordinate system at the end of the robotic arm and the world reference coordinate system of the robotic base is determined A rigid body transformation matrix between the calibration board coordinate system and the camera coordinate system A rigid body transformation matrix between the calibration board coordinate system and the world reference coordinate system of the robotic base And a rigid body transformation matrix between the camera coordinate system to be calibrated and the coordinate system at the end of the robotic arm

[0068] Control the robot to move in different spatial postures, and use the camera installed at the end of the robotic arm to photograph the calibration block. In each movement, directly read and record the rigid body transformation matrix between the coordinate system at the end of the robotic arm and the world reference coordinate system of the robotic base through the robot control panel

[0069] Figure 4 Fig. shows a schematic flow chart of determining the rigid body transformation matrix between the camera coordinate system and the calibration board coordinate system in another embodiment of the present application, as Figure 4 shown, determining the rigid body transformation matrix between the camera coordinate system and the calibration board coordinate system includes the following steps:

[0070] In step 410, standard point cloud data of the calibration board is obtained.

[0071] Point cloud is a large number of point sets that express the spatial distribution of the target and the surface characteristics of the target under the same spatial reference system. After obtaining the spatial coordinates of each sampling point on the object surface, what is obtained is a set of points, which is called point cloud.

[0072] In step 420, actual point cloud data of the calibration board is collected by the camera;

[0073] With the continuous breakthrough of computer vision technology and sensor technology, the method of generating object point cloud by laser scanning has been rapidly developed and improved. In this embodiment, actual point cloud data of the calibration board is obtained through devices such as depth cameras, binocular cameras, or 3D laser scanning cameras.

[0074] In step 430, the standard point cloud data and the actual point cloud data are registered, and after registration, a rigid body transformation matrix between the calibration board coordinate system and the camera coordinate system is obtained

[0075] In some embodiments, the standard point cloud data and the actual point cloud data are registered through the ICP registration algorithm. The ICP (Iterative Closest Point) algorithm is one of the most widely used 3D point cloud registration algorithms, which solves the rotation and translation matrix of the standard point cloud data and the actual point cloud data through Euclidean transformation.

[0076] In step 330, the pose information of the robot under multiple motions is obtained, and the hand-eye calibration equation is determined based on the pose information and the rigid body transformation matrix.

[0077] Let the hand-eye relationship to be solved, that is, the rigid body transformation matrix between the camera coordinate system and the end coordinate system of the robot arm, be According to the pose information under multiple motions and the conversion relationship between the calibration board and the world reference coordinate system of the robot base that remains constant

[0078] Since the calibration board is fixed in the world reference coordinate system of the robot base, the rigid body transformation matrix between the calibration board and the world reference coordinate system of the robot base remains constant, and the following relational expression can be established:

[0079]

[0080] Control the robot to take pictures of the calibration board in more than 2 different poses, and the following relational expression can be obtained:

[0081]

[0082] where n represents the number of poses;

[0083] Transforming formula (2) gives:

[0084]

[0085] Based on formula (3), a standard hand-eye calibration equation AX = XB is constructed, where

[0086]

[0087] In step 120, the hand-eye calibration equation is linearly solved to obtain the initial rigid body transformation matrix between the camera coordinate system and the end coordinate system of the robot arm.

[0088] The initial rigid body transformation matrix includes an initial rotation matrix equation and an initial translation vector.

[0089] Figure 5 Fig. shows the flow diagram of linearly solving the hand-eye calibration equation to obtain the initial rigid body transformation matrix in another embodiment of the present application, as Figure 5As shown, a linear solution is performed on the hand-eye calibration equation to obtain the initial rigid body transformation matrix between the camera coordinate system and the end coordinate system of the robotic arm, including the following steps:

[0090] In step 510, the hand-eye calibration equation is converted into a homogeneous hand-eye calibration matrix equation, and the homogeneous hand-eye calibration matrix equation is expressed as:

[0091]

[0092] where t A , t B , t X represent the translation matrices of A, B, and X respectively, and R A , R B , R X represent the rotation matrices of A, B, and X respectively.

[0093] In step 520, the homogeneous hand-eye calibration matrix is converted into a rotation matrix equation and a translation vector equation, and the rotation matrix equation and the translation vector equation are respectively:

[0094] R A R X =R X R B

[0095] R A t X +t A =R X t B +t X

[0096] where the rotation axes n A corresponding to the rotation matrix R B and the rotation axis n A and the rotation axis n B have the following relationship:

[0097] R X n B =n A

[0098] In step 530, the rotation matrix equation is solved using the least squares method to obtain the initial rotation matrix R X :

[0099]

[0100] where M is the covariance matrix, k is the total number of motion postures of the manipulator, T is the matrix transpose, n Bi , n Ai are the rotation axes of the rotation matrices A and B in the corresponding postures respectively;

[0101] In step 540, the initial rotation matrix is substituted into the translation vector equation and solved by the least squares method to obtain the initial translation vector.

[0102] The following relationship can be deduced from the translation vector equation:

[0103] (R A -I)t X =R X t B -t A ;

[0104] where I represents the identity matrix; substituting the initial rotation matrix R X , rotation matrix R B , and translation matrix t B into the above formula respectively, the initial translation vector t X . can be calculated.

[0105] In step 130, an initial objective function is established for non-linear iterative optimization of the initial rigid body transformation matrix, and the initial objective function is used to characterize the relationship between the rigid body transformation matrix to be solved and the error of the rigid body transformation matrix to be solved;

[0106] This application draws on the idea of the quasi-Newton non-linear optimization algorithm and proposes a non-linear iterative optimization algorithm for the hand-eye calibration equation based on quasi-Newton. Therefore, an initial objective function is established based on the initial rigid body transformation matrix, that is, an initial objective function is established based on the initial rotation matrix R X . and the initial translation vector t X . respectively. Specifically:

[0107] The rotation matrix objective function is expressed as:

[0108] min max||R Ai R X -R X R Bi ||2

[0109] st.pp T =1

[0110] p T p'=0

[0111] where p' is the conjugate of the quaternion, p is the quaternion representation of R X , R Ai represents the rotation matrix of matrix A corresponding to the hand-eye standard equation AX = XB, and R Bi represents the rotation matrix of matrix B corresponding to the hand-eye standard equation AX = XB in the i-th group of postures.

[0112] The translational vector objective function is expressed as: min max||R Ai t X +t A -R X t B +t X ||2;

[0113] The initial objective function established based on the quasi - Newton algorithm is expressed as:

[0114]

[0115] where x is the rigid - body transformation matrix to be solved, k represents the iteration number, ξ is the target residual value, and λ is the residual factor;

[0116] The error function therein is:

[0117] φ i (x k ,ξ k ) = |e i (x k )| - ξ k ,e i (x k ) = Ax k -b

[0118] where e i represents the error formula, A, x k and b are the components of the evolution of the e i formula. Here, e i (x k ) refers to the sum of the errors of the rotation matrix and the translation matrix, specifically expressed as:

[0119] ||R Ai R X -R X R Bi ||2 + ||R Ai t X +t A -R X t B +t X ||2.

[0120] Among them, I1 and I2 represent data set classification, specifically:

[0121] I1 = {all i fulfilling φ i (x k ,ξ k ) > 0, λ ki > 0}

[0122] I2 = {all i fulfilling φi (x k , ξ k ) > 0, λ ki = 0}.

[0123] In step 140, a target rigid body transformation matrix corresponding to the initial rigid body transformation matrix is generated using the new objective function, where the target rigid body transformation matrix is used to transform the coordinates in the camera coordinate system into the coordinate system at the end of the robotic arm. The new objective function is obtained by nonlinearly iteratively optimizing the initial objective function with the initial rigid body transformation matrix as the initial value.

[0124] Figure 6 The flowchart shows the process of generating the target rigid body transformation matrix corresponding to the initial rigid body transformation matrix using the new objective function. The new objective function includes at least a first new objective function and a second new objective function, and the optimized rigid body transformation matrix includes at least a first optimized rigid body transformation matrix and a second optimized rigid body transformation matrix. As Figure 6 shown, generating the target rigid body transformation matrix corresponding to the initial rigid body transformation matrix using the new objective function includes the following steps:

[0125] In step 610, the parameters included in the initial objective function are initialized using the initial rigid body transformation matrix to obtain the initialization parameters, and the initialization parameters are:

[0126] ξ k-1＝0 = 0, x k-1＝0 = X

[0127] For i = 0 to N: λ k-1i = 1

[0128] where i represents the equation number, there are a total of N equations, and k is the number of iterations.

[0129] In step 620, the initialization parameters are substituted into the initial objective function to obtain the first new objective function;

[0130] In step 630, a first optimized rigid body transformation matrix is determined based on the first new objective function, and the first error of the first optimized rigid body transformation matrix is calculated.

[0131] The error corresponding to the current iteration, that is, the first error, is expressed as:

[0132] E k = E k (x k ) = max|e i (x k )|, 1 ≤ i ≤ N.

[0133] In step 640, the parameters included in the first new objective function are updated based on the first optimized rigid body transformation matrix to obtain the updated parameters;

[0134] In some embodiments, the parameters included in the first new objective function are updated based on the following formula:

[0135]

[0136] In step 650, the updated parameters are substituted into the first new objective function to obtain a second new objective function;

[0137] In step 660, a second optimized rigid body transformation matrix is determined based on the second new objective function, and a second error of the second optimized rigid body transformation matrix is calculated.

[0138] In some embodiments, the second error is expressed as E K-1 =E K-1 (x)=max|e i-1 (x K )|, 1 ≤ i ≤ N.

[0139] In step 670, when the difference between the first error and the second error is less than the threshold and the number of iterations is less than the maximum number of iterations, the iteration is stopped and the second optimized rigid body transformation matrix is output, and the second optimized rigid body transformation matrix is the target transformation matrix.

[0140] In some embodiments, if |E K -E K-1 | < ε, the iteration is stopped, and the rigid body transformation matrix output at the (k - 1)th iteration is output. Where ε represents the threshold, and the value range is from 1e - 7 to 1e - 8.

[0141] After step 660, if the difference between the first error and the second error is greater than the threshold and the number of iterations is less than the maximum number of iterations, the parameters included in the second new objective function are updated based on the second optimized rigid body transformation matrix to obtain a corresponding new objective function, and then a corresponding optimized rigid body transformation matrix is determined based on the new objective function, until the condition that the difference between the errors corresponding to the optimized rigid body transformation matrices obtained in two adjacent iterations is less than the threshold or the number of iterations reaches the maximum number of iterations is satisfied, and the optimized rigid body transformation matrix obtained in the last iteration is output and used as the target rigid body transformation matrix.

[0142] In some embodiments, the non - linear iterative optimization method in step 140 can also be replaced by optimization algorithms such as Levenberg - Marquarelt (LM), gradient descent, etc. Using different non - linear optimization algorithms can obtain better results than linear closed - form solutions. However, compared with optimization algorithms such as Levenberg - Marquarelt and gradient descent, the non - linear optimization results obtained by the quasi - Newton non - linear iterative optimization method are better and more efficient.

[0143] In this application, a hand-eye calibration equation is determined, and the initial rigid body transformation matrix is determined by solving the hand-eye calibration equation through a linear solution method. Then, the initial rigid body transformation matrix is nonlinearly iteratively optimized using the initial objective function. Finally, the globally optimal optimized rigid body transformation matrix found through the nonlinear iteration is used as the target rigid body transformation matrix. The target rigid body transformation matrix found through the nonlinear iteration is more accurate than the initial rigid body transformation matrix, achieving the purpose of improving the accuracy of solving the hand-eye calibration equation.

[0144] Figure 7 The structural block diagram of the hand-eye calibration device of this application is shown, as Figure 7 shown, the hand-eye calibration device 700 includes a determination module 710, a linear solution module 720, an initial objective function establishment module 730, and a nonlinear iterative optimization module 740, specifically:

[0145] The determination module is used to determine the hand-eye calibration equation of the hand-eye vision system. The hand-eye vision system includes a camera and the end of the robot arm. The camera is installed at the end of the robot arm. The hand-eye calibration equation is used to solve the rigid body transformation matrix between the camera coordinate system and the end coordinate system of the robot arm;

[0146] The linear solution module is used to linearly solve the hand-eye calibration equation to obtain the initial rigid body transformation matrix between the camera coordinate system and the end coordinate system of the robot arm;

[0147] The initial objective function establishment module is used to establish an initial objective function for nonlinearly iteratively optimizing the initial rigid body transformation matrix. The initial objective function is used to characterize the relationship between the rigid body transformation matrix to be solved and the error between the rigid body transformation matrices to be solved;

[0148] The nonlinear iterative optimization module is used to generate the target rigid body transformation matrix corresponding to the initial rigid body transformation matrix using the new objective function. Among them, the target rigid body transformation matrix is used to transform the coordinates of the camera coordinate system into the end coordinate system of the robot arm. The new objective function is obtained by nonlinearly iteratively optimizing the initial objective function with the initial rigid body transformation matrix as the initial value.

[0149] In some embodiments, the nonlinear iterative optimization module is further used to: initialize the parameters included in the initial objective function using the initial rigid body transformation matrix to obtain the initialization parameters;

[0150] Substitute the initialization parameters into the initial objective function to obtain the first new objective function;

[0151] Determine the first optimized rigid body transformation matrix based on the first new objective function, and calculate the first error of the first optimized rigid body transformation matrix;

[0152] Update the parameters included in the first new objective function based on the first optimized rigid body transformation matrix to obtain updated parameters;

[0153] Substitute the updated parameters into the first new objective function to obtain a second new objective function;

[0154] Determine a second optimized rigid body transformation matrix based on the second new objective function, and calculate the second error of the second optimized rigid body transformation matrix;

[0155] When the difference between the first error and the second error is less than the threshold and the number of iterations is less than the maximum number of iterations, stop the iteration and output the second optimized rigid body transformation matrix, and the second optimized rigid body transformation matrix is the target transformation matrix.

[0156] In some embodiments, the non-linear iterative optimization module is further configured to, when the difference between the first error and the second error is greater than the threshold and the number of iterations is less than the maximum number of iterations, update the parameters included in the second new objective function based on the second optimized rigid body transformation matrix to obtain a corresponding new objective function, and then determine a corresponding optimized rigid body transformation matrix based on the new objective function, until the condition that the difference between the errors corresponding to the optimized rigid body transformation matrices obtained in two adjacent iterations is less than the threshold or the number of iterations reaches the maximum number of iterations is satisfied, and output the optimized rigid body transformation matrix obtained in the last iteration and use it as the target rigid body transformation matrix.

[0157] In some embodiments, the determination module is further configured to: establish a camera coordinate system, a robot arm end coordinate system, a robot base world reference coordinate system, and a calibration board coordinate system; determine the rigid body transformation matrix between the robot arm end coordinate system and the robot base world reference coordinate system, the rigid body transformation matrix between the camera coordinate system and the calibration board coordinate system, the rigid body transformation matrix between the calibration board coordinate system and the robot base world reference coordinate system, and the rigid body transformation matrix between the to-be-calibrated robot arm end coordinate system and the camera coordinate system; obtain the pose information of the robot under multiple motions, and determine the hand-eye calibration equation based on the pose information and the rigid body transformation matrix.

[0158] In some embodiments, the determination module is further configured to: determine the rigid body transformation matrix between the robot arm end coordinate system and the robot base world reference coordinate system, which further includes:

[0159] Read the rigid body transformation matrix between the robot arm end coordinate system and the robot base world reference coordinate system from the robot control panel.

[0160] In some embodiments, the determination module is further configured to: determine the rigid body transformation matrix between the camera coordinate system and the calibration board coordinate system, further including: obtaining the standard point cloud data of the calibration board; collecting the actual point cloud data of the calibration board by the camera; registering the standard point cloud data and the actual point cloud data, and obtaining the rigid body transformation matrix between the camera coordinate system and the calibration board coordinate system after registration.

[0161] In some embodiments, the initial rigid body transformation matrix includes an initial rotation matrix equation and an initial translation vector. The linear solution module is further configured to: convert the hand-eye calibration equation into a homogeneous hand-eye calibration matrix equation; convert the homogeneous hand-eye calibration matrix into a rotation matrix equation and a translation vector equation; solve the rotation matrix equation by using the least squares method to obtain the initial rotation matrix; substitute the initial rotation matrix into the translation vector equation and solve it by using the least squares method to obtain the initial translation vector.

[0162] Each of the above modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0163] In some embodiments, a terminal device is provided, including: at least one processor and a memory; the memory is used for storing program instructions; the processor is used for calling and executing the program instructions stored in the memory, so that the terminal device executes the above-mentioned eye calibration method. Its implementation principle and technical effects are similar to those of the above method embodiments and will not be elaborated here.

[0164] In some embodiments, a computer-readable storage medium is provided, characterized in that instructions are stored in the computer-readable storage medium, and when it runs on a computer, it causes the computer to execute the above-mentioned eye calibration method. Its implementation principle and technical effects are similar to those of the above method embodiments and will not be elaborated here.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties.

[0166] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAM), magnetoresistive random access memories (MRAM), ferroelectric random access memories (FRAM), phase change memories (PCM), etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include relational databases. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., without limitation.

[0167] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0168] The above embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A hand-eye calibration method, characterized in that The method includes: Determine the hand-eye calibration equation of the hand-eye vision system, where the hand-eye vision system includes a camera and the end of a robotic arm, the camera is mounted at the end of the robotic arm, and the hand-eye calibration equation is used to solve the rigid body transformation matrix between the camera coordinate system and the coordinate system of the end of the robotic arm; Linearly solve the hand-eye calibration equation to obtain the initial rigid body transformation matrix between the camera coordinate system and the coordinate system of the end of the robotic arm; Establish an initial objective function for nonlinearly iteratively optimizing the initial rigid body transformation matrix, where the initial objective function is used to characterize the relationship between the rigid body transformation matrix to be solved and the error of the rigid body transformation matrix to be solved; Generate a target rigid body transformation matrix corresponding to the initial rigid body transformation matrix using a new objective function, where the target rigid body transformation matrix is used to transform the coordinates of the camera coordinate system into the coordinate system of the end of the robotic arm, and the new objective function is obtained by nonlinearly iteratively optimizing the initial objective function with the initial rigid body transformation matrix as the initial value; The step of generating a target rigid body transformation matrix corresponding to the initial rigid body transformation matrix using the new objective function further includes: Initialize the parameters included in the initial objective function using the initial rigid body transformation matrix to obtain initialization parameters; Substitute the initialization parameters into the initial objective function to obtain a first new objective function; Determine a first optimized rigid body transformation matrix based on the first new objective function and calculate the first error of the first optimized rigid body transformation matrix; Update the parameters included in the first new objective function based on the first optimized rigid body transformation matrix to obtain updated parameters; Substitute the updated parameters into the first new objective function to obtain a second new objective function; Determine a second optimized rigid body transformation matrix based on the second new objective function and calculate the second error of the second optimized rigid body transformation matrix; When the difference between the first error and the second error is less than a threshold and the number of iterations is less than the maximum number of iterations, stop the iteration and output the second optimized rigid body transformation matrix, where the second optimized rigid body transformation matrix is the target rigid body transformation matrix; The formula for the initial rigid body transformation matrix is as follows: (R A -I)t X =R X t B -t A ; where M is the covariance matrix, k is the total number of the motion postures of the manipulator, T is the matrix transpose, n Bi and n Ai are the rotation axes of the rotation matrices A and B in the corresponding postures respectively, and I represents the identity matrix; substituting the initial rotation matrix R X , the rotation matrix R B , and the translation matrix t B into the above formula respectively, the initial translation vector t X can be calculated; The formula for the initial objective function is as follows: where x is the rigid body transformation matrix to be solved, k represents the number of iterations, ξ is the target residual value, and λ is the residual factor; The expression formula for the first error is as follows: E k = E k (x k ) = max|e i (x k )|, 1 ≤ i ≤ N; Update the parameters included in the first new objective function through the following formula: The formula for obtaining the second error is as follows: E K-1 = E K-1 (x) = max|e i-1 (x K )|, 1 ≤ i ≤ N.

2. The hand-eye calibration method according to claim 1, wherein When the difference between the first error and the second error is greater than the threshold and the number of iterations is less than the maximum number of iterations, update the parameters included in the second new objective function based on the second optimized rigid body transformation matrix to obtain a corresponding new objective function, and then determine the corresponding optimized rigid body transformation matrix based on the new objective function until the condition that the difference between the errors corresponding to the optimized rigid body transformation matrices obtained in two adjacent iterations is less than the threshold or the number of iterations reaches the maximum number of iterations is satisfied. Output the optimized rigid body transformation matrix obtained in the last iteration and use it as the target rigid body transformation matrix.

3. The hand-eye calibration method according to claim 1, wherein The determination of the hand-eye calibration equation of the hand-eye vision system further includes: Establish a camera coordinate system, a robot arm end coordinate system, a robot base world reference coordinate system, and a calibration board coordinate system; Determine the rigid body transformation matrix between the robot arm end coordinate system and the robot base world reference coordinate system, the rigid body transformation matrix between the calibration board coordinate system and the camera coordinate system, the rigid body transformation matrix between the calibration board coordinate system and the robot base world reference coordinate system, and the rigid body transformation matrix between the camera coordinate system to be calibrated and the robot arm end coordinate system; Obtain the pose information of the robot during multiple movements, and determine the hand-eye calibration equation based on the pose information and the rigid body transformation matrix.

4. The hand-eye calibration method according to claim 3, wherein The determination of the rigid body transformation matrix between the robot arm end coordinate system and the robot base world reference coordinate system further includes: Read the rigid body transformation matrix between the robot arm end coordinate system and the robot base world reference coordinate system from the robot control panel.

5. The hand-eye calibration method according to claim 3, wherein The determination of the rigid body transformation matrix between the camera coordinate system and the calibration board coordinate system further includes: Obtain the standard point cloud data of the calibration board; Collect the actual point cloud data of the calibration board through the camera; Register the standard point cloud data and the actual point cloud data, and obtain the rigid body transformation matrix between the camera coordinate system and the calibration board coordinate system after registration.

6. The hand-eye calibration method according to claim 1, wherein The initial rigid body transformation matrix includes an initial rotation matrix equation and an initial translation vector. The linear solution of the hand-eye calibration equation to obtain the initial rigid body transformation matrix between the camera coordinate system and the robot arm end coordinate system further includes: Convert the hand-eye calibration equation into a homogeneous hand-eye calibration matrix equation; Convert the homogeneous hand-eye calibration matrix into a rotation matrix equation and a translation vector equation; Solve the rotation matrix equation using the least squares method to obtain the initial rotation matrix; Substitute the initial rotation matrix into the translation vector equation and solve it using the least squares method to obtain the initial translation vector.

7. A hand-eye calibration device, the device being used to execute the method according to any one of claims 1 to 6, characterized in that, including: A determination module for determining the hand-eye calibration equation of a hand-eye vision system, the hand-eye vision system including a camera and a robot arm end, the camera being installed at the robot arm end, and the hand-eye calibration equation being used to solve the rigid body transformation matrix between the camera coordinate system and the robot arm end coordinate system; A linear solution module for linearly solving the hand-eye calibration equation to obtain the initial rigid body transformation matrix between the camera coordinate system and the robot arm end coordinate system; An initial objective function establishing module, configured to establish an initial objective function for nonlinearly iteratively optimizing the initial rigid body transformation matrix, where the initial objective function is used to characterize the relationship between the to-be-solved rigid body transformation matrix and the error of the to-be-solved rigid body transformation matrix; A nonlinearly iterative optimization module, configured to generate a target rigid body transformation matrix corresponding to the initial rigid body transformation matrix by using a new objective function, where the target rigid body transformation matrix is used to transform the coordinates in the camera coordinate system into the coordinate system at the end of the robot arm, and the new objective function is obtained by nonlinearly iteratively optimizing the initial objective function with the initial rigid body transformation matrix as the initial value.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the hand-eye calibration method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Instructions are stored in the computer-readable storage medium, and when it runs on a computer, the computer is caused to execute the hand-eye calibration method according to any one of claims 1-6.

Images