A method for collecting hand-eye calibration data

Abstract

The present invention belongs to the field of robot vision, and discloses a method for collecting hand-eye calibration data, comprising the following steps: collecting a calibration plate image in a first posture, and obtaining a first group of robot transformation matrices and camera extrinsic matrix; rotating only the continuous multiple joints (at least 2 joints) of the robot end to obtain a second posture, collecting a calibration plate image, and obtaining a second group of robot transformation matrices and camera extrinsic matrix; obtaining a group of A and B through two groups of robot transformation matrices and camera extrinsic matrix; repeating multiple times to obtain multiple groups of A and B, constructing an equation AX=XB to calculate and obtain a solution X. The present invention improves the hand-eye calibration data collection process, and the second posture is only a rotation of multiple continuous joints at the end compared to the first posture, which effectively reduces the error of the robot transformation matrix, thereby improving the hand-eye calibration accuracy, and making the robot hand-eye calibration result more accurate in industrial application scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of robot vision, and more specifically, relates to a hand-eye calibration data acquisition method. Background Art

[0002] Industrial robots can not only improve production quality and efficiency, but also reduce costs, liberate manpower and improve resource utilization, which is an inevitable trend in the development of the manufacturing industry. In traditional industrial robot applications, robots can only perform fixed grasping actions after being taught. With the improvement of modern production requirements, the combination of machine vision and industrial robots has become the mainstream. The key to achieving coordinated work between vision and manipulators is hand-eye calibration technology.

[0003] The accuracy of hand-eye calibration directly affects the position accuracy of the robot when performing operations, so improving the accuracy of hand-eye calibration is of great significance to the application field of robots. Conventional data acquisition methods do not restrict the movement of the robot arm, which will introduce unrestricted robot joint rotation angle data errors, resulting in large errors in the robot transformation matrix. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the purpose of the present invention is to provide a hand-eye calibration data acquisition method, wherein starting from the robot kinematic model and based on in-depth research on the error source, by improving the hand-eye calibration data acquisition process, a set of transformation matrices A are obtained by using the first set of robot transformation matrices and camera extrinsic matrix in the first posture and the second set of robot transformation matrix and camera extrinsic matrix in the second posture i and B i , and the second posture is compared with the first posture, only the continuous joints (at least 2 joints) at the end are rotated (translation operation is excluded at the same time), and the other joints far away from the end remain fixed. The transformation matrix A is obtained in this way i and B i ; By designing the robot's posture when collecting data, the error of the robot's transformation matrix can be effectively reduced, thereby improving the hand-eye calibration accuracy and making the robot hand-eye calibration results more accurate in industrial application scenarios.

[0005] To achieve the above object, according to one aspect of the present invention, a hand-eye calibration data acquisition method is provided, which is characterized by comprising the following steps:

[0006] S1. Move the robot to any suitable position, in which the image of the calibration plate is clearly visible in the camera lens connected to the robot end effector; record this position as the first position, and collect the target calibration plate image;

[0007] S2. Obtain the first set of robot transformation matrices and camera extrinsic matrix in the first posture;

[0008] S3. For the robot in the first posture, only rotate at least two consecutive joints of the end, and the new posture of the robot is recorded as the second posture. In this posture, the image of the calibration plate in the camera lens fixed to the end effector of the robot is also clearly visible; then, in the second posture, collect the image of the calibration plate;

[0009] S4. Obtain a second set of robot transformation matrices and camera extrinsic matrix in a second posture;

[0010] S5. Using the first set of robot transformation matrices and camera extrinsic matrix and the second set of robot transformation matrix and camera extrinsic matrix, obtain transformation matrices A1 and B1;

[0011] S6. Repeat S1 to S5 to obtain multiple sets of transformation matrices A2 and B2, ..., A n and B n ;

[0012] S7. Set A1, A2, ..., A n Concatenate to get matrix A, and replace B1, B2, ..., B n The matrix B is obtained by concatenation, and the solution X of the equation AX=XB is calculated.

[0013] As a further preferred embodiment of the present invention, the step S7 is specifically:

[0014] A1~A n Splice to get matrix A, and put B1~B n The matrix B is concatenated to construct the hand-eye equation AX=XB. The optimal solution X of the equation is calculated based on the least squares method through the matrix vectorization operator method.

[0015] As a further preferred embodiment of the present invention, step S1 specifically includes the following sub-steps:

[0016] S11. Move the robot until the image of the calibration plate is clearly visible in the camera lens fixed to the end effector, and obtain the first pose;

[0017] S12. Take the calibration plate image and extract the feature points;

[0018] S13. Record the rotation angle of each joint of the robot at this moment.

[0019] As a further preferred embodiment of the present invention, step S2 specifically includes the following sub-steps:

[0020] S21. Read the rotation angles of each joint of the robot recorded in S1;

[0021] S22. Calculate the robot transformation matrix according to the robot joint rotation angle;

[0022] S23. Read the extracted feature point coordinates;

[0023] S24. Calculate the camera extrinsic parameter matrix based on the feature point coordinates and their corresponding actual coordinates.

[0024] As a further preferred embodiment of the present invention, step S3 specifically includes the following sub-steps:

[0025] S31. For the robot in the first posture, only rotate at least two consecutive joints of the end, and the new posture of the robot is recorded as the second posture. In this posture, the image of the calibration plate in the camera lens fixed to the end effector of the robot is also clearly visible;

[0026] S32. Capture the calibration plate image and extract feature points;

[0027] S33. Record the rotation angle of each joint of the robot at this moment, wherein the rotation angle of each joint that is not rotating remains unchanged.

[0028] As a further preferred embodiment of the present invention, step S4 specifically includes the following sub-steps:

[0029] S41. Read the rotation angles of each joint of the robot recorded in S3;

[0030] S42. Calculate the robot transformation matrix according to the robot joint rotation angle;

[0031] S43. Read the extracted feature point coordinates;

[0032] S44. Calculate the camera extrinsic parameter matrix based on the feature point coordinates and their corresponding actual coordinates.

[0033] According to another aspect of the present invention, a hand-eye calibration data acquisition method is provided, characterized in that it comprises the following steps:

[0034] S1. Move the robot to any suitable position, in which the image of the calibration plate fixed to the robot end effector is clearly visible in the camera lens; record this position as the first position, and collect the image of the target calibration plate;

[0035] S2. Obtain the first set of robot transformation matrices and camera extrinsic matrix in the first posture;

[0036] S3. For the robot in the first posture, only rotate at least two consecutive joints of the end, and the new posture of the robot is recorded as the second posture. In this posture, the image of the calibration plate fixed to the robot end effector is also clearly visible in the camera lens; then, in the second posture, collect the image of the calibration plate;

[0037] S4. Obtain a second set of robot transformation matrices and camera extrinsic matrix in a second posture;

[0038] S5. Using the first set of robot transformation matrices and camera extrinsic matrix and the second set of robot transformation matrix and camera extrinsic matrix, obtain transformation matrices A1 and B1;

[0039] S6. Repeat S1 to S5 to obtain multiple sets of transformation matrices A2 and B2, ..., A n and B n ;

[0040] S7. Set A1, A2, ..., A n Concatenate to get matrix A, and replace B1, B2, ..., B n The matrix B is obtained by concatenation, and the solution X of the equation AX=XB is calculated.

[0041] As a further preferred embodiment of the present invention, the step S7 is specifically:

[0042] A1~A n Splice to get matrix A, and put B1~B n The matrix B is concatenated to construct the hand-eye equation AX=XB. The optimal solution X of the equation is calculated based on the least squares method through the matrix vectorization operator method.

[0043] Through the above technical scheme conceived by the present invention, compared with the prior art, the present invention starts with the robot kinematic model, based on in-depth research on the error source, in the process of collecting data for constructing the hand-eye equation AX=XB, a set of transformation matrices A are obtained by using the first set of robot transformation matrices and camera extrinsic matrix in the first posture and the second set of robot transformation matrices and camera extrinsic matrix in the second posture. i and B i , and the second posture is compared with the first posture, only the continuous joints (at least 2 joints) at the end are rotated (translation operation is excluded at the same time), and the other joints far away from the end remain fixed. In this way, the transformation matrix A obtained by the robot's posture when collecting data is designed i and B i , which can effectively reduce the error of the robot's transformation matrix and thus improve the hand-eye calibration accuracy.

[0044] According to the present invention, after collecting a set of camera poses and robot poses, only a plurality of continuous joint angles starting from the end are changed, and the remaining joint angles are kept unchanged, and then a second set of camera poses and robot poses are collected and a set of A is calculated. i and B i ; Through multiple repetitions, multiple sets of A required for calculating the hand-eye calibration results can be obtained i and B iCompared with conventional data collection methods, in the same scenario, the data collected by the present invention can make the calculated hand-eye calibration more accurate, and is applicable to all methods for hand-eye calibration based on AX=XB, and is applicable to both "eye-in-hand" type robot-camera systems and "eye-outside-hand" type robot-camera systems.

[0045] The method of the present invention can effectively improve the accuracy of robot hand-eye calibration results for industrial applications; compared with conventional data acquisition methods, in the same scenario, the data collected by the present invention can make the calculated hand-eye calibration more accurate, and is applicable to all methods for hand-eye calibration based on AX=XB.

[0046] In summary, the present invention starts with the robot kinematic model, deeply analyzes the factors that cause the robot transformation matrix calculation error, improves the data acquisition method, limits the robot arm movement mode to reduce the error of the acquired joint rotation angle data, and improves the accuracy of the robot transformation matrix, thereby further improving the accuracy of hand-eye calibration. Since only the data acquisition method is changed, the present invention is applicable to all methods for hand-eye calibration based on AX=XB, and can provide more accurate hand-eye calibration results for robot hand-eye calibration in industrial application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the hand-eye calibration data acquisition method provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0049] Example 1

[0050] like Figure 1 As shown, for an "eye-in-hand" type robot-camera system (i.e., a system in which the camera is fixed on a robot flange), the hand-eye calibration data acquisition method of the present invention can be specifically performed according to the following steps:

[0051] S1. Move the robot to a suitable position and collect the image of the calibration plate;

[0052] S2. Obtain a first set of robot transformation matrices and camera extrinsic matrix;

[0053] S3. Rotate only the continuous multiple joints at the end of the robot arm to move the robot to a suitable position and collect the calibration plate image; for example, by rotating only the continuous multiple joints at the end, the continuous multiple joint angles starting from the end can be changed (it can be two joint angles, or three, four, or five joint angles), and the remaining joint angles are fixed;

[0054] S4. Repeat S2 to obtain a second set of robot transformation matrix and camera extrinsic matrix;

[0055] S5. Obtain transformation matrices A1 and B1; specifically, based on known methods in the prior art, A1 can be calculated according to the first and second groups of robot transformation matrices, and B1 can be calculated according to the first and second groups of camera extrinsic parameter matrices.

[0056] S6. Repeat S1 to S5 to obtain multiple sets of transformation matrices A n and B n ;

[0057] S7. Calculate the optimal solution X of the equation AX=XB.

[0058] The above process can be used to reduce the error of the robot transformation matrix and improve the accuracy of hand-eye calibration.

[0059] Specific analysis:

[0060] (1) Based on the robot kinematic model, the error sources are analyzed. In the hand-eye calibration equation AX=XB, the error sources are mainly A and B. A is related to the robot transformation matrix, and B is related to the camera extrinsic matrix. Because the camera accuracy is usually much higher than that of the robot, the present invention analyzes the error sources of the robot transformation matrix. The local POE (exponential product) kinematic model for serial robots can be described as:

[0061]

[0062] in is the rigid transformation matrix from n-1 to n joints in the initial position. ξ, Δξ∈se(3) describe the joint rotation angle and its deviation (se(3) has the same meaning as in mathematical Lie group, representing the Lie algebra of SE(3) special Euclidean group), which is the main source of robot kinematic error. θ and Δθ are the joint angle values ​​and their errors. The joint angle θ is defined by the robot encoder with high accuracy, so we ignore the influence of Δθ, and the formula becomes:

[0063]

[0064] Therefore, the kinematic error of a complete single-joint robot can be described as:

[0065]

[0066] In hand-eye calibration, two different robot poses and A can be composed as:

[0067]

[0068] The kinematic model represents the robot's position as follows:

[0069]

[0070] Among them, ξ n and Δξ n represents ξ and Δξ of joint n, θ n1 represents the joint angle value of joint n when it is in posture 1. According to the method of the present invention, during the data collection process, if the robot posture and The k (k≤n) joint angles of are set to the same value, that is:

[0071] θ 11 =θ 11 ,θ 21 =θ 21 ,…,θ k1 =θ k2

[0072] At this time, the equation and satisfy:

[0073]

[0074] therefore:

[0075]

[0076] because and The relationship between A and B can be expressed as:

[0077]

[0078] The above formula avoids Δξ1~Δξ k The influence of , therefore the accuracy of A can be increased.

[0079] (2) Based on the method of the present invention, a solvability analysis is performed. The previous article proves that setting the same joint angle value in adjacent positions can reduce the robot kinematic error. In theory, the larger k is, the more accurate A is. However, in order to solve the equation AX=XB, the selection of k is limited. The equation AX=XB can be decomposed into a rotation part and a translation part:

[0080] RA R x =R X R B

[0081] R A t X +t A =R X t B +t X

[0082] Where R and t are the rotation and translation parts of the transformation matrix. Let α^, β^∈so(3), let:

[0083] α^=log R A , β^=1ogR B

[0084] Among them, α, Respectively represent R A and R B The logarithm of Consistent with the meaning in mathematical linear algebra, it represents a 3-dimensional real number space). Therefore, SO(3) is transformed into so(3) (consistent with the meaning in mathematical Lie groups, SO(3) represents the three-dimensional special orthogonal group of 3×3 matrices, and so(3) represents the Lie algebra of the orthogonal group of SO(3)), and the rotation equation can be written as:

[0085]

[0086] By combining the equations:

[0087] (Rω)^=Rω^R T

[0088] exp((Rω)^)=Rexp(ω^)R T

[0089] where R∈SO(3), The rotation equation can be written as:

[0090] α=R X β

[0091] Where R X There are three degrees of freedom, and the above formula can be written as three equations. However, applying the Cauchy-Schwarz inequality, we have:

[0092] ||R x β||≤||R X ||||β||=||β||

[0093]

[0094] Then we have:

[0095] ||α||=||R X β||=||β||

[0096] Due to the constraint between α and β, only two independent equations are needed: α = R X β can be used to solve R X .

[0097] For the equation AX=XB, when k=n, the equation becomes:

[0098] IX=XI

[0099] Obviously, no effective solution can be obtained. When k = n-1, the equation becomes:

[0100]

[0101] Make A i and A j A from different AX=XB, then:

[0102]

[0103]

[0104] where θ n1i and θ n1 j is taken from A i and A j A j You can use A i It is expressed as:

[0105]

[0106] where μ = (θ n1j -θ n2j ) / (θ n1i -θ n2i ), so α i and α j The connection can be written as:

[0107] α j =μα i

[0108] And α j =R X β j It can be written as:

[0109] μα i =μR X β i

[0110] From a geometric point of view, αi and α j are vectors with the same axis and different norms. Therefore, A i and A j It is impossible to provide independent equations to solve the rotation equations. Therefore, when k = n-1, the equation AX = XB is also unsolvable. Therefore, in order to make A i and A j Provides sufficiently efficient equations to solve the rotation equations, since α i and α j It is a vector with different axes, which requires at least two joint angles with different values. Therefore, k≤n-2 is required, and the equation AX=XB can be solved.

[0111] For this reason, the method of the present invention requires that after obtaining the first set of robot transformation matrices and camera extrinsic parameter matrices, only at least 2 consecutive joints of the end are rotated (the number of these rotated joints is i, i ≥ 2 and not more than n, then the first ni joints of the robot away from the end remain stationary, corresponding to the k ≤ n-2 requirement; and, for these consecutive i joints of the rotated end, only rotation is performed, excluding translation operations), and then the second set of robot transformation matrices and camera extrinsic parameter matrices are obtained.

[0112] (3) In addition, for hand-eye calibration, solve the position and posture of the camera coordinate system in the robot flange coordinate system:

[0113] When solving the optimal solution X of the equation AX=XB, the nonlinear equation AX=XB can also be linearized by the matrix vectorization operator method, and then the equation can be solved by the least squares method. Since the solution obtained by the least squares method often cannot satisfy the orthogonal constraints of the transformation matrix, the solution obtained needs to be processed by the known condition ||R||=1 and the existing QR decomposition method to obtain the position and posture of the camera coordinate system in the robot flange coordinate system.

[0114] For example, for n groups A1 and B1, A2 and B2, ..., A obtained by repeating steps S1 to S5 n times, n and B n , we can convert matrices A1, A2, ..., A n As vector elements, they are concatenated in the form of n rows and 1 column vectors to obtain The matrices B1, B2, ..., B n As vector elements, they are concatenated in the form of n rows and 1 column vectors to obtain Then, based on least squares, the optimal solution X is obtained for AX=XB.

[0115] In addition, in this embodiment, in order to improve the efficiency of data collection, the ABB robot and the RobotStudio industrial software developed by it can be used to manipulate the robot through designing programs to realize automatic data collection.

[0116] Embodiment 2:

[0117] The system used in the above embodiment 1 is an "eye in hand" system. In addition, the present invention is also applicable to an "eye outside hand" system, that is, a system in which the camera is located on a fixed bracket that does not move with the robot. In this case, the calibration plate is usually fixedly connected to the robot end effector. The calibration method is similar to the calibration method of the "eye in hand" system, and can also include the following steps:

[0118] S1. Move the robot to a suitable position and collect the image of the calibration plate;

[0119] S2. Obtain a first set of robot transformation matrices and camera extrinsic matrix;

[0120] S3. Rotate only the continuous multi-joints at the end to move the robot to a suitable position and collect the calibration plate image;

[0121] S4. Repeat S2 to obtain a second set of robot transformation matrix and camera extrinsic matrix;

[0122] S5. Obtain transformation matrices A1 and B1;

[0123] S6. Repeat S1 to S5 to obtain multiple sets of transformation matrices A n and B n ;

[0124] S7. Calculate the optimal solution X of the equation AX=XB.

[0125] This method is similar to the method in Example 1 and can also effectively improve the accuracy of hand-eye calibration, so it will not be described in detail here.

[0126] The above embodiments are merely examples. For example, in addition to the matrix vectorization operator method, the solution of AX=XB may also adopt a variety of other hand-eye calibration calculation methods known in the prior art, which will not be listed one by one in the present invention.

[0127] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A hand-eye calibration data collection method, characterized in that: The following steps are involved: S1. Move the robot to any suitable position, in which the image of the calibration plate is clearly visible in the camera lens connected to the robot end effector; record this position as the first position, and collect the target calibration plate image; S2. Obtain the first set of robot transformation matrices and camera extrinsic matrix in the first posture; S3. For the robot in the first posture, only rotate at least two consecutive joints of the end, and the new posture of the robot is recorded as the second posture. In this posture, the image of the calibration plate in the camera lens fixed to the end effector of the robot is also clearly visible; then, in the second posture, collect the image of the calibration plate; S4. Obtain a second set of robot transformation matrices and camera extrinsic matrix in a second posture; S5. Using the first set of robot transformation matrices and camera extrinsic matrix and the second set of robot transformation matrix and camera extrinsic matrix, obtain transformation matrices A1 and B1; S6. Repeat S1 to S5 to obtain multiple sets of transformation matrices A2 and B2, ..., A n and B n ; S7. Set A1, A2, ..., A n Concatenate to get matrix A, and replace B1, B2, ..., B n The matrix B is obtained by concatenation, and the solution X of the equation AX=XB is calculated.

2. The method according to claim 1, characterized in that: The step S7 specifically comprises: A1~A n Splice to get matrix A, and put B1~B n The matrix B is concatenated to construct the hand-eye equation AX=XB. The optimal solution X of the equation is calculated based on the least squares method through the matrix vectorization operator method.

3. The method according to claim 1, characterized in that: The step S1 specifically includes the following sub-steps: S11. Move the robot until the image of the calibration plate is clearly visible in the camera lens fixed to the end effector, and obtain the first pose; S12. Take the calibration plate image and extract the feature points; S13. Record the rotation angle of each joint of the robot at this moment.

4. The method according to claim 3, characterized in that: The step S2 specifically includes the following sub-steps: S21. Read the rotation angles of each joint of the robot recorded in S1; S22. Calculate the robot transformation matrix according to the robot joint rotation angle; S23. Read the extracted feature point coordinates; S24. Calculate the camera extrinsic parameter matrix based on the feature point coordinates and their corresponding actual coordinates.

5. The method according to claim 1, characterized in that: The step S3 specifically includes the following sub-steps: S31. For the robot in the first posture, only rotate at least two consecutive joints of the end, and the new posture of the robot is recorded as the second posture. In this posture, the image of the calibration plate in the camera lens fixed to the end effector of the robot is also clearly visible; S32. Capture the calibration plate image and extract feature points; S33. Record the rotation angle of each joint of the robot at this moment, wherein the rotation angle of each joint that is not rotating remains unchanged.

6. The method according to claim 5, characterized in that The step S4 specifically includes the following sub-steps: S41. Read the rotation angles of each joint of the robot recorded in S3; S42. Calculate the robot transformation matrix according to the robot joint rotation angle; S43. Read the extracted feature point coordinates; S44. Calculate the camera extrinsic parameter matrix based on the feature point coordinates and their corresponding actual coordinates.

7. A hand-eye calibration data collection method, characterized in that: The following steps are involved: S1. Move the robot to any suitable position, in which the image of the calibration plate fixed to the robot end effector is clearly visible in the camera lens; record this position as the first position, and collect the image of the target calibration plate; S2. Obtain the first set of robot transformation matrices and camera extrinsic matrix in the first posture; S3. For the robot in the first posture, only rotate at least two consecutive joints of the end, and the new posture of the robot is recorded as the second posture. In this posture, the image of the calibration plate fixed to the robot end effector is also clearly visible in the camera lens; then, in the second posture, collect the image of the calibration plate; S4. Obtain a second set of robot transformation matrices and camera extrinsic matrix in a second posture; S5. Using the first set of robot transformation matrices and camera extrinsic matrix and the second set of robot transformation matrix and camera extrinsic matrix, obtain transformation matrices A1 and B1; S6. Repeat S1 to S5 to obtain multiple sets of transformation matrices A2 and B2, ..., A n and B n ; S7. Set A1, A2, ..., A n Concatenate to get matrix A, and replace B1, B2, ..., B n The matrix B is obtained by concatenation, and the solution X of the equation AX=XB is calculated.

8. The method according to claim 7, characterized in that: The step S7 specifically comprises: A1~A n Splice to get matrix A, and put B1~B n The matrix B is concatenated to construct the hand-eye equation AX=XB. The optimal solution X of the equation is calculated based on the least squares method through the matrix vectorization operator method.

Images