A binocular vision hand-eye calibration method using two-layer nonlinear optimization

Abstract

This invention discloses a binocular vision hand-eye calibration method based on two-layer nonlinear optimization. The method first establishes the hand-eye matrix equation and solves for the initial values ​​of the hand-eye matrix using the linear least squares method. Secondly, it addresses the rotation component R in the hand-eye matrix. h2e Euler angle transformation is performed to ensure the orthogonality of the rotation matrix R. Then, an optimized model H1 of the hand-eye matrix is ​​constructed, and the initial values ​​of the hand-eye matrix are optimized for the first time using the traditional LM nonlinear optimization method to obtain the hand-eye matrix after the first optimization. Finally, the optimized model H1 of the hand-eye matrix is ​​corrected to obtain the second optimized model H2. Then, the LM nonlinear optimization method with sample selection is added to optimize the hand-eye matrix after the first optimization for the second optimization to obtain the second optimized hand-eye matrix. In view of the problem of insufficient accuracy of traditional hand-eye calibration methods, this method can reduce the influence of random errors in robot monocular vision systems and improve the calibration accuracy of robot hand-eye matrix.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and robotics, and specifically relates to a binocular vision hand-eye calibration method using two-layer nonlinear optimization. Background Technology

[0002] With the development of industrial intelligence, cases of combining vision technology with industrial robots are emerging one after another. The method of mounting a vision system on the end effector of a robot is called "eye-on-hand," which benefits from the robot's flexibility. It has a wider operating range than vision sensors fixed in one position. Currently, it is widely used in various robot manufacturing systems, typically in robotic welding, grinding, assembly, aerospace, and automotive fields. However, for vision robots, the accuracy of hand-eye calibration directly affects the final positioning accuracy of the robot's end effector; therefore, the stability and accuracy of hand-eye calibration are particularly important.

[0003] Currently, there are three main methods for hand-eye calibration: Tsai's calibration method, the Dual-Quaternion calibration method, and Zhang Zhengyou's calibration method. Tsai's calibration method obtains the calibration matrix by performing nonlinear least-squares solutions on multiple known relationships between the robot's end effector and the camera. The Dual-Quaternion calibration method represents the motion relationship between the robot's end effector and the camera as a dual quaternion, and then obtains the calibration matrix by solving the parameters of the dual quaternion through nonlinear optimization. Zhang Zhengyou's calibration method is a hand-eye calibration method based on camera intrinsic and extrinsic parameters. This method calibrates the camera's intrinsic and extrinsic parameters and obtains the calibration matrix through optimized matching between camera measurements and the robot's motion trajectory.

[0004] Current solutions for hand-eye calibration, such as the "Calibration Method, Device, Controller, and Storage Medium for Live-Line Working Equipment" disclosed in Chinese Invention Patent No. CN115383749A, describe a specific hand-eye calibration process. A dual-calibration plate is fixed to the end of a robotic arm. The robotic arm is moved, and an image acquisition device is used to acquire a first calibration plate image. The first calibration plate image is used to perform dual-calibration on the image acquisition device. If the dual-calibration result is satisfactory, and the hand-eye calibration plate is fixed to the end of the robotic arm, the robotic arm is moved, and the image acquisition device is used to acquire a second calibration plate image. The second calibration plate image is used to perform hand-eye calibration on the image acquisition device and the robotic arm until the hand-eye calibration result is satisfactory. However, this method does not optimize the hand-eye matrix, leading to errors that can cause inaccurate hand-eye calibration and affect the accuracy and robustness of the robot system. Summary of the Invention

[0005] This invention addresses the random errors present in robot vision systems by designing a two-layer Levenberg-Marquardt (LM) optimization method based on the principle of nonlinear optimization. After initial optimization of the constructed hand-eye matrix optimization model using the traditional LM method, a secondary optimization is performed using an LM optimization method with added sample point filtering function. This method can complete calibration without the need for special targets, and the accuracy of the proposed calibration method meets the working requirements of vision robots in the industrial field.

[0006] To achieve the above objectives, the main technical solution of the present invention is as follows:

[0007] First, select a chessboard calibration board, fix its position, and then change the position and posture of the robot's end effector.

[0008] The checkerboard calibration board is fully displayed within the field of view of the binocular camera system; then, the expression of feature point P on the checkerboard calibration board within the field of view in robot base coordinates 1-1 is solved, and the initial values ​​of the hand-eye matrix are obtained. Finally, the initial values ​​of the hand-eye matrix are optimized using LM nonlinear optimization. The initial optimization yields the hand-eye matrix after the first optimization. Then, a second nonlinear optimization is performed based on sample selection to obtain the hand-eye matrix after the second optimization. During the iteration process, points with large random errors are removed from the sample point set to obtain the hand-eye matrix after the second optimization. This can be used as the final result of hand-eye alignment, and specifically includes the following steps:

[0009] Step 1: Create three coordinate systems in the robot system: camera coordinate system OX. C Y C Z C (1-3), the robot's end effector TCP coordinate system OX E Y E Z E (1-2) and robot base coordinate system OX B Y B Z B (1-1);

[0010] Step 2: Solve for the coordinate expression of feature point P in the robot's base coordinate system (1-1), which includes at least the following two steps:

[0011] Step 2-1: Mark the coordinates of the first corner point, i.e., feature point P, on the checkerboard calibration board in camera coordinate system 1-3. C P is marked with coordinates P in the robot's base coordinate system 1-1. BThe intrinsic and extrinsic parameters of the left and right cameras obtained from the camera calibration program can be used to calculate P using equation (1). C The value;

[0012]

[0013] Step 2-2: According to Figure 1 The coordinate transformation relationship shown can be expressed by equation (2).

[0014]

[0015] Obtain the coordinates P of the feature point in the robot base coordinate system 1-1. B ;

[0016] Step 3: Initial values ​​of the hand-eye matrix The determination of [the value] includes at least the following three steps:

[0017] Step 3-1: First, change the robot's posture so that the camera takes pictures of feature point P in different poses, and collect n (n≥4) sets of pictures. Each set contains two images collected by two cameras respectively. The checkerboard calibration board can be fully presented in the field of view of the binocular camera vision system. The coordinates of feature point P in the world coordinate system under different poses can be solved by equation (3). B :

[0018]

[0019] For each shooting pose, feature points P1, P2, ..., P in n sets of images n Its position in the robot's base coordinate system 1-1 remained unchanged, therefore Equation (3) can then be rewritten as:

[0020]

[0021] Step 3-2: In equation (4), and Let i and j represent the transformation matrices from the robot end-effector TCP coordinate system 1-2 to the robot base coordinate system 1-1 in the i-th and j-th samples, respectively (i = 1, 2, ..., n-1, j = i+1, i+2, ..., n).

[0022] Will and Rewritten as equation (5):

[0023]

[0024] The matrix equation is obtained from equations (4) and (5), as shown in equation (6):

[0025]

[0026] Step 3-3: Let A be the left-hand coefficient matrix, X be the unknown quantity of the hand-eye matrix, and b be the right-hand coefficient matrix. Given A and b, the problem of solving X is transformed into the problem of solving the matrix equation. Since the number of collected images is redundant for solving the above matrix equation, the least squares method is used to solve the matrix equation to minimize the final error. The least squares solution of the matrix equation is shown in equation (7):

[0027] X = (A T A) -1 A T b (7)

[0028] After solving for the initial hand-eye matrix X, that is, the initial value of the hand-eye matrix... Hand-eye matrix rotational component R h2e pacing h2e The components are shown in equation (8):

[0029]

[0030] Step 4, Initial LM Nonlinear Optimization, includes at least the following five steps:

[0031] Step 4-1: Initialize the hand-eye matrix obtained in Step 3. It is represented as shown in equation (9):

[0032]

[0033] Step 4-2: Hand-eye matrix The rotational component R in h2e Perform Euler angle transformation to determine the transformation angle θ between the x, y, and z axes of the camera coordinate system 1-3 and the x, y, and z axes of the robot's end effector TCP coordinate system 1-2 during rotation. x θ y θ z As shown in equation (10):

[0034]

[0035] Step 4-3: Use the inverse Euler angle transformation to find the rotation matrix R, as shown in equation (11):

[0036]

[0037] Therefore, the initial value of the hand-eye matrix As shown in equation (12):

[0038]

[0039] Step 4-4: Constructing the error function W1 and loss function L1: Since the coordinates of feature point P are constant in the world coordinate system, an initial optimization model H1 is established. The error function W1 of the initial optimization model H1 is shown in equation (13):

[0040]

[0041] The loss function L1 of the initial optimization model H1 is shown in equation (14):

[0042]

[0043] Step 4-5: Using the error function W1 and loss function L1 from equations (13) and (14), the LM algorithm is used to initialize the hand-eye matrix. The first optimization is performed, resulting in the optimized hand-eye matrix. The Euler angles are represented as shown in equation (15):

[0044] [θ x θ y θ z t x t y t z ] T (15)

[0045] Step 5, combined with the second-order nonlinear optimization of sample point selection, includes at least the following six steps:

[0046] Step 5-1: Calculate the first optimized hand-eye matrix 1T E C The Euler angle representation is converted into a matrix representation, as shown in equation (16):

[0047]

[0048] Step 5-2: Use the method of calculating the mean to obtain the standard point P. S The coordinates of the in the robot's base coordinate system 1-1 are shown in equation (17):

[0049]

[0050] Step 5-3: Correct the initial optimization model H1 and substitute the standard point P. S The coordinates; the corrected error function W2 is shown in equation (18):

[0051]

[0052] The corrected loss function L2 is shown in equation (19):

[0053]

[0054] Step 5-4: Initialize the iteration-related parameters for the second optimization and substitute them into the hand-eye matrix after the first optimization. These are the initial values ​​for the second optimization iteration;

[0055] Step 5-5: Based on the comparison between each sample point and the standard point P S The distance values ​​between the two points are used to mark sample points that deviate significantly from the standard point, and the distance between each sample point and the standard point P is calculated. S The distance d between i As shown in equation (20):

[0056]

[0057] Distance d i <d i The sample points from point A' form a new sample set D, and the mean of sample set D is calculated. The variance σd; the distribution of random errors follows a normal distribution. From the law of normal distribution, we know... If it follows a standard normal distribution, then based on the boundary values ​​of the confidence interval of the standard normal distribution, we can obtain the condition... Mark the samples outside the t% confidence interval;

[0058] Step 5-6: Obtain the number of iterations m before sample point screening. If the number reaches s, screen the sample points. The screening step refers to traversing all samples to determine the number of times the samples are labeled in the above s iterations. Sample points with a labeling number exceeding 0.5*s are considered as sample points with large errors, and all sample points with large errors are removed from the sample point set.

[0059] The features and beneficial effects of this invention are:

[0060] 1. The present invention provides a dual-layer nonlinear optimized binocular vision hand-eye calibration method. Compared with the hand-eye matrix calibration method of traditional methods, the dual-layer nonlinear optimized binocular vision hand-eye calibration method can improve the stability of the robot vision system and meet industrial requirements in terms of accuracy.

[0061] 2. This algorithm introduces a two-layer optimization method for sample detection. During the optimization process, Euler angle transformation is used to ensure the orthogonality of the hand-eye matrix, which can accurately calibrate the hand-eye matrix of the visual robot, reduce the influence of random errors in the system, and improve the robustness of hand-eye calibration. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of coordinate system transformation.

[0063] Figure 2 A flowchart of the overall process for a binocular vision hand-eye calibration method with two-layer nonlinear optimization;

[0064] Figure 3 This is the flowchart for the first nonlinear optimization.

[0065] Figure 4 The flowchart shows the LM algorithm with sample point selection.

[0066] In the attached diagram, 1-1 shows the robot's base coordinate system OX. B Y B Z B ;1-2 Robot end effector TCP coordinate system OX E Y E Z E ; 1-3 Camera coordinate system OX C Y C Z C . Detailed Implementation

[0067] The present invention will be further described below with reference to the accompanying drawings.

[0068] Appendix Figure 1 The construction methods of the robot base coordinate system 1-1, camera coordinate system 1-3, and robot end effector TCP coordinate system 1-2, and the transformation relationships among them are explained; see attached. Figure 3 As shown, the hand-eye matrix equation is first established, and then solved using the linear least squares method.

[0069] Initial values ​​of the hand-eye matrix Secondly, the initial values ​​of the hand-eye matrix The rotational component R in h2e Euler angle transformation is performed to ensure the orthogonality of the rotation matrix R; then, an initial optimization model H1 for the hand-eye matrix is ​​constructed, and the initial values ​​of the hand-eye matrix are determined using the traditional LM nonlinear optimization method. Perform the first optimization; as shown in the attached document. Figure 4 As shown, the initial optimization model H1 of the hand-eye matrix is ​​finally corrected to obtain the secondary optimization model H2. Then, the LM nonlinear optimization method with the addition of a sample selection method is used to optimize the hand-eye matrix after the first optimization. A second optimization is performed to obtain the optimized hand-eye matrix. The aforementioned method for binocular vision hand-eye calibration using a two-layer nonlinear optimized method includes the following steps:

[0070] Step 1: Solve for the coordinates of feature point P in camera coordinate system 1-3. C Solve for the coordinate expression of feature point P in robot base coordinate system 1-1. B ;

[0071] Step 2: Use the least squares method to solve for the initial values ​​of the hand-eye matrix.

[0072] Step 3: Orthogonalize the hand-eye matrix using inverse Euler angle transformation;

[0073] Step 4: Construct the error function W1 and the loss function L1: Since the coordinates of feature point P in the robot base coordinate system 1-1 remain unchanged, establish the initial optimization model H1;

[0074] Step 5: LM Optimization Solution: Based on the error function W1 and the loss function L1, the LM algorithm is used for the first optimization to obtain the hand-eye matrix after the first optimization. Euler angles representation;

[0075] Step 6: Initialize the iteration-related parameters for the second optimization and substitute them into the hand-eye matrix after the first optimization. These are the initial values ​​for the second optimization iteration;

[0076] Step 7: Determine if the iteration count m and error e meet the stopping requirements. If the error e is less than the set error threshold e′, stop the iteration; otherwise, obtain the step size Δh for this iteration. If the step size Δh is less than the set step size threshold Δh′, stop the iteration; otherwise, update the hand-eye matrix. Parameters related to the iteration of the LM algorithm;

[0077] Step 8: Solve for the distance P from each sample point to the standard point. S The distance is calculated, and sample points that deviate significantly from the standard point are marked based on the distance value. The specific steps are as follows: First, calculate the distance between each sample point and the standard point P. S The distance d between i Then, use d i Construct a new sample set D, and calculate the mean of D. and variance σ d Finally, assuming the distribution of random errors follows a normal distribution, according to the law of the normal distribution, we know... If it follows a standard normal distribution, then based on the boundary values ​​of the confidence interval of the standard normal distribution, we can obtain the condition... Mark the samples outside the t% confidence interval;

[0078] Step 9: Mark the sample points in each iteration; obtain the number of iterations m that have not been filtered. If the number reaches s, filter the sample points. The filtering step refers to traversing all samples to determine the number of times the samples are marked in the above s iterations. Sample points with a number of markings exceeding 0.5*s are regarded as sample points with large errors, and all sample points with large errors are removed from the sample point set.

[0079] The above are merely specific application examples of the present invention and do not constitute any limitation on the scope of protection of the present invention. In addition to the above embodiments, the present invention may have other implementation methods. All technical solutions formed by equivalent substitution or equivalent transformation fall within the scope of protection claimed by the present invention.

Claims

1. A binocular vision hand-eye calibration method using dual-layer nonlinear optimization, comprising at least a binocular camera system, a checkerboard calibration board, and an industrial robot. The binocular camera system is fixed to the end effector of the robot, and the checkerboard calibration board remains in a fixed position during calibration. The binocular camera system refers to a vision system consisting of two cameras mounted in parallel. The method places the checkerboard calibration board within the field of view of the binocular camera system and calibrates the hand-eye matrix by acquiring images from the binocular camera system. The binocular vision hand-eye calibration method uses a two-layer nonlinear optimization algorithm, which includes at least the following steps: Step 1: Simultaneously acquire images of the checkerboard calibration board under the binocular camera vision system: Adjust the robot's end effector pose and acquire images of the checkerboard calibration board captured by the binocular camera vision system. A total of n sets of images are required, where n≥4. Each set contains two images acquired by two cameras respectively. The checkerboard calibration board should be fully presented in the field of view of the binocular camera vision system. The checkerboard calibration board should remain unchanged relative to the robot's base coordinate system (1-1) during the image acquisition process. Step 2: Calculate the coordinates of feature point P in the robot base coordinate system (1-1). B The coordinates of feature point P in the camera coordinate system (1-3) are determined by the rotation and translation relationship between the two fixed cameras in the binocular camera vision system. C Then according to Calculate P C Coordinates P in the robot's base coordinate system (1-1) B The expression; the feature point P refers to a fixed corner point of the chessboard calibration board; the It is calculated from the robot's motion parameters. Let the initial values ​​of the hand-eye matrix be those to be determined. Step 3: Solve for the initial values ​​of the hand-eye matrix. Includes rotational component R h2e Translation component t h2e And ensure the orthogonality of the rotation matrix R; establish the hand-eye matrix equation. In the formula Let P be the homogeneous form of the coordinates of feature point P in the camera coordinate system (1-3) in the sample acquired during the i-th photo capture. The initial values ​​of the hand-eye matrix are solved using the linear least squares method. And the initial hand-eye matrix The rotational component R in h2e Perform Euler angle transformation to ensure the orthogonality of the rotation matrix R; Step 4: Initialize the hand-eye matrix First optimization: Based on the assumption that the coordinates of feature point P in the world coordinate system remain unchanged, an initial optimization model H1 for the hand-eye matrix is ​​constructed. The initial optimization model H1 includes the error function W1 and loss function L1 of the LM optimization, where the error function... The loss function of the initial optimization model H1 In the formula (θ) x ,θ y ,θ z ) is the hand-eye transformation matrix rotational component R h2e Perform Euler angle transformation, which involves the transformation angles of the x, y, and z axes of the camera coordinate system (1-3) around the x, y, and z axes of the robot's end effector TCP coordinate system (1-2) during the rotation. x ,t y ,t z ) is the hand-eye transformation matrix Translation component t h2e The initial values ​​of the hand-eye matrix are determined using the LM nonlinear optimization method based on the error function W1 and the loss function L1. The first optimization is performed, resulting in the optimized hand-eye matrix. Euler angles are represented as: (θ) x ,θ y ,θ z ,t x ,t y ,t z ) T ; Step 5: Optimize the hand-eye matrix after the first optimization. A second optimization with sample selection is performed: the initial optimization model H1 of the hand-eye matrix is ​​modified to obtain the secondary optimization model H2, and then the LM nonlinear optimization method with sample selection function is used to optimize the hand-eye matrix after the first optimization. A second optimization is performed to obtain the optimized hand-eye matrix. Constructing a quadratic optimization model H2 and performing a second optimization includes at least the following steps: Step 5-1: Optimize the hand-eye matrix after the first step. The Euler angle representation is converted into a matrix representation, as shown in Equation 1: Step 5-2: Obtain the standard point P by using the method of averaging s coordinate value in the robot base coordinate system (1-1) as shown in equation 2: Step 5-3: Correct the initial optimization model H1 with the coordinates of the standard point P; the corrected error function W2 is shown in Equation 3: s W2 = (H1 - P)2 The corrected loss function L2 is shown in Equation 4: Step 5-4: Initialize the iteration-related parameters for the second optimization and substitute them into the hand-eye matrix after the first optimization. The initial values ​​for the second optimization iteration are set. The iteration count m and error e are checked to see if they meet the stopping criteria. If the error e is less than the set error threshold e', the iteration stops; otherwise, the step size Δh for this iteration is obtained. If the step size Δh is less than the set step size threshold Δh', the iteration stops; otherwise, the hand-eye matrix is ​​updated. Parameters related to the iteration of the LM algorithm; Step 5-5: Filter the sample points during the iteration process. When the number of iterations reaches the preset threshold s, execute the following sample filtering process: (a) Marking abnormal samples: Traverse all sample points. For each sample point i, calculate its distance to the standard point P. s distance Distance d i <d i The sample points 'd' form a new sample set D,d i ' represents the distance threshold used for outlier removal, and calculates the average value of the sample set D. With standard deviation σ d ; Calculate the standardized distance for each sample point: Determine the critical value u of the standard normal distribution based on the selected confidence level t%. α ,in If |z i |>u α If the sample point is outside the t% confidence interval, then a marker is added to it in this screening. (b) Remove persistently anomalous samples: Count the cumulative number of labels for each sample point in the most recent S consecutive iterations; sample points with a cumulative number of labels exceeding 0.5×S are considered as samples with persistently anomalous behavior and are permanently removed from the current sample set.

2. The binocular vision hand-eye calibration method using dual-layer nonlinear optimization according to claim 1, characterized in that: In step 1, the robot end-effector pose is adjusted, and images of the checkerboard calibration board captured by the binocular camera vision system are acquired. A total of n sets of images are acquired, n≥4, and each set contains two images acquired by two cameras respectively. The checkerboard calibration board can completely represent the field of view of the binocular camera vision system. The corner points of the checkerboard calibration board are distributed as 8 corner points horizontally and 5 corner points vertically. Adjusting the robot end-effector pose means requiring the position and posture of the robot end-effector to change during the shooting process.

3. The binocular vision hand-eye calibration method using dual-layer nonlinear optimization according to claim 1, characterized in that: In step 2, the coordinates of feature point P in the robot base coordinate system (1-1) are calculated. B The calculation process for obtaining the coordinates P is as follows: the coordinates P of feature point P in the camera coordinate system (1-3) are calculated from the intrinsic and extrinsic parameters of the left and right cameras obtained from camera calibration. C ;according to The coordinates of feature point P in the robot base coordinate system (1-1) can be obtained. B .

4. The binocular vision hand-eye calibration method using dual-layer nonlinear optimization according to claim 1, characterized in that: In step 3, the initial value of the hand-eye matrix is ​​solved using the linear least squares method. And the initial hand-eye matrix The rotational component R in h2e Perform Euler angle transformations to ensure the orthogonality of the rotation matrices; this involves at least the following five steps: Step 4-1: For each shooting pose, the feature points P1, P2, ..., P in the n sets of images n Its position in the robot's base coordinate system (1-1) remains unchanged, therefore It can be written as: in, and Let i = 1, 2, ..., n-1, j = i+1, i+2, ..., n be the transformation matrices from the robot end-effector TCP coordinate system (1-2) to the robot base coordinate system (1-1) in the i-th and j-th samples, respectively. Step 4-2: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require and Rewritten as: The matrix equation is obtained from equations 5 and 6: Let A be the left-hand coefficient matrix, X be the unknowns of the hand-eye matrix, and b be the right-hand coefficient matrix; Step 4-3: Solve the matrix equation using the least squares method. The least squares solution to the matrix equation is: X=(A Τ A) -1 A Τ b (8) After solving for the initial hand-eye matrix X, that is, the initial value of the hand-eye matrix... Hand-eye matrix rotational component R h2e Translation component t h2e As shown in Equation 9: Rotational component R h2e Equation 8 shows that the rotation matrix R does not meet the requirement that it must be an orthogonal matrix, and therefore orthogonalization is required. Step 4-4: Initialize the hand-eye matrix Represented as: Hand-eye matrix The rotational component R in h2e Perform Euler angle transformation to determine the transformation angle θ between the x, y, and z axes of the camera coordinate system (1-3) and the x, y, and z axes of the robot's end effector TCP coordinate system (1-2) during rotation. x θ y θ z As shown in Equation 11: Step 4-5: Use the inverse Euler angle transformation to find the rotation matrix R, as shown in Equation 12: Therefore, the initial value of the hand-eye matrix As shown in Equation 13:

Images