An enhanced robust hand-eye calibration method based on adaptive noise information strategy

Abstract

The application provides an enhanced robust hand-eye calibration method based on an adaptive noise information strategy, which selects a group of key frame data as the input of a model, adopts a Zhang Zhengyou calibration method to calibrate the key frame data and the posture information of the corresponding actuator, so as to obtain the initial value of the hand-eye matrix; the initial value of the hand-eye matrix is screened, a plane space conversion operation model is adopted to convert the pixel coordinates of the key frame data into three-dimensional world coordinates by using the initial value of the hand-eye matrix; the key frame data carrying noise information is adjusted by a single parameter iteration method, and each parameter in the hand-eye matrix is adjusted one by one, so that the hand-eye matrix can better adapt to the environmental noise in actual application, and finally a hand-eye matrix model which can better cope with complex actual environment, has high precision, robustness and stability is generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arms and hand-eye calibration technology, and particularly relates to an enhanced robust hand-eye calibration method based on an adaptive noise information strategy. BACKGROUND

[0002] Hand-eye calibration refers to determining the pose transformation relationship between a robot manipulator and a camera. In the field of industrial development, hand-eye calibration plays a key role in intelligent manufacturing and quality control in the field of computer vision. By establishing an accurate relationship calibration between a robot manipulator and a camera vision device, efficient production processes and accurate object grasping can be achieved, thereby improving production efficiency and product quality. In addition, hand-eye calibration also ensures the safety and coordination of robots and operators in human-robot collaboration scenarios, facilitating the joint completion of complex tasks. In the field of augmented reality and virtual reality, hand-eye calibration constitutes a key technical support, providing indispensable support for the realization of immersive virtual experiences and augmented reality effects. In recent years, hand-eye calibration technology has also been increasingly applied in the field of medical devices.

[0003] However, accurately estimating the precise transformation relationship between a robot end effector and a camera is a major challenge for engineers and researchers. Although many scholars have proposed different hand-eye calibration methods, these methods can only produce satisfactory results in ideal numerical simulation environments. In actual applications, the algorithm is affected by noise introduced by camera calibration and robot arm movement, as well as some other unquantifiable scene noise in the scene. However, in the actual application of hand-eye technology, the algorithm model often encounters various different noises in real-world scenarios, which often leads to the inability of the hand-eye matrix calculated using traditional methods to adapt to real-world scenarios. SUMMARY

[0004] The present application overcomes the shortcomings of the prior art and provides an enhanced robust hand-eye calibration method based on an adaptive noise information strategy. The scene noise information of a small number of sample points in the environment is used to fine-tune and optimize the hand-eye matrix initially estimated using classical methods, thereby improving its robustness and accuracy.

[0005] To achieve the above purpose, the present application provides an enhanced robust hand-eye calibration method based on an adaptive noise information strategy, comprising:

[0006] A group of key frame data is selected as the input of the model, and the Zhang Zhengyou calibration method is used to calibrate the key frame data and the corresponding effector pose information to obtain the initial value of the hand-eye matrix;

[0007] The initial value of the hand-eye matrix is ​​screened, and the pixel coordinates of the key frame data are converted into three-dimensional world coordinates using the initial value of the hand-eye matrix using a plane space conversion operation model;

[0008] The key frame data carrying noise information is adjusted one by one in the hand-eye matrix through single parameter iteration to generate the final hand-eye matrix model.

[0009] Preferably, when selecting key frame data, the input frame is selected according to the specified maximum and minimum relative rotation angles of adjacent frames and the maximum and minimum relative translation distances between adjacent frames.

[0010] Preferably, when acquiring key frame data, the steps that need to be performed in advance include:

[0011] Calibrate the tool coordinate system of the robot arm end effector;

[0012] Operate the robot arm to capture the calibration image, select a certain number of sample points in the calibration image, and record the sample point information;

[0013] Get the actual 3D coordinates of the target point.

[0014] Preferably, when screening the initial value of the hand-eye matrix, the initial value of the hand-eye matrix is ​​screened by calculating the coordinates of the upper left corner of the checkerboard calibration plate in the robot arm base coordinate system, and then calculating the mean square error between the coordinates of the upper left corner of the checkerboard calibration plate and the corresponding axis.

[0015] Preferably, the coordinates of the upper left corner of the checkerboard calibration plate in the robot arm base coordinate system are calculated as follows:

[0016]

[0017] Among them, x, y, z are the coordinates of the target point in the robot arm base coordinate system; X w , Y w , Z w is the coordinate of the target point in the world coordinate system; T1 is the transformation matrix between the world coordinate system and the camera coordinate system, X is the hand-eye matrix, and T2 is the transformation matrix between the robot arm end coordinate system and the robot arm base coordinate system.

[0018] Preferably, the formula for calculating the mean square error is:

[0019]

[0020] Among them, e x 、e y 、e z The errors on the three axes are respectively, i 、y i 、zi is the coordinate of the upper left corner of the chessboard calibration plate in the robot arm base coordinate system in the i-th calibration image, is the average number.

[0021] Preferably, the calculation formula of the three-dimensional world coordinates is:

[0022]

[0023] wherein R x is the rotation part of the hand-eye matrix, t x is the translation part of the hand-eye matrix. are the rotation part and the translation part of the conversion matrix between the robot arm end coordinate system and the robot arm base coordinate system, respectively; x, y and z are the three-dimensional coordinates of the target point in the coordinate system; u and v are the pixel coordinates of the target point in the image; Z c is the depth information of the target point in the camera coordinate system; f x , f y , u0 and v0 are the self-provided parameters in the camera intrinsic parameters.

[0024] Preferably, when the key frame data is adjusted by a single parameter, the Nelder-Mead algorithm is used to perform multiple rounds of iterative optimization on each parameter in the hand-eye matrix through the objective function; wherein each round of iteration updates a parameter value that minimizes the objective function.

[0025] Preferably, the calculation formula of the minimized objective function minf(x) is:

[0026]

[0027] wherein, is the actual three-dimensional coordinate of the key frame data; the three-dimensional coordinates (x i , y i , z i ) of each key frame data are calculated by using the hand-eye matrix through the formula of the three-dimensional world coordinates.

[0028] The enhanced robust hand-eye calibration method based on the adaptive noise information strategy provided by the application has the beneficial effects that:

[0029] 1. The scene noise information of a small amount of sample points in the environment is used to obtain the initial value through the initial value obtaining (Tsai) module, that is, the initial value of the hand-eye matrix is obtained by calibrating the data of the input model through the Zhang Zhengyou calibration method, thereby improving the accuracy and reliability of the hand-eye calibration process.

[0030] 2. In order to improve the quality of the initial value, the initial value of the hand-eye matrix is screened, and an initial value with smaller error is obtained, thereby providing data guarantee for forming an initial matrix close to the true value.

[0031] 3. The screened initial value is input into the back projection module, the back projection module adopts a plane space conversion operation model, converts the pixel coordinates of the key frame data into three-dimensional world coordinates by using the initial value of the hand-eye matrix, compares the predicted coordinate value with the detected actual three-dimensional coordinate value, and reverses the optimization of the hand-eye matrix, thereby improving the accuracy of the hand-eye matrix.

[0032] 4. The noise adaptive module adjusts and optimizes each parameter in the hand-eye matrix one by one through single-parameter iteration, so as to obtain the "optimal solution" of each parameter in the hand-eye matrix, and finally obtain a hand-eye matrix model that can better adapt to the environmental noise in actual application and has higher precision, robustness and stability. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A schematic diagram of an enhanced robust hand-eye calibration method based on an adaptive noise information strategy provided by the present application;

[0034] Figure 2 A schematic diagram of obtaining actual three-dimensional coordinates of sample points by the present application; DETAILED DESCRIPTION

[0035] The embodiments of the present application will be described in detail below through specific, concrete examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in the present specification. The present application can also be implemented or applied through other different specific embodiments, and various modifications or changes can be made to the details in the present specification based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following examples and features in the examples can be combined with each other without conflict.

[0036] As shown in Figure 1 The present application provides a hand-eye calibration method, which forms an enhanced robust hand-eye calibration model based on an adaptive noise information strategy, which can be called "ERHEC-NI". The model provided by the present application uses the scene noise information carried by a small amount of sample points in the environment to fine-tune and optimize the hand-eye matrix initially estimated by the classical method, so as to improve its robustness and accuracy. The specific steps of the enhanced robust hand-eye calibration method based on the adaptive noise information strategy provided by the present application include:

[0037] S1: Select a group of key frame data as the input of the model, calibrate the key frame data and the corresponding actuator pose information by using the Zhang Zhengyou calibration method, and obtain the initial value of the hand-eye matrix;

[0038] S2: adopting a plane space conversion operation model to convert pixel coordinates of the key frame data into three-dimensional world coordinates by using the initial value of the hand-eye matrix;

[0039] S3: adjusting each parameter in the hand-eye matrix by single parameter iteration to generate a final hand-eye matrix model.

[0040] Specifically, the application selects a group of appropriate key frames as the input of the model according to the actual application scene, thereby providing a data basis for obtaining the initial value of the model. The executor selected for the key frame is a robot arm, and the robot arm has a camera. The robot arm obtains images in the application scene through the camera, and selects a certain amount of frame images from the images as key frame data for obtaining the initial value of the hand-eye calibration model. The key frame data and the posture information of the corresponding executor (robot arm) are jointly input into the model, and the model is provided with an initial value obtaining (Tsai) module. The Tsai module calibrates the data input into the model by Zhang Zhengyou's calibration method, thereby obtaining the initial value of the hand-eye matrix, and improving the accuracy and reliability of the hand-eye calibration process. In order to improve the quality of the initial value, the initial value of the hand-eye matrix is screened, and an initial value with small error is obtained, thereby providing data guarantee for forming an initial matrix close to the true value. The screened initial value is input into the back projection module, the back projection module adopts a plane space conversion operation model to convert the pixel coordinates of the key frame data into three-dimensional world coordinates by using the initial value of the hand-eye matrix, and compares the predicted coordinate value with the actual three-dimensional coordinate value detected, and reversely optimizes the hand-eye matrix, thereby improving the accuracy of the hand-eye matrix. In the coordinate system of the robot arm, the world coordinate system usually refers to the coordinate system of the base of the robot arm. Then, the key frame data carrying noise information is input into the noise adaptive module, the noise adaptive module adjusts and optimizes each parameter in the hand-eye matrix one by one through single parameter iteration, to obtain the "optimal solution" of each parameter in the hand-eye matrix, and finally obtain a hand-eye matrix model that can better adapt to the environmental noise in the actual application and has higher precision, robustness and stability.

[0041] As shown in Figure 1 , the yellow posture data represents the posture information of the end executor (the end of the robot arm, i.e. the end of the robot tool) corresponding to the calibration image, and the pink preparation data (i.e. the key frame data) represents the prepared sample point information. The information of each sample point includes: its three-dimensional coordinates P in the robot arm base coordinate system, the pixel coordinates p in the image, the depth information Z c (three-dimensional world coordinate information) and the corresponding end executor posture G of the image.

[0042] When obtaining the key frame data, the steps to be performed in advance include:

[0043] Calibrate the tool coordinate frame at the end effector of the robot arm; ensure that the vertex of the tool center point (TCP) is the reference vertex of the end effector coordinate frame.

[0044] The robot arm is operated to capture a calibration image, a certain number of sample points in the calibration image are selected, and sample point information is recorded; in this embodiment, eight sample points are selected as key frame information of the initial value of the hand-eye matrix, and the selection method is as follows: eight sample points distributed in the working plane of the robot arm are selected as preparation data, and the eight sample points are distributed as evenly as possible in the working plane, and the information of the eight sample points is recorded.

[0045] The actual three-dimensional coordinates of the target point are obtained. The actual coordinate acquisition method of the target point is: the TCP vertex of the robot arm is in contact with the target point, and the three-dimensional coordinates of the TCP vertex in the robot arm base coordinate system can be captured, as shown in Figure 2 .

[0046] In this embodiment, when selecting key frame data, the input frame is selected according to the specified maximum and minimum relative rotation angles of adjacent frames and the maximum and minimum relative translation distances between adjacent frames.

[0047] Specifically, when selecting key frame data, the greater the range difference between each coordinate and image data, the more visual information can be collected, the robustness of the final result is enhanced, and the result can adapt to a larger spatial range when sufficient viewing is provided. Therefore, according to the above selection method, sufficient parallax can be provided while maintaining the stability of the algorithm.

[0048] In this embodiment, the selected key frame data and the corresponding end effector pose information are input into the Tsai module, which integrates the Zhang Zhengyou calibration method. The key frame data and the pose information are calibrated through the camera (camera) to obtain the initial value of the hand-eye matrix. The Zhang Zhengyou calibration method is to detect the corner points from the image by shooting a chessboard pattern of known size, establish the relationship between the two-dimensional image and the three-dimensional world coordinates, and then solve the camera's internal and external parameters by using linear and nonlinear optimization, thereby correcting the lens distortion. However, the reason for choosing the Zhang Zhengyou calibration method is that it has the strongest stability and is the easiest to implement in camera calibration methods.

[0049] In this embodiment, when screening the initial value of the hand-eye matrix, the coordinates of the upper left corner of the checkerboard calibration plate in the robot arm base coordinate system are calculated, and the mean square error between the upper left corner of the checkerboard calibration plate and the corresponding axis is calculated to screen the initial value of the hand-eye matrix.

[0050] Specifically, in order to improve the accuracy and precision of the model, the initial value of the hand-eye matrix needs to be screened, and the filtering and screening method is as follows: the coordinates of the upper left corner of the checkerboard calibration board (i.e. the target point coordinates) in the robot arm base coordinate system are calculated (assuming that the origin of the world coordinate system is represented as W0=[0,0,0,1]), and then the mean square error between the target point coordinates and each axis is calculated, and the most accurate initial value of the hand-eye matrix is obtained according to the mean square error of each axis. A large number of experiments have proved that when the error of each axis is less than 3mm, the obtained data is the most accurate initial value of the hand-eye matrix, which provides a guiding reference for the screening of the initial value to a certain extent.

[0051] In this embodiment, the calculation method of the coordinates of the upper left corner of the checkerboard calibration board in the robot arm base coordinate system (target point coordinates) is as follows:

[0052]

[0053] Wherein, x, y, z are the coordinates of the target point in the robot arm base coordinate system; X w , Y w , Z w are the coordinates of the target point in the world coordinate system; T1 is the conversion matrix between the world coordinate system and the camera coordinate system, X is the hand-eye matrix (conversion matrix between the camera coordinate system and the robot arm end coordinate system), and T2 is the conversion matrix between the robot arm end (the vertex of the tool end of the robot arm) coordinate system and the robot arm base coordinate system.

[0054] The calculation formula of the mean square error between the target point coordinates and each axis is as follows:

[0055]

[0056] Wherein, e x , e y , e z are the errors of the three axes, x i , y i , z i are the coordinates of the upper left corner of the checkerboard calibration board in the robot arm base coordinate system in the i-th calibration image, is the average.

[0057] In this embodiment, in order to improve the accuracy of the hand-eye calibration, the three-dimensional coordinates of the previously sampled points are obtained by using the hand-eye matrix initial value obtained in the previous part, and the corresponding three-dimensional world coordinates are obtained by using the two-dimensional to three-dimensional plane space conversion operation model, so as to compare with the actual three-dimensional coordinate values in the subsequent and optimize the hand-eye matrix in reverse.

[0058] Wherein, the calculation formula of the three-dimensional world coordinates is as follows:

[0059]

[0060] wherein R x is the rotation part of the hand-eye matrix, t x is the translation part of the hand-eye matrix; are the rotation part and the translation part of the transformation matrix between the robot arm end coordinate system and the robot arm base coordinate system, respectively; x, y, z are the three-dimensional coordinates of the target point in the coordinate system; u and v are the pixel coordinates of the target point in the image; Z c is the depth information of the target point in the camera coordinate system; f x , f y , u0, v0 are the self-provided parameters in the camera intrinsic parameters.

[0061] In the present embodiment, when the key frame data is adjusted by a single parameter, the Nelder-Mead algorithm is used to perform multiple rounds of iterative optimization on each parameter in the target function of the hand-eye matrix; wherein each round of iteration updates a parameter value of the minimized target function.

[0062] Specifically, in the process of optimizing the hand-eye matrix X, the reason for adopting separate optimization for each parameter instead of simultaneously optimizing multiple parameters is to avoid potential problems commonly seen in multivariate optimization problems, such as local optimal solution and low computational efficiency. In the optimization process, the 12 parameters in the rotation and translation parts of the hand-eye matrix X are considered as a whole, and the process of optimizing the 12 parameters one by one constitutes a single round of optimization process. Each round of optimization finds 12 optimal solutions for the minimized target function one by one through the Nelder-Mead algorithm. In order to improve the accuracy, multiple rounds of optimization iterations are performed on the hand-eye matrix X. This iterative method helps to refine the parameter values and improves the accuracy in the calibration process.

[0063] wherein each iteration updates a parameter value of the minimized target function, and the calculation formula of the minimized target function minf(x) is:

[0064]

[0065] wherein, is the actual three-dimensional coordinate of the key frame data; through the formula of the three-dimensional world coordinate, the three-dimensional coordinate (x i , y i , z i ) of each key frame data is calculated by using the hand-eye matrix. The present application finds the "optimal solution" of each parameter in the hand-eye matrix by minimizing the target function, so that the ERHEC-NI model of the present application can better cope with complex actual environments and has higher precision, robustness and stability.

[0066] Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

Claims

1. An enhanced robust hand-eye calibration method based on adaptive noise information strategy, characterized in that: include: A set of keyframe data is selected as the input of the model, and the keyframe data and the corresponding actuator posture information are calibrated using Zhang Zhengyou's calibration method to obtain the initial value of the hand-eye matrix; The initial value of the hand-eye matrix is ​​screened, and the pixel coordinates of the key frame data are converted into three-dimensional world coordinates using the initial value of the hand-eye matrix using a plane space conversion operation model; When screening the initial value of the hand-eye matrix, the coordinates of the upper left corner of the checkerboard calibration plate in the robot arm base coordinate system are calculated, and then the mean square error between the coordinates of the upper left corner of the checkerboard calibration plate and the corresponding axis is calculated to screen the initial value of the hand-eye matrix to obtain the initial value with the smallest error; The key frame data carrying noise information is adjusted one by one in the hand-eye matrix through single parameter iteration to generate the final hand-eye matrix model.

2. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 1, characterized in that: When selecting keyframe data, input frames are selected based on the specified maximum and minimum relative rotation angles between adjacent frames and the maximum and minimum relative translation distances between adjacent frames.

3. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 2, characterized in that: When acquiring the key frame data, the steps that need to be performed in advance include: Calibrate the tool coordinate system of the robot arm end effector; Operate the robot arm to capture the calibration image, select a certain number of sample points in the calibration image, and record the sample point information; Get the actual 3D coordinates of the target point.

4. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 1, characterized in that: The coordinates of the upper left corner of the checkerboard calibration plate in the robot arm base coordinate system are calculated as follows: ； in, , , is the coordinate of the target point in the robot arm base coordinate system; , , is the coordinate of the target point in the world coordinate system; is the transformation matrix between the world coordinate system and the camera coordinate system, is the hand-eye matrix, is the transformation matrix between the robot arm end coordinate system and the robot arm base coordinate system.

5. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 1, characterized in that: The calculation formula of the mean square error is: ； ； ； in, 、 、 are the errors on the three axes, 、 、 Respectively The coordinates of the upper left corner of the checkerboard calibration plate in the calibration image in the robot arm base coordinate system, is the average.

6. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 1, characterized in that: The calculation formula of the three-dimensional world coordinate is: ； in, is the rotation part of the hand-eye matrix, is the translation part of the hand-eye matrix; 、 They are the rotation and translation parts of the transformation matrix between the robot arm end coordinate system and the robot arm base coordinate system respectively; 、 、 are the three-dimensional coordinates of the target point in the coordinate system; and is the pixel coordinate of the target point in the image; is the depth information of the target point in the camera coordinate system; 、 、 、 It is a built-in parameter in the camera's internal parameters.

7. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 1, characterized in that: When the key frame data is subjected to single parameter adjustment, the Nelder-Mead algorithm is used to perform multiple rounds of iterative optimization on each parameter in the hand-eye matrix through the objective function; wherein each round of iteration updates a parameter value that minimizes the objective function.

8. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 7, characterized in that: The minimization objective function The calculation formula is: ； in, The actual three-dimensional coordinates of the key frame data; the three-dimensional coordinates of each key frame data are calculated using the hand-eye matrix through the three-dimensional world coordinate formula .

Images