Enhanced robust hand-eye calibration method based on adaptive noise information strategy

Abstract

The invention provides an enhanced robust hand-eye calibration method based on an adaptive noise information strategy, and the method comprises the steps: selecting a group of key frame data as the input of a model, and calibrating the key frame data and the attitude information of a corresponding actuator through employing a Zhang Zhengyou calibration method, so as to obtain an initial value of a hand-eye matrix; screening initial values of the hand-eye matrix, and converting pixel coordinates of the key frame data into three-dimensional world coordinates by using the initial values of the hand-eye matrix by adopting a plane space conversion operation model; according to the method, all parameters in a hand-eye matrix are adjusted one by one according to key frame data carrying noise information in a single-parameter iteration mode, so that the hand-eye matrix can better adapt to environmental noise in actual application, and finally a hand-eye matrix model which can better cope with a complex actual environment and has high precision, robustness and stability is generated.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arms and hand-eye calibration technology, and in particular to an enhanced robust hand-eye calibration method based on an adaptive noise information strategy. Background Art

[0002] Hand-eye calibration refers to determining the position transformation relationship between the robot manipulator and the camera. In the field of industrial development, hand-eye calibration plays a key role in intelligent manufacturing and quality control in the field of computer vision. By establishing a precise relationship calibration between the robot manipulator and the camera vision device, the production process can be made efficient and the object can be grasped accurately, thereby improving production efficiency and product quality. In addition, hand-eye calibration also ensures the safety and coordination of robots and operators in human-machine collaboration scenarios, and promotes the joint completion of complex tasks. In the field of augmented reality and virtual reality, hand-eye calibration constitutes a key technical support, providing indispensable support for the realization of immersive virtual experience and augmented reality effects. In recent years, the application of hand-eye calibration technology in the field of medical devices has also been developing.

[0003] However, accurately estimating the precise transformation relationship between the robot end effector and the camera is a major challenge for engineers and researchers. Although many scholars have proposed different hand-eye calibration methods, these methods can only produce satisfactory results in an ideal numerical simulation environment. In practical applications, the algorithm will be affected by the noise introduced by camera calibration and the movement of the robot arm, as well as some other unquantifiable scene noise in the scene. However, in the actual application of hand-eye technology, the algorithm model often encounters various noises in real scenes. These noises often cause the hand-eye matrix calculated by traditional methods in the system to be unable to adapt to real scenes. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides an enhanced robust hand-eye calibration method based on an adaptive noise information strategy, which utilizes the scene noise information of a small number of sample points in the environment to fine-tune and optimize the hand-eye matrix initially estimated by the classical method, so as to improve its robustness and accuracy.

[0005] In order to achieve the above object, the present invention provides an enhanced robust hand-eye calibration method based on an adaptive noise information strategy, comprising:

[0006] A set of key frame data is selected as the input of the model, and the key frame data and the posture information of the corresponding actuator are calibrated using Zhang Zhengyou's calibration method to obtain the initial value of the hand-eye matrix;

[0007] The initial value of the hand-eye matrix is ​​screened, and the pixel coordinates of the key frame data are converted into three-dimensional world coordinates using the initial value of the hand-eye matrix by using a plane space conversion operation model;

[0008] The key frame data carrying noise information is adjusted one by one in the hand-eye matrix through a single parameter iteration method to generate the final hand-eye matrix model.

[0009] Preferably, when selecting key frame data, the input frame is selected according to the specified maximum and minimum relative rotation angles of adjacent frames and the maximum and minimum relative translation distances between adjacent frames.

[0010] Preferably, when acquiring key frame data, the steps that need to be performed in advance include:

[0011] Calibrate the tool coordinate system where the robot arm end effector is located;

[0012] Operate the robot arm to capture the calibration image, select a certain number of sample points in the calibration image, and record the sample point information;

[0013] Get the actual 3D coordinates of the target point.

[0014] Preferably, when screening the initial value of the hand-eye matrix, the initial value of the hand-eye matrix is ​​screened by calculating the coordinates of the upper left corner of the checkerboard calibration plate in the robot arm base coordinate system, and then calculating the mean square error between the upper left corner coordinates of the checkerboard calibration plate and the corresponding axis.

[0015] Preferably, the coordinates of the upper left corner of the checkerboard calibration plate in the robot arm base coordinate system are calculated as follows:

[0016]

[0017] Among them, x, y, z are the coordinates of the target point in the robot arm base coordinate system; X w , Y w , Z w is the coordinate of the target point in the world coordinate system; T1 is the transformation matrix between the world coordinate system and the camera coordinate system, X is the hand-eye matrix, and T2 is the transformation matrix between the robot arm end coordinate system and the robot arm base coordinate system.

[0018] Preferably, the calculation formula of the mean square error is:

[0019]

[0020] Among them, e x 、e y 、e z are the errors on the three axes, x i ,y i 、zi are the coordinates of the upper left corner of the checkerboard calibration plate in the i-th calibration image in the robot arm base coordinate system, is the average.

[0021] Preferably, the calculation formula of the three-dimensional world coordinates is:

[0022]

[0023] Among them, R x is the rotation part of the hand-eye matrix, t x is the translation part of the hand-eye matrix; are the rotation part and translation part of the transformation matrix between the robot arm end coordinate system and the robot arm base coordinate system; x, y, z are the three-dimensional coordinates of the target point in the coordinate system; u and v are the pixel coordinates of the target point in the image; Z c is the depth information of the target point in the camera coordinate system; f x 、f y , u0, and v0 are the built-in parameters in the camera's intrinsic parameters.

[0024] Preferably, when a single parameter is adjusted for key frame data, the Nelder-Mead algorithm is used to perform multiple rounds of iterative optimization on each parameter in the hand-eye matrix through the objective function; wherein each round of iteration updates a parameter value that minimizes the objective function.

[0025] Preferably, the calculation formula for minimizing the objective function minf(x) is:

[0026]

[0027] in, is the actual 3D coordinate of the key frame data; the 3D coordinate of each key frame data is calculated using the hand-eye matrix through the formula of the 3D world coordinate (x i ,y i , z i ).

[0028] The enhanced robust hand-eye calibration method based on an adaptive noise information strategy provided by the present invention has the following significant effects compared with the prior art:

[0029] 1. Utilize the scene noise information of a small number of sample points in the environment and obtain the initial value through the initial value acquisition (Tsai) module. That is, calibrate the input model data through the Zhang Zhengyou calibration method to obtain the initial value of the hand-eye matrix, thereby improving the accuracy and reliability of the hand-eye calibration process.

[0030] 2. In order to improve the quality of the initial values, the initial values ​​of the hand-eye matrix are screened to obtain initial values ​​with smaller errors, which provides data guarantee for forming an initial matrix that approximates the true value.

[0031] 3. Input the selected initial values ​​into the back-projection module. The back-projection module adopts a plane space conversion operation model to convert the pixel coordinates of the key frame data into three-dimensional world coordinates using the initial values ​​of the hand-eye matrix. The predicted coordinate values ​​are compared with the actual three-dimensional coordinate values ​​detected, and the hand-eye matrix is ​​optimized in reverse to improve its accuracy.

[0032] 4. The noise adaptation module adjusts and optimizes each parameter in the hand-eye matrix one by one through single parameter iteration to obtain the "optimal solution" for each parameter in the hand-eye matrix, and finally obtains a hand-eye matrix model that can better adapt to environmental noise in practical applications and has higher accuracy, robustness and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of an enhanced robust hand-eye calibration method based on an adaptive noise information strategy provided by the present invention;

[0034] Figure 2 A schematic diagram of obtaining the actual three-dimensional coordinates of the sample points in the present invention; DETAILED DESCRIPTION

[0035] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0036] like Figure 1 As shown, the present invention provides a hand-eye calibration method, through which an enhanced robust hand-eye calibration model based on an adaptive noise information strategy is formed, which can be called "ERHEC-NI". The model provided by the present invention uses the scene noise information of a small number of sample points in the environment to fine-tune and optimize the hand-eye matrix initially estimated by the classical method to improve its robustness and accuracy. The present invention provides an enhanced robust hand-eye calibration method based on an adaptive noise information strategy, and its specific steps include:

[0037] S1: Select a set of key frame data as the input of the model, use Zhang Zhengyou calibration method to calibrate the key frame data and the posture information of the corresponding actuator, and obtain the initial value of the hand-eye matrix;

[0038] S2: using a plane space conversion operation model to convert the pixel coordinates of the key frame data into three-dimensional world coordinates using the initial value of the hand-eye matrix;

[0039] S3: The key frame data carrying noise information is adjusted one by one in the hand-eye matrix through a single parameter iteration method to generate the final hand-eye matrix model.

[0040] Specifically, the present invention selects a group of appropriate key frames as the input of the model according to the actual application scenario, providing a data basis for the model to obtain the initial value. Among them, the actuator for selecting the key frame is a robot arm, and the robot arm has a camera. The robot arm obtains the image in the application scenario through the camera, and selects a certain amount of frame images from the image as the key frame data for obtaining the initial value of the hand-eye calibration model. The key frame data and the posture information of the corresponding actuator (robot arm) are input into the model together. The model is provided with an initial value acquisition (Tsai) module. The Tsai module calibrates the data of the input model through the Zhang Zhengyou calibration method to obtain the initial value of the hand-eye matrix, thereby improving the accuracy and reliability of the hand-eye calibration process. In order to improve the quality of the initial value, the initial value of the hand-eye matrix is ​​screened, and an initial value with a small error can be obtained, which provides data guarantee for forming an initial matrix close to the true value. The selected initial values ​​are input into the back-projection module. The back-projection module uses a plane space conversion operation model to convert the pixel coordinates of the key frame data into three-dimensional world coordinates using the initial values ​​of the hand-eye matrix. The predicted coordinate values ​​are compared with the actual three-dimensional coordinate values ​​detected, and the hand-eye matrix is ​​optimized in reverse to improve the accuracy of the hand-eye matrix. In the coordinate system of the robot, the world coordinate system usually refers to the coordinate system of the robot base. The key frame data carrying noise information is then input into the noise adaptive module. The noise adaptive module adjusts and optimizes each parameter in the hand-eye matrix one by one through single parameter iteration to obtain the "optimal solution" for each parameter in the hand-eye matrix, and finally obtains a hand-eye matrix model that can better adapt to environmental noise in practical applications and has higher accuracy, robustness and stability.

[0041] like Figure 1 As shown in the figure, the yellow posture data represents the posture information of the end effector (the end of the robot arm, i.e. the end of the robot arm tool) corresponding to the calibration image, and the pink preparation data (i.e. key frame data) represents the prepared sample point information. The information of each sample point includes: its three-dimensional coordinates P in the robot arm base coordinate system, the pixel coordinates p in the image, and the depth information Z c (3D world coordinate information) and the corresponding end effector posture G of the image.

[0042] When obtaining key frame data, the steps that need to be performed in advance include:

[0043] Calibrate the tool coordinate system at the end effector of the robot arm; make sure the vertex of the tool center point (TCP) is the reference vertex of the end effector coordinate system.

[0044] Operate the robot arm to capture the calibration image, select a certain number of sample points in the calibration image, and record the sample point information. In this embodiment, eight sample points are selected as the key frame information of the initial value of the hand-eye matrix, and the selection method is: evenly select eight sample points distributed in the working plane of the robot arm as preparation data, try to make the eight sample points evenly distributed in the working plane and record the information of the eight sample points.

[0045] Get the actual three-dimensional coordinates of the target point. The actual coordinates of the target point are obtained by operating the TCP vertex of the robot arm to contact the target point, and then the three-dimensional coordinates of the TCP vertex in the robot arm base coordinate system can be captured, such as Figure 2 shown.

[0046] In this embodiment, when selecting key frame data, the input frame is selected according to the specified maximum and minimum relative rotation angles of adjacent frames and the maximum and minimum relative translation distances between adjacent frames.

[0047] Specifically, when selecting key frame data, when sufficient parallax is provided, the greater the range difference between each coordinate and image data, the more visual information can be collected, the more robust the final result is, and the result can be adapted to a larger spatial range. Therefore, according to the selection method set above, sufficient parallax can be provided and the stability of the algorithm can be maintained.

[0048] In this embodiment, the selected key frame data and the posture information of the corresponding end effector are input into the Tsai module. The Tsai module integrates the Zhang Zhengyou calibration method, and the key frame data and posture information are calibrated through the camera (camera) to obtain the initial value of the hand-eye matrix. Among them, the Zhang Zhengyou calibration method is to shoot a chessboard pattern of known size, detect corners from the image, establish the relationship between the two-dimensional image and the three-dimensional world coordinates, and then use linear and nonlinear optimization to solve the camera's intrinsic and extrinsic parameters to correct lens distortion. However, the reason for choosing Zhang Zhengyou's calibration method is that it has the strongest stability and is easiest to implement among the camera calibration methods.

[0049] In this embodiment, when screening the initial value of the hand-eye matrix, the initial value of the hand-eye matrix is ​​screened by calculating the coordinates of the upper left corner of the checkerboard calibration plate in the robot arm base coordinate system, and then calculating the mean square error between the upper left corner coordinates of the checkerboard calibration plate and the corresponding axis.

[0050] Specifically, in order to improve the accuracy and precision of the model, the initial value of the hand-eye matrix needs to be screened. The filtering method is: by calculating the coordinates of the upper left corner of the checkerboard calibration plate (i.e., the target point coordinates) (assuming it is the origin of the world coordinate system, expressed as W0 = [0,0,0,1]) in the robot arm base coordinate system, and then calculating the mean square error between the target point coordinates and each axis, the most accurate initial value of the hand-eye matrix can be verified and evaluated based on the mean square error obtained for each axis. Among them, after a large number of experiments, it has been proved that the data obtained when the error of each axis is less than 3 mm is the most accurate initial value of the hand-eye matrix, which provides a guiding reference for the screening of initial values ​​to a certain extent.

[0051] In this embodiment, the coordinates of the upper left corner of the checkerboard calibration plate in the robot arm base coordinate system (target point coordinates) are calculated as follows:

[0052]

[0053] Among them, x, y, z are the coordinates of the target point in the robot arm base coordinate system; X w , Y w , Z w is the coordinate of the target point in the world coordinate system; T1 is the transformation matrix between the world coordinate system and the camera coordinate system, X is the hand-eye matrix (the transformation matrix between the camera coordinate system and the robot arm end coordinate system), and T2 is the transformation matrix between the robot arm end (the vertex of the tool end of the robot arm) coordinate system and the robot arm base coordinate system.

[0054] The calculation formula for the mean square error between the target point coordinates and each axis is:

[0055]

[0056] Among them, e x 、e y 、e z are the errors on the three axes, x i ,y i 、z i are the coordinates of the upper left corner of the checkerboard calibration plate in the i-th calibration image in the robot arm base coordinate system, is the average.

[0057] In this embodiment, in order to improve the accuracy of hand-eye calibration, a two-dimensional to three-dimensional plane space conversion operation model is used to convert the three-dimensional coordinates of the pre-sampled points into corresponding three-dimensional world coordinates using the initial values ​​of the hand-eye matrix obtained in the previous part, so as to subsequently compare them with their actual three-dimensional coordinate values ​​and reversely optimize the hand-eye matrix.

[0058] Among them, the calculation formula of the three-dimensional world coordinates is:

[0059]

[0060] Among them, R x is the rotation part of the hand-eye matrix, t x is the translation part of the hand-eye matrix; are the rotation part and translation part of the transformation matrix between the robot arm end coordinate system and the robot arm base coordinate system; x, y, z are the three-dimensional coordinates of the target point in the coordinate system; u and v are the pixel coordinates of the target point in the image; Z c is the depth information of the target point in the camera coordinate system; f x 、f y , u0, and v0 are the built-in parameters in the camera's intrinsic parameters.

[0061] In this embodiment, when a single parameter is adjusted for key frame data, the Nelder-Mead algorithm is used to perform multiple rounds of iterative optimization on each parameter in the hand-eye matrix through the objective function; wherein each round of iteration updates a parameter value that minimizes the objective function.

[0062] Specifically, in the process of optimizing the hand-eye matrix X, the reason for optimizing each parameter separately instead of optimizing multiple parameters at the same time is to avoid potential problems common in multivariable optimization problems, such as local optimal solutions and low computational efficiency. During the optimization process, the 12 parameters in the rotation and translation part of the hand-eye matrix X are regarded as a whole, and the process of optimizing the 12 parameters in turn constitutes a single round of optimization process. Each round of optimization uses the Nelder-Mead algorithm to find 12 optimal solutions in turn to minimize the objective function. In order to improve accuracy, multiple rounds of optimization iterations are performed on the hand-eye matrix X. This iterative method helps to refine the parameter values ​​and improve accuracy during the calibration process.

[0063] Among them, each iteration updates a parameter value of the minimized objective function, and the calculation formula for minimizing the objective function minf(x) is:

[0064]

[0065] in, is the actual three-dimensional coordinate of the key frame data; the three-dimensional coordinate (x i ,y i ,z i ). The present invention finds the "optimal solution" of each parameter in the hand-eye matrix by minimizing the objective function, so that the ERHEC-NI model of the present invention can better cope with complex practical environments and has higher accuracy, robustness and stability.

[0066] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

Claims

1. An enhanced robust hand-eye calibration method based on adaptive noise information strategy, characterized in that: include: A set of key frame data is selected as the input of the model, and the key frame data and the posture information of the corresponding actuator are calibrated using Zhang Zhengyou's calibration method to obtain the initial value of the hand-eye matrix; The initial value of the hand-eye matrix is ​​screened, and the pixel coordinates of the key frame data are converted into three-dimensional world coordinates using the initial value of the hand-eye matrix by using a plane space conversion operation model; The key frame data carrying noise information is adjusted one by one in the hand-eye matrix through a single parameter iteration method to generate the final hand-eye matrix model.

2. According to the enhanced robust hand-eye calibration method based on adaptive noise information strategy described in claim 1, when selecting key frame data, the input frame is selected according to the specified maximum and minimum relative rotation angles of adjacent frames and the maximum and minimum relative translation distances between adjacent frames.

3. According to the enhanced robust hand-eye calibration method based on adaptive noise information strategy in claim 2, when acquiring the key frame data, the steps to be performed in advance include: Calibrate the tool coordinate system where the robot arm end effector is located; Operate the robot arm to capture the calibration image, select a certain number of sample points in the calibration image, and record the sample point information; Get the actual 3D coordinates of the target point.

4. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 1, characterized in that: When screening the initial value of the hand-eye matrix, the initial value of the hand-eye matrix is ​​screened by calculating the coordinates of the upper left corner of the checkerboard calibration plate in the robot arm base coordinate system, and then calculating the mean square error between the upper left corner coordinates of the checkerboard calibration plate and the corresponding axis.

5. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 4 is characterized in that: The coordinates of the upper left corner of the chessboard calibration plate in the robot arm base coordinate system are calculated as follows: Where x, y, and z are the coordinates of the target point in the robot arm base coordinate system; w , Y w , Z w is the coordinate of the target point in the world coordinate system; T1 is the transformation matrix between the world coordinate system and the camera coordinate system, X is the hand-eye matrix, and T2 is the transformation matrix between the robot arm end coordinate system and the robot arm base coordinate system.

6. According to the enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 4, the calculation formula of the mean square error is: in, e x 、e y 、e z are the errors on the three axes, x i ,y i 、z i are the coordinates of the upper left corner of the checkerboard calibration plate in the i-th calibration image in the robot arm base coordinate system, is the average.

7. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 1, characterized in that: The calculation formula of the three-dimensional world coordinates is: Among them, R x is the rotation part of the hand-eye matrix, t x is the translation part of the hand-eye matrix; are the rotation part and translation part of the transformation matrix between the robot arm end coordinate system and the robot arm base coordinate system; x, y, z are the three-dimensional coordinates of the target point in the coordinate system; u and v are the pixel coordinates of the target point in the image; Z c is the depth information of the target point in the camera coordinate system; f x 、f y , u0, and v0 are the built-in parameters in the camera's intrinsic parameters.

8. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 1, characterized in that: When the key frame data is adjusted for a single parameter, the Nelder-Mead algorithm is used to perform multiple rounds of iterative optimization on each parameter in the hand-eye matrix through the objective function; wherein each round of iteration updates a parameter value that minimizes the objective function.

9. The enhanced robust hand-eye calibration method based on adaptive noise information strategy according to claim 8, characterized in that: The calculation formula of the minimization objective function minf(x) is: in, is the actual 3D coordinate of the key frame data; the 3D coordinate of each key frame data is calculated using the hand-eye matrix through the formula of the 3D world coordinate (x i ,y i ,z i ).

Images