Hard refueling application-oriented hand-eye calibration optimization method and system

Abstract

The invention discloses a hard refueling application-oriented hand-eye calibration optimization method and system, and belongs to the technical field of hand-eye calibration optimizing.A motion capture system is used for obtaining the pose of a camera rigid body in a global coordinate system, and the pose of a calibration plate in a camera optical center coordinate system is obtained through a computer vision algorithm; calculating an estimated pose of the calibration board in a global coordinate system in combination with a hand-eye calibration matrix obtained by a Tsai method; a mean square error is adopted to construct a loss function, the difference between an estimated pose and a real pose is measured, and finally, hand-eye calibration matrix parameters are adjusted by means of deep learning to optimize the loss function, so that a more accurate hand-eye calibration matrix is obtained, and the error problem caused by matrix deviation in a traditional method is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of hand-eye calibration optimization, and particularly relates to a hand-eye calibration optimization method and system for rigid refueling applications. Background Technique

[0002] With the continuous in-depth development of automation and intelligence in the industrial field, various high-precision positioning technologies have gradually moved from the laboratory to the actual production site. Especially in rigid refueling applications, the precise docking between the fuel dispenser and the fuel receiver not only affects the safety and efficiency of the refueling process but also directly impacts the service life and maintenance cost of the equipment. For this reason, vision-based three-dimensional positioning technology has gradually been introduced into rigid refueling applications to verify and improve the positioning accuracy of the system.

[0003] A vision positioning system usually consists of multiple parts, including an image acquisition module, an image processing and feature extraction module, a pose estimation module, and a sensor data fusion module. Among them, hand-eye calibration, as an important bridge connecting the vision system and the control system, its accuracy directly affects the accuracy of subsequent three-dimensional positioning and control. In the rigid refueling scenario, the refueling arm of the fuel dispenser or the fuel receiver usually needs to maintain high-precision positioning within a large working distance, while traditional hand-eye calibration methods often have cumulative errors in this process, which has become a key bottleneck restricting the improvement of system accuracy.

[0004] The role of hand-eye calibration is to determine the conversion relationship between the camera coordinate system and the robot (or robotic arm) coordinate system, that is, to solve the hand-eye calibration matrix. This matrix describes the relative position and attitude relationship between the camera and the robot and is the basis for realizing vision guidance and control. Precise hand-eye calibration can ensure that the object pose information obtained by the vision system is accurately mapped into the robot coordinate system, thereby achieving precise operation and control.

[0005] The traditional Tsai method solves the hand-eye calibration matrix using the pose relationship between the movement of the robot hand and the image captured by the camera; formulates the hand-eye calibration problem as a matrix equation AX = XB, where A and B respectively represent the movement change matrices of the robot arm or the camera in two different poses, and X is the hand-eye calibration matrix to be solved. Usually, the rotation and translation parts are solved separately, first finding the rotation and then the translation to reduce the computational complexity. According to the constraint relationship A i X = XB i , and minimizes the objective function through the least squares method To solve the hand-eye calibration matrix, although this method can achieve the coordinate transformation between the basic camera and the receiver aircraft, due to the slight deviation of the rotation matrix, it will lead to significant errors in practical applications. For example, when the relative distance between the refueling equipment of the tanker and the receiver aircraft is about 1.8 meters, the final positioning accuracy will introduce an error of about 2.4 centimeters, which is unacceptable in the rigid refueling task with high precision requirements.

[0006] The hand-eye calibration method based on deep learning utilizes the powerful learning ability of the neural network to directly infer the spatial transformation relationship between the camera and the end effector of the robotic arm from the image data; first, collect the image data and corresponding pose information of the robotic arm in different poses to ensure data diversity and coverage; then design a neural network architecture suitable for the hand-eye calibration task, often using a convolutional neural network (CNN) to process image features, the network inputs the camera image and the pose information of the robotic arm, and outputs the spatial transformation relationship between the camera and the end of the robotic arm; use the collected data to train the network, define an appropriate loss function and use an optimization algorithm to update the parameters. However, this method has the disadvantages that a large amount of labeled data is required for training, which is difficult to obtain in practical applications; the training process requires high computing resources and requires high-performance hardware support; the generalization ability is weak, and the performance of the model in unseen scenarios is not as good as that in the training data. Summary of the Invention

[0007] To solve the above problems existing in the prior art, the present invention is achieved through the following technical solutions:

[0008] In the first aspect of the present invention, a hand-eye calibration optimization method for rigid refueling applications is provided, including the following steps:

[0009] Estimated pose determination step: Calculate the estimated pose of the calibration board in the global coordinate system according to the pose of the camera rigid body in the global coordinate system, the hand-eye calibration matrix, and the pose of the calibration board in the camera optical center coordinate system;

[0010] Loss function construction step: Construct a loss function according to the estimated pose of the calibration board in the global coordinate system and the pose of the calibration board in the global coordinate system;

[0011] Deep learning optimization step: Iteratively optimize the loss function by adjusting the hand-eye calibration matrix parameters, and determine the optimized hand-eye calibration matrix according to the optimized loss function.

[0012] The calculation formula for the estimated pose of the calibration board in the global coordinate system in the estimated pose determination step is as follows:

[0013]

[0014] Wherein, is the pose of the camera rigid body in the global coordinate system, is the hand-eye calibration matrix, is the pose of the calibration board in the camera optical center coordinate system, is the estimated pose of the calibration board in the global coordinate system.

[0015] The steps of constructing the loss function also include constructing the loss function using the mean square error. The calculation formula of the loss function is as follows:

[0016]

[0017] where, is the estimated pose of the calibration board in the global coordinate system for the i-th sample, is the pose of the calibration board in the global coordinate system for the i-th sample, Loss is the loss function, N is the number of samples, and ∥∥ 2 represents the square of the two-norm of the vector difference.

[0018] The hand-eye calibration matrix is obtained by the Tsai method.

[0019] The hand-eye calibration matrix is:

[0020]

[0021] where, is the hand-eye calibration matrix, the parameter R is the rotation matrix, and the parameter t is the translation vector, which is used to describe the translation relationship of the camera coordinate system relative to the robot coordinate system;

[0022] The rotation matrix R and the translation vector t are used to iteratively optimize the loss function.

[0023] The pose of the calibration board in the camera optical center coordinate system is detected by a computer vision algorithm.

[0024] The pose of the camera rigid body in the global coordinate system and the pose of the calibration board in the global coordinate system are obtained by tracking with a motion capture system.

[0025] In the second aspect of the present invention, a hand-eye calibration optimization system for hard refueling applications is provided, including:

[0026] An estimated pose determination module, which calculates the estimated pose of the calibration board in the global coordinate system according to the pose of the camera rigid body in the global coordinate system, the hand-eye calibration matrix, and the pose of the calibration board in the camera optical center coordinate system;

[0027] A loss function construction module, which constructs a loss function according to the estimated pose of the calibration board in the global coordinate system and the pose of the calibration board in the global coordinate system;

[0028] The deep learning optimization module iteratively optimizes the loss function by adjusting the parameters of the hand-eye calibration matrix, and determines the optimized hand-eye calibration matrix according to the optimized loss function.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] 1. An innovative method of refinement and optimization by combining deep learning on the basis of the traditional Tsai hand-eye calibration method is proposed; on the basis of the rough calibration obtained by the prior art, the hand-eye calibration matrix is further optimized by deep learning, improving the calibration accuracy and overcoming the error problem caused by the deviation of the rotation matrix in the traditional method.

[0031] 2. The error between the attitude of the Aruco calibration board obtained by the motion capture system and the attitude obtained by transforming the camera detection result to the global coordinate system through the hand-eye matrix is used for optimization. This closed-loop optimization mechanism effectively utilizes the data in the actual environment for regression training, making the solution of the hand-eye calibration matrix more accurate, thereby improving the overall positioning and control accuracy of the system.

[0032] The present invention will be further described in detail below with reference to the drawings and embodiments. Description of the Drawings

[0033] Figure 1 is a flowchart of a hand-eye calibration optimization method for hard refueling applications provided by an embodiment of the present invention.

[0034] Figure 2 is a schematic diagram of a hand-eye calibration optimization system for hard refueling applications provided by an embodiment of the present invention; Detailed Embodiments

[0035] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following describes the solution according to the present invention in detail with reference to the drawings and specific embodiments.

[0036] The foregoing and other technical contents, features and effects of the present invention can be clearly presented in the following detailed description in conjunction with the drawings. Through the description of the specific embodiments, a more in-depth and specific understanding of the technical means and effects adopted by the present invention to achieve the predetermined purpose can be obtained. However, the attached drawings are only for reference and illustration, and are not used to limit the technical solution of the present invention.

[0037] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant are intended to cover non-exclusive inclusion, so that an article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed. Without more limitations, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the article or device including the said element.

[0038] As Figure 1 shown, a hand-eye calibration optimization method for hard refueling applications provided by an embodiment of the present invention includes the following steps:

[0039] Step 1, Estimated pose determination: Calculate the estimated pose of the calibration board in the global coordinate system according to the pose of the camera rigid body in the global coordinate system, the hand-eye calibration matrix, and the pose of the calibration board in the camera optical center coordinate system.

[0040] In the present invention, the camera rigid body is a rigid structure composed of multiple marker points (Markers), and the motion capture system is composed of multiple high-precision optical tracking cameras with sub-millimeter measurement accuracy for real-time tracking of the spatial pose of the marker points (Markers); the Markers are recognized by the motion capture system and used to calculate the three-dimensional pose information of the camera rigid body, that is, the pose of the camera rigid body in the global coordinate system; the calibration board is an Aruco calibration board, and its pose in the camera optical center coordinate system is obtained by computer vision algorithms.

[0041] The camera optical center is the imaging center of the camera, and its coordinate system is used to describe the pose of the target obtained by the visual detection algorithm. Since the camera rigid body is composed of multiple Markers and the camera optical center is not at the geometric center of these Markers, the camera rigid body, as an intermediate reference coordinate system, converts the global coordinate system under the motion capture system and the camera optical center coordinate system. Therefore, there is a rigid body transformation between the camera optical center and the camera rigid body, and this transformation is the transformation matrix of the camera optical center relative to the camera rigid body, that is, the hand-eye calibration matrix. In the present invention, the hand-eye calibration matrix is obtained by the existing Tsai method.

[0042] The pose of the camera rigid body in the global coordinate system, the hand-eye calibration matrix, and the pose of the Aruco calibration board in the camera optical center coordinate system are associated and transformed to obtain the estimated pose of the Aruco calibration board in the global coordinate system; the calculation formula for the estimated pose of the Aruco calibration board in the global coordinate system is as follows:

[0043]

[0044] Among them, is the pose of the camera rigid body in the global coordinate system, is the hand-eye calibration matrix, is the pose of the calibration board in the camera optical center coordinate system, is the estimated pose of the calibration board in the global coordinate system.

[0045] The high-precision global pose is provided by the motion capture system, the local pose of the calibration board is obtained by computer vision algorithms, and the existing Tsai method is used to realize the coordinate system transformation. That is, the Tsai method determines the hand-eye calibration matrix by establishing the pose relationship between the robot motion and the images captured by the camera, and realizes the transformation between the camera coordinate system and the robot coordinate system.

[0046] The high-precision global pose of the motion capture system can correct the errors generated by lens distortion and illumination in the computer vision algorithm. The real-time pose of the calibration board obtained by the computer vision algorithm provides more detailed local information for the motion capture system, and the hand-eye calibration matrix of the Tsai method ensures the accurate transformation between different coordinate systems. Through the fusion of the above multi-source data, the estimated pose of the calibration board in the global coordinate system can be calculated more accurately.

[0047] Step 2: Construction of the loss function: Construct a loss function based on the estimated pose of the calibration board in the global coordinate system and the pose of the calibration board in the global coordinate system.

[0048] The calibration board is an Aruco calibration board with Aruco Markers. The Markers are recognized by the motion capture system and used to calculate the three-dimensional pose information of the Aruco calibration board, that is, the pose of the Aruco calibration board in the global coordinate system.

[0049] The calculation formula of the loss function is as follows:

[0050]

[0051] Among them, is the estimated pose of the calibration board in the global coordinate system under the i-th sample, is the pose of the calibration board in the global coordinate system under the i-th sample, Loss is the loss function, N is the number of samples, and ∥∥ 2 represents the square of the two-norm of the vector difference.

[0052] The mean squared error is used to construct the loss function, which comprehensively considers the information of each dimension of the estimated pose and the true pose of the calibration board in the global coordinate system. By accumulating and averaging the squares of the two-norm of the difference between the estimated pose and the true pose vector under multiple samples, the difference between the two can be measured comprehensively and meticulously. Whether it is translational deviation or rotational deviation, it can be reflected in the value of the loss function, avoiding the situation of missing errors due to only focusing on partial pose information.

[0053] Step 3, Deep learning optimization: Iteratively optimize the loss function by adjusting the parameters of the hand-eye calibration matrix, and determine the optimized hand-eye calibration matrix according to the optimized loss function.

[0054] The hand-eye calibration matrix consists of a rotation matrix and a translation vector, and is expressed as:

[0055]

[0056] Among them, is the hand-eye calibration matrix, the parameter R is the rotation matrix, and the parameter t is the translation vector, which is used to describe the translation relationship of the camera coordinate system relative to the robot coordinate system.

[0057] Specifically, R is a 3×3 rotation matrix, R = [R1 R2 R3], R3 = R1 × R2, R1, R2, and R3 are the first column, the second column, and the third column of the rotation matrix respectively. In order to ensure that the rotation matrix meets the orthogonality requirements, the third column is obtained by taking the cross product of R1 and R2, R3 = R1 × R2; t is a 3×1 rotation vector, where, t x represents the translation distance in the x-axis direction, t y represents the translation distance in the y-axis direction, t z represents the translation distance in the z-axis direction.

[0058] Based on deep learning, the loss function Loss is iteratively optimized by continuously adjusting the parameters of the hand-eye calibration matrix, namely the rotation matrix R and the translation vector t, until the iterative optimization termination condition is met. The iterative optimization termination condition is that the loss function reaches the minimum value or reaches the number of iterations. The number of iterations in this application is 200 times. When the iterative optimization terminates, the hand-eye calibration matrix parameters corresponding to the optimized loss function constitute the optimized hand-eye calibration matrix.

[0059] In each iteration process, the rotation matrix R and the translation vector t are slightly adjusted according to the feedback of the loss function. As the number of iterations increases, the value of the loss function gradually decreases, that is, the error between the estimated pose and the true pose is continuously reduced, and the accuracy of the hand-eye calibration matrix is significantly improved, making the docking between the fuel dispenser and the fuel receiver more accurate.

[0060] In the scenario of taking 1000 sample data in this application, it only takes 5 minutes to optimize and iterate 200 times. When the optimized hand-eye calibration matrix is used, when the relative distance between the refueling equipment of the fuel dispenser and the receiver aircraft is about 1.8 meters, the positioning accuracy of the final vision algorithm can reach 1.2 centimeters, which is significantly better than 2.4 centimeters in the prior art.

[0061] As Figure 2 shown, a hand-eye calibration optimization system for rigid refueling applications provided by an embodiment of the present invention includes:

[0062] An estimated pose determination module that calculates the estimated pose of the calibration board in the global coordinate system according to the pose of the camera rigid body in the global coordinate system, the hand-eye calibration matrix, and the pose of the calibration board in the camera optical center coordinate system.

[0063] The estimated pose determination module includes a first pose acquisition module and an estimated pose calculation module. The first pose acquisition module includes a motion capture system for acquiring the pose of the camera rigid body in the global coordinate system, a hand-eye calibration matrix acquisition module, and a computer vision algorithm module.

[0064] In the present invention, the camera rigid body is a rigid structure composed of multiple marker points. The marker points of the camera rigid body are 4-6 passive infrared marker points with a diameter of (15 mm - 20 mm). The marker points are evenly distributed on the surface of the camera rigid body to form a collinear structure, such as a tetrahedron or a rectangle. The distance between adjacent marker points is greater than or equal to 5 cm to ensure measurement accuracy, and the marker points are made of matte material to reduce reflection interference.

[0065] The motion capture system is composed of multiple high-precision optical tracking cameras. The number of high-precision optical tracking cameras is 6-8, and it has a measurement accuracy of sub-millimeter level. Among them, the motion capture system is preferably 6 high-precision optical tracking cameras with an installation height of 1.5 m - 3.5 m, which are distributed in a hemispherical shape around the camera rigid body and the calibration board. The tracking angle of view is 180° in the horizontal plane and 90° in the vertical plane. The angle between adjacent high-precision optical tracking cameras is greater than or equal to 30° to ensure that the marker points are observed by at least two cameras simultaneously to ensure measurement accuracy.

[0066] The motion capture system is used to track the spatial pose of the marker points in real time; the marker points can be recognized by the passive capture system and used to calculate the three-dimensional pose information of the camera rigid body, that is, the pose of the camera rigid body in the global coordinate system; the calibration board is an Aruco calibration board, and its pose in the camera optical center coordinate system is detected by the computer vision algorithm module.

[0067] The camera optical center is the imaging center of the camera. Its coordinate system takes the camera imaging center and the camera optical center as the origin, and the coordinate axis directions follow the camera coordinate system standard, which is used to describe the target pose obtained by the visual detection algorithm. Since the camera rigid body is composed of multiple Markers, the geometric center of the marker points (Markers) on the camera rigid body is the origin, and the coordinate axis directions are determined by the rigid body structure. However, the camera optical center is not at the geometric center of the Marker. The camera rigid body serves as an intermediate reference coordinate system to convert the global coordinate system under the motion capture system and the camera optical center coordinate system. Therefore, there is a rigid body transformation between the camera optical center and the camera rigid body, and this transformation is the transformation matrix of the camera optical center relative to the camera rigid body, that is, the hand-eye calibration matrix. In the present invention, the hand-eye calibration matrix acquisition module is used to obtain the hand-eye calibration matrix through the existing Tsai method.

[0068] The estimated pose calculation module is used to perform an associated transformation on the pose of the camera rigid body in the global coordinate system, the hand-eye calibration matrix, and the pose of the Aruco calibration board in the camera optical center coordinate system to obtain the estimated pose of the Aruco calibration board in the global coordinate system.

[0069] The loss function construction module constructs a loss function based on the estimated pose of the calibration board in the global coordinate system and the pose of the calibration board in the global coordinate system.

[0070] The loss function construction module includes a second pose acquisition module and a loss function calculation module. The second pose acquisition module is used to obtain the pose of the calibration board in the global coordinate system, and the loss function calculation module is used to construct a loss function based on the estimated pose of the calibration board in the global coordinate system and the pose of the calibration board in the global coordinate system.

[0071] Specifically, the calibration board is a high-contrast Aruco calibration board with Aruco Markers. The number of marker points on the Aruco calibration board is 4 - 6 non-collinear marker points arranged in a square or stepped shape. Among them, the distance between adjacent marker points is greater than or equal to 5 cm. The marker points are recognized by the motion capture system and used to calculate the three-dimensional pose information of the Aruco calibration board, that is, the pose of the Aruco calibration board in the global coordinate system.

[0072] The deep learning optimization module iteratively optimizes the loss function by adjusting the hand-eye calibration matrix parameters, and determines the optimized hand-eye calibration matrix according to the optimized loss function.

[0073] Specifically, based on deep learning, the loss function Loss is iteratively optimized by continuously adjusting the hand-eye calibration matrix parameters, that is, the rotation matrix R and the translation vector t, until the iterative optimization termination condition is met. The iterative optimization termination condition is that the loss function reaches the minimum value or reaches the number of iterations. The number of iterations in this application is 200 times. When the iterative optimization terminates, the hand-eye calibration matrix parameters corresponding to the optimized loss function constitute the optimized hand-eye calibration matrix.

[0074] This application proposes a hand-eye calibration optimization method and system for rigid refueling applications. The pose of the camera rigid body in the global coordinate system is obtained by using a motion capture system, the pose of the calibration board in the camera optical center coordinate system is obtained through computer vision algorithms, and the estimated pose of the calibration board in the global coordinate system is calculated by combining the hand-eye calibration matrix obtained by the Tsai method. Then, the mean square error is used to construct a loss function to measure the difference between the estimated pose and the true pose. Finally, the parameters of the hand-eye calibration matrix are adjusted by deep learning to optimize the loss function, so as to obtain a more accurate hand-eye calibration matrix, significantly improving the system positioning and control accuracy and overcoming the error problem caused by the rotation matrix deviation in traditional methods.

[0075] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. It cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should all be regarded as belonging to the protection scope of the present invention.

Claims

1. A hand-eye calibration optimization method for rigid refueling applications, characterized in that Including the following steps: Estimated pose determination step: Calculate the estimated pose of the calibration board in the global coordinate system based on the pose of the camera rigid body in the global coordinate system, the hand-eye calibration matrix, and the pose of the calibration board in the camera optical center coordinate system; Loss function construction step: Construct a loss function based on the estimated pose of the calibration board in the global coordinate system and the pose of the calibration board in the global coordinate system; Deep learning optimization step: Iteratively optimize the loss function by adjusting the hand-eye calibration matrix parameters, and determine the optimized hand-eye calibration matrix according to the optimized loss function.

2. The hand-eye calibration optimization method for hard refueling applications according to claim 1, wherein, The formula for calculating the estimated pose of the calibration board in the global coordinate system in the step of determining the estimated pose is as follows: Wherein, is the pose of the camera rigid body in the global coordinate system, is the hand-eye calibration matrix, is the pose of the calibration board in the camera optical center coordinate system, is the estimated pose of the calibration board in the global coordinate system.

3. The hand-eye calibration optimization method for rigid refueling applications according to claim 1, wherein The step of constructing the loss function further includes constructing the loss function using the mean square error, and the formula for the loss function is as follows: Among them, is the estimated pose of the calibration board under the i-th sample in the global coordinate system, is the pose of the calibration board under the i-th sample in the global coordinate system, Loss is the loss function, N is the number of samples, and ∥∥ 2 represents the square of the two-norm of the vector difference.

4. The hand-eye calibration optimization method for rigid refueling applications according to claim 1, characterized in that, The hand-eye calibration matrix is obtained by the Tsai method.

5. A hand-eye calibration optimization method for rigid refueling applications according to claim 1, 2 or 4, characterized in that The hand-eye calibration matrix is: Among them, is the hand-eye calibration matrix, the parameter R is the rotation matrix, and the parameter t is the translation vector, which is used to describe the translation relationship of the camera coordinate system relative to the robot coordinate system; The rotation matrix R and the translation vector t are used to iteratively optimize the loss function.

6. The hand-eye calibration optimization method for rigid refueling applications according to claim 1 or 2, characterized in that, The pose of the calibration board in the camera optical center coordinate system is detected by a computer vision algorithm.

7. The hand-eye calibration optimization method for rigid refueling applications according to claim 1, wherein The pose of the camera rigid body in the global coordinate system and the pose of the calibration board in the global coordinate system are obtained by tracking with a motion capture system.

8. The hand-eye calibration optimization system for rigid refueling applications according to claim 1, wherein Including: An estimated pose determination module, which calculates the estimated pose of the calibration board in the global coordinate system based on the pose of the camera rigid body in the global coordinate system, the hand-eye calibration matrix, and the pose of the calibration board in the camera optical center coordinate system; A loss function construction module, which constructs a loss function based on the estimated pose of the calibration board in the global coordinate system and the pose of the calibration board in the global coordinate system; A deep learning optimization module, which iteratively optimizes the loss function by adjusting the hand-eye calibration matrix parameters, and determines the optimized hand-eye calibration matrix according to the optimized loss function.

Images