High-precision hand-eye calibration algorithm based on nonlinear optimization

Abstract

The invention discloses a high-precision hand-eye calibration algorithm based on nonlinear optimization. The algorithm comprises the following steps: S1, data acquisition; s2, constructing a rotation matrix error model and performing precision analysis; s3, formulating an optimal calibration track rule according to the error model; s4, quaternion parameterization is used, and a global optimization problem is established; s5, solving a global optimization problem through a convex relaxation technology; and S6, outputting and verifying a calibration result. The invention relates to the technical field of robot vision and automatic control, and has the beneficial effects that a rotation matrix error analysis model is constructed (step S2), and the internal mathematical relationship between the hand-eye calibration precision and the motion track parameters (such as the rotation angle and the rotation axis included angle) of the mechanical arm is quantitatively revealed for the first time.

Description

Technical Field

[0001] This invention relates to the field of robot vision and automatic control, and in particular to a high-precision hand-eye calibration algorithm based on nonlinear optimization. Background Technology

[0002] In vision-guided automated systems, such as high-precision assembly, welding, grasping, and medical surgical robots, accurately determining the relative position and orientation between the camera ("eye") and the robotic arm's end effector ("hand") is the cornerstone of seamlessly integrating visual information with robot movements. This process of determining the spatial transformation relationship between the two is called "hand-eye calibration." Its calibration accuracy directly determines the final performance of vision guidance and is a key enabling technology for intelligent robot operations.

[0003] To achieve this calibration, existing technologies typically employ methods based on fixed calibration objects (such as checkerboard patterns, dot arrays, etc.). The core principle can be summarized as follows: A robotic arm is controlled to move a camera to acquire images of the same stationary calibration board from at least two different poses. Let the first... At each pose, the transformation matrix from the checkerboard coordinate system (which can be considered as the world coordinate system) to the camera coordinate system is: This matrix can be obtained through camera calibration and the PnP (Perspective-n-Point) algorithm; simultaneously, the robot controller reads the... Transformation matrix from the end effector flange coordinate system to its base coordinate system at each pose of the robotic arm This matrix is ​​obtained from the forward kinematics solution of the robot model. The transformation matrix from the constant camera coordinate system to the end flange coordinate system is: .

[0004] When the robotic arm moves the camera from its pose Exercise posture At that time, the motion observed in the camera coordinate system is The motion observed in the coordinate system at the end of the robotic arm is According to the chain rule of coordinate system transformation, these two descriptions should be expressed through a hand-eye transformation matrix. By connecting these, we arrive at the classic hand-eye relationship equation: ;in, , , All The rigid body transformation matrix.

[0005] To solve the above equation The most representative existing technology is the Tsai-Lenz algorithm and its many derivative methods. These methods typically employ a two-step solution strategy: first, the matrix equation is decomposed, focusing on the rotation part, based on the equation... Solve the rotation matrix separately Then, the obtained Substitute the known quantities into the translation part of the equation. In the next step, the translation vector is further solved. .Should The algorithm constructs a system of linear equations based on rotation axes and angles, and uses mathematical tools such as Kronecker products or quaternions to find the optimal rotation matrix that minimizes the residuals of the rotation equations. Summary of the Invention

[0006] The purpose of this invention is to solve the above problems by designing a high-precision hand-eye calibration algorithm based on nonlinear optimization.

[0007] The technical solution of the present invention to achieve the above objectives is a high-precision hand-eye calibration algorithm based on nonlinear optimization, comprising the following steps:

[0008] Step S1: Data Acquisition;

[0009] Step S2: Construct the rotation matrix error model and perform accuracy analysis;

[0010] Step S3: Formulate the optimal calibration trajectory rules based on the error model;

[0011] Step S4: Parameterize using quaternions and establish a global optimization problem;

[0012] Step S5: Solve the global optimization problem using convex relaxation techniques;

[0013] Step S6: Output and verify calibration results.

[0014] In step S1: data acquisition includes: coordinate system, data acquisition process and data preprocessing;

[0015] Coordinate system:

[0016] Robot base coordinate system Fixed to the robot's base, it serves as a reference frame describing the robot's end effector pose;

[0017] End flange coordinate system : Fixed to the flange at the end of the robotic arm, and moves with the robotic arm;

[0018] Camera coordinate system With the camera's optical center as the origin, the Z-axis runs along the optical axis.

[0019] checkerboard coordinate system : Taking a fixed corner point on the chessboard as a reference, the Z-axis is perpendicular to the plane of the chessboard;

[0020] The hand-eye transformation matrix to be determined That is, from arrive The transformation matrix;

[0021] Data acquisition process: Synchronously acquire original pose pairs :

[0022] Specifically: Setting and fixing: Placing the chessboard calibration plate securely in a fixed position; robotic arm pose data. Acquisition: By controlling the robotic arm, its end effector carries a camera. ( ,recommend To improve robustness, images of a fixed chessboard are acquired from various poses distributed throughout the workspace; each pose is read and recorded in real time via the robot controller's high-precision communication interface. Transformation matrix from the end flange coordinate system to the base coordinate system of the lower robotic arm ( ); Camera pose data Acquisition: In each of the above robotic arm poses Simultaneously, the camera is triggered to acquire images of a fixed chessboard grid; the acquired images... Using a chessboard-patterned image, and employing the mature Zhang Zhengyou camera calibration method combined with a sub-pixel-level corner extraction algorithm, the camera's intrinsic parameter matrix was calculated with high precision. and lens distortion coefficient Using methods for solving PnP problems, combined with calibrated intrinsic parameters and distortions, the pose of each position can be calculated. Transformation matrix from subscript plate coordinate system to camera coordinate system ( );

[0023] Ultimately, obtain The original transformation matrix pairs that are strictly synchronized in time and pose are... ;

[0024] Data preprocessing: Constructing relative motion pairs :

[0025] Specifically:

[0026] Classic hand-eye equation This describes the relative motion relationship between two coordinate systems. To apply the acquired static pose data to this equation, preprocessing is required to construct relative motion pairs:

[0027] from From each pose, select all that satisfy Posture combination ;

[0028] For each combination, calculate separately:

[0029] Relative motion in the camera coordinate system: ;

[0030] Relative motion in the end-effector coordinate system of the robotic arm: ;

[0031] By iterating through all non-repeating combinations in order, from From the original pose, A group of relative motion pairs; for simplicity, these motion pairs are denoted as .

[0032] In step S2: using the data provided in S1, an error analysis model for the rotation matrix is ​​constructed;

[0033] Assuming the robotic arm and camera are in two relative movements In this context, its actual rotation matrix is: , Due to various noise sources, the actual observations are subject to error. Under the small error assumption, the noisy rotation matrix can be represented as the actual rotation plus a small rotational perturbation. The composite:

[0034] ;

[0035] ;

[0036] Substituting the erroneous observations into the ideal hand-eye rotation equation RARx = RxRB, and through linearization approximation and derivation based on Lie group Lie algebras, the solution is obtained. error and , The relationship; derive the hand-eye transformation matrix. Rotation matrix root mean square error An approximate expression for;

[0037] With three poses, two sets of relative motions can be formed. Taking motion 12 and motion 23 as an example, the rotation matrix... root mean square error Approximately expressed as: (Formula 1);

[0038] Explanation of formula symbols:

[0039] Hand-eye transformation matrix Rotation matrix The root mean square error estimate reflects The solution uncertainty;

[0040] : Represents the unit rotation axis corresponding to two consecutive relative movements of the robotic arm's end effector (e.g., movement 12 from pose 1 to pose 2, and movement 23 from pose 2 to pose 3). and The spatial angle between them;

[0041] , : These represent the two relative movements of the robotic arm's end effector. The magnitude of the rotation angle;

[0042] The camera's first relative motion ( In ), its rotation matrix The error variance; it is generally considered that ,in , These are the camera extrinsic parameter rotation matrices at pose 1 and pose 2, respectively. , The standard deviation of the error;

[0043] The robotic arm in its first relative motion ( In ), the error variance of its rotation matrix R_B12; it is generally considered that ,in , These are the rotation matrices read back by the robotic arm at pose 1 and pose 2, respectively. , The standard deviation of the error.

[0044] In step S3: Based on the quantitative error model established in step S2, insights are transformed into a set of rigorously enforceable engineering principles, actively planning the motion trajectory of the robotic arm, and collecting the dataset. It possesses high information content and strong numerical stability from the very source;

[0045] Specifically:

[0046] Maximizing the rotation angle of a single motion: Under the premise of the robotic arm's workspace and avoiding mutual obstruction between the camera and the calibration plate, the motion of the robotic arm is planned so that each relative motion used for calibration includes a large rotation angle; this directly corresponds to reducing the error in the formula (1). The value of the term; maximizing the angle between rotation axes between motions: when planning multiple sets of calibration poses, ensure that the rotation axes of the different relative motion pairs formed by these poses have the largest possible angle in space, close to 90°; directly corresponding to increasing the value of the term in the error formula (1). The value of the term; achieved by planning the end effector to rotate at large angles around its X, Y, and Z axes in the robot's base coordinate system; minimizing translational motion: while satisfying the above rotational requirements, minimizing the translation vector of the robot arm's end effector in each set of relative motions. The modulus is close to zero; minimize the distance between the camera and the calibration board: while ensuring that the calibration board is fully imaged in the camera's field of view, reduce the distance between the camera's optical center and the plane of the calibration board; e. improve the accuracy of the basic pose; robot: use a robot with high absolute positioning accuracy; camera: use a sub-pixel method to finely extract the corner points of the chessboard grid and perform high-precision camera intrinsic parameter calibration to minimize f. Collect excessive data: Collect far more calibration data than the theoretical minimum requirement (2 sets) (Q>>2, it is generally recommended that Q≥15 to 20).

[0047] In step S4: a singular, compact parameterization method is used to represent rotation; based on all acquired high-quality data. Construct a cost function that minimizes the overall geometric error; transform the constrained optimization problem into a form that can be reliably solved using modern convex optimization techniques, including:

[0048] Quaternion parameterization: The hand-eye transformation matrix X to be determined is parameterized using unit quaternions. Translation vector Parameterization: (Formula 2) where the rotation matrix is... Quaternions The calculation yielded: (Formula 3);

[0049] Global optimization problem establishment: Establish a nonlinear least squares problem with the objective of minimizing the total geometric error of all S sets of calibration data on the hand-eye equation: (Formula 4) Constraints: ;

[0050] Explanation of formula symbols:

[0051] The parameterized hand-eye transformation matrix is ​​a representation of... and 4×4 matrix function;

[0052] : Represents the unit quaternion of rotation, ,in It is the scalar part. It is the vector part;

[0053] : Represents a three-dimensional vector of translation. ;

[0054] : by quaternions The resulting 3×3 rotation matrix;

[0055] , : The transformation matrix of the k-th group of relative motions;

[0056] The Frobenius norm of a matrix; for a matrix M, its Frobenius norm is... It measures the overall level of difference between all corresponding elements of two matrices.

[0057] The objective function is the sum of squares of the total geometric errors of all S sets of calibration data on the hand-eye equation.

[0058] Constraints on unit quaternions ensure The modulus length is 1, thus ensuring This represents a pure rotation without scaling.

[0059] Non-negativity constraint; due to and Representing the same rotation, this constraint is used to select a unique solution from two symmetric global optimal solutions.

[0060] In step S5: a non-convex polynomial optimization problem is relaxed into a convex, efficient, and reliable semidefinite programming problem by introducing auxiliary variables and semidefinite constraints.

[0061] Consider the optimization problem established in step S4: ;

[0062] Constructing Lasserre relaxation:

[0063] Variable boosting: Introduce a higher-dimensional vector y, whose elements are the original variables. The expected value of all monomials of order not exceeding d;

[0064] Constructing a moment matrix: Using vector y, construct a moment matrix. The matrix is ​​positive semi-definite, that is... ;

[0065] Constraint Expression: Expressing the constraints of the original problem This can also be expressed as a linear constraint on the moment variable y;

[0066] Objective function linearization: original objective function It is a polynomial, which, in the space of the moment variable y, is represented as a linear function of y. ;

[0067] Through the above steps, the original non-convex problem P is relaxed into the following convex semidefinite programming problem: (Formula 5);

[0068] Solving and restoring:

[0069] Efficiently solve semidefinite programming problems using the interior-point method and convex optimization algorithm. The optimal moment vector is obtained. ;

[0070] from Extract the first moment, i.e. and Since the moment vector is determined at the global optimum, that is... ;

[0071] Because of numerical calculations, the recovered q_x may not strictly satisfy the unit constraint; it needs to be normalized. ;

[0072] Explanation of formula symbols:

[0073] P: The original non-convex polynomial optimization problem;

[0074] The convex relaxation problem with relaxation order d;

[0075] y: Moment variable vector;

[0076] : A d-order moment matrix, its positive semidefiniteness It is a convex constraint;

[0077] : The linear representation of the original objective function f in the moment space;

[0078] : constraint Linear representation in moment space;

[0079] Localization moment matrix, used to handle non-standard constraints. ;

[0080] : indicates that the matrix is ​​positive semi-definite.

[0081] In step S6: after solving for the optimal quaternion in step S5... Translation vector Next, the results are finally synthesized and output, and the correctness and accuracy of the calibration results are verified to ensure that they can be used for practical robot vision guidance tasks; specifically:

[0082] Result synthesis: and Substituting into formula (2), the final hand-eye transformation matrix can be reconstructed. ; (Formula 6);

[0083] Result verification: To evaluate the accuracy of the calibration results, the following methods were used:

[0084] Reprojection error verification: using Transform the corner points of the chessboard from the chessboard coordinate system to the robot arm's base coordinate system and compare them with their theoretical positions calculated by the robot's kinematics model; calculate the average reprojection error.

[0085] Leave-one-out cross-validation: From the S sets of data, one set is left unused for calibration, and the remaining data are used for calibration. Then use it to predict whether A or B will be left out of that set of data, calculate the prediction error; execute multiple times and take the average.

[0086] Real-world application testing: Its absolute accuracy was tested in real-world visual guidance tasks.

[0087] This invention presents a high-precision hand-eye calibration algorithm based on nonlinear optimization. It innovatively constructs a rotation matrix error analysis model (step S2), revealing for the first time the intrinsic mathematical relationship between hand-eye calibration accuracy and robotic arm motion trajectory parameters (such as rotation angle and rotation axis angle). Based on this model, the derived optimal calibration trajectory rule (step S3) actively guides the data acquisition process, ensuring high information content and strong numerical stability of the acquired data from the source. This transforms a previously blind, experience-dependent data acquisition process into an optimization process with clear theoretical guidance and quantifiable evaluation, fundamentally overcoming the accuracy bottleneck caused by poor data quality in traditional methods. Quaternion parameterization (step S4) effectively avoids the singularity problems that may arise from parameterization methods such as Euler angles.

[0088] At its core, this invention employs a convex relaxation technique (step S5) to transform the original non-convex polynomial optimization problem into a convex positive semi-definite programming (SDP) problem for solution. This transformation mathematically guarantees that the solver can find and converge to the global optimum, completely resolving the inherent flaw of traditional nonlinear optimization methods that are prone to getting trapped in local optima, thus ensuring the theoretical optimality and high reliability of the calibration results. This invention organically combines error modeling, trajectory planning, data acquisition, global optimization, and result verification, forming a logically rigorous and progressively advanced complete technical loop. This scheme not only provides an optimized algorithm but also a set of executable engineering principles, making the high-precision calibration process repeatable and predictable, significantly reducing reliance on manual experience and operational randomness. This invention has no special restrictions on the specific models of robotic arms and cameras; the proposed error model, trajectory planning principles, and optimization solution framework are universal and can be widely applied to various industrial robots and vision-guided scenarios requiring high-precision hand-eye systems, effectively improving the absolute positioning accuracy of the entire robot system. Attached Figure Description

[0089] Figure 1 This is a flowchart of a high-precision hand-eye calibration algorithm based on nonlinear optimization as described in this invention. Detailed Implementation

[0090] The present invention will now be described in detail with reference to the accompanying drawings, such as... Figure 1 As shown, the high-precision hand-eye calibration algorithm based on nonlinear optimization includes step S1: data acquisition.

[0091] This step is fundamental to all subsequent analysis and calculations. Its goal is to acquire a series of high-precision, synchronized robot end-effector pose and camera pose data. This data will be directly used to construct the hand-eye equations and provide a unique input for error analysis, trajectory optimization, and global solution. This invention begins with systematic data acquisition and precise coordinate system definition. This is the foundation of the entire calibration process, and its quality directly affects the upper limit of the final result's accuracy.

[0092] Coordinate system definition: This invention relates to the following four core coordinate systems:

[0093] Robot base coordinate system Fixed to the robot's base, it serves as a reference frame describing the robot's end effector pose.

[0094] End flange coordinate system It is fixed on the flange at the end of the robotic arm and moves with the robotic arm.

[0095] Camera coordinate system With the camera's optical center as the origin, the Z-axis runs along the optical axis.

[0096] checkerboard coordinate system The Z-axis is perpendicular to the plane of the chessboard, with a fixed corner point (usually the origin) as the reference.

[0097] The hand-eye transformation matrix to be determined That is, from arrive The transformation matrix.

[0098] Data acquisition process: Synchronously acquire original pose pairs Setup and Fixation: Place the checkerboard calibration plate securely in a fixed position. Ensure that the calibration plate remains within the camera's field of view and that the image is clear throughout the robotic arm's movement. b. Robotic Arm Pose Data Acquisition: By controlling the robotic arm, its end effector carries a camera. ( ,recommend To improve robustness, images of a fixed chessboard are acquired from various poses, distributed as widely as possible across the workspace and with varying orientations. Simultaneously, each pose is read and recorded in real-time via a high-precision communication interface (such as EtherCAT or TCP / IP) of the robot controller. Transformation matrix from the end flange coordinate system to the base coordinate system of the lower robotic arm ( ).These The matrix constitutes the pose data at the robotic arm end. (Camera pose data) Acquisition: In each of the above robotic arm poses Simultaneously, the camera is triggered to acquire images of a fixed chessboard grid. The acquired images... Using a checkerboard image, mature techniques such as Zhang Zhengyou's camera calibration method, combined with sub-pixel level corner extraction algorithms (such as the cornerSubPix function in OpenCV), were employed to calculate the camera's intrinsic parameter matrix with high precision. and lens distortion coefficient Subsequently, using methods for solving the Perspective-n-Point (PnP) problem, combined with the calibrated intrinsic parameters and distortions, the pose for each position was calculated. Transformation matrix from subscript plate coordinate system to camera coordinate system ( ).These The matrix constitutes the pose data at the camera end.

[0099] Ultimately, we obtained The original transformation matrix pairs that are strictly synchronized in time and pose are... .

[0100] Data preprocessing: Constructing relative motion pairs

[0101] Classic hand-eye equation This describes the relative motion relationship between two coordinate systems. To apply the acquired static pose data to this equation, preprocessing is required to construct relative motion pairs:

[0102] from From each pose, select all that satisfy Posture combination .

[0103] For each combination, calculate separately:

[0104] Relative motion in the camera coordinate system:

[0105] Relative motion in the end-effector coordinate system of the robotic arm:

[0106] By iterating through all non-repeating combinations in sequence, we can... From the original pose, A set of relative motion pairs. For simplicity, we denote these motion pairs as... .

[0107] The core function of the data in this step: the output of this step Group of relative motion pairs It is the sole data foundation of the entire hand-eye calibration process, and its specific functions are as follows:

[0108] Directly input to step S2: Used to construct the rotation matrix error model and quantitatively analyze the data quality and calibration accuracy upper limit.

[0109] Directly input to step S4: This serves as all the observation data for the global optimization problem, used to construct the minimization objective function, which is used to solve the hand-eye transformation matrix. The direct basis.

[0110] The decision driving step S3: The error analysis conclusions drawn from this data in step S2 directly guide step S3 in planning the optimal data acquisition trajectory to obtain higher quality data. .

[0111] Finally, it is used in step S6 to verify the final calibration result. The accuracy and robustness (such as reprojection test, cross-validation).

[0112] Step S2: Construct the rotation matrix error model and perform accuracy analysis.

[0113] Detailed logic:

[0114] This step involves the relative motion pair output from the previous step S1. The analysis focuses on the following: When performing high-precision calibration of hand-eye systems, the existing "two-step method" directly uses data obtained from sensors and models, which inevitably contain errors. , This approach has fundamental limitations. The limitation lies in the fact that the accuracy of the first step, solving for the rotation matrix, directly determines the upper limit of the final calibration result, while observation errors are unavoidable. This sensitivity to input data noise directly leads to a serious technical problem in engineering practice: the accuracy and robustness of the hand-eye calibration results can fluctuate significantly due to minute changes in robot absolute positioning accuracy, camera calibration accuracy, and even ambient lighting and vibration conditions, making it difficult to guarantee the reproducibility and reliability of the calibration process.

[0115] To fundamentally understand and solve this problem, this step utilizes the data provided in S1 to construct an error analysis model for the rotation matrix. The core idea of ​​this model is to transform the transformation matrix... , Calibration error of rotational component By explicitly incorporating these errors into the ideal hand-eye equation, and through rigorous mathematical derivation, we quantitatively reveal how these errors propagate and affect the hand-eye transformation matrix to be determined. Rotation matrix The model improves the accuracy of the solution. It is not only used for post-analysis, but more importantly, it provides a theoretical basis for actively planning the optimal calibration trajectory in step S3.

[0116] Formula logic:

[0117] Assuming the robotic arm and camera are in two relative movements In this context, its actual rotation matrix is: , However, due to various noise sources, what we actually observe is subject to error. Under the assumption of small errors, these noisy rotation matrices can be represented as the actual rotation plus a small rotational perturbation. (Li algebra can be used) Composition of (representation):

[0118] ;

[0119] ;

[0120] Substituting the erroneous observations into the ideal hand-eye rotation equation RARx = RxRB, and through a series of linearization approximations and derivations based on Lie group Lie algebras, the solution can be obtained. error and , The relationship between them. Ultimately, to provide a more intuitive guide for practice, the hand-eye transformation matrix can be derived (or summarized through simulation). Rotation matrix root mean square error An approximate expression for .

[0121] Taking the case of three poses (which can constitute two sets of relative motions, such as motion 12 and motion 23) as an example, the rotation matrix... root mean square error It can be approximated as: (Formula 1);

[0122] Explanation of formula symbols:

[0123] Hand-eye transformation matrix Rotation matrix The root mean square error (RMSE) estimate reflects The solution uncertainty.

[0124] : Represents the unit rotation axis corresponding to two consecutive relative movements of the robotic arm's end effector (e.g., movement 12 from pose 1 to pose 2, and movement 23 from pose 2 to pose 3). and The spatial angle between them. This angle determines the independence of the constraint information provided by the motion combination.

[0125] , : These represent the two relative movements of the robotic arm's end effector. The magnitude of the rotation angle (in radians).

[0126] The camera's first relative motion ( In ), its rotation matrix The error variance. It is generally considered that... ,in , These are the camera extrinsic parameter rotation matrices at pose 1 and pose 2, respectively. , The standard deviation of the error. This reflects the cumulative inaccuracy of camera-end pose estimation.

[0127] The robotic arm in its first relative motion ( In the equation, the error variance of its rotation matrix R_B12 is considered to be... ,in , These are the rotation matrices read back by the robotic arm at pose 1 and pose 2, respectively. , The standard deviation of the error. This reflects the inaccuracy of the robot's kinematic model.

[0128] Analysis of the function of each part of the formula: The core value of the error model constructed in step S2 of this invention lies in decomposing the complex factors affecting calibration accuracy into several clear and quantifiable engineering indicators.

[0129] Rotation axis angle Core function: error and Inversely proportional. This is a crucial discovery. When the two axes of rotation are nearly parallel, the denominator approaches zero, leading to a sharp amplification of the error and causing the equation system to become ill-conditioned. Conversely, when the two axes of rotation are perpendicular (with an angle of 90°), At this point, the amplification effect on the error is minimal. This directly guides rule (b) in step S3: the rotation axis directions between different motions must have a large difference in space.

[0130] Rotation angle Inhibition effect: error With rotation angle It is inversely proportional to the square of the value. A larger rotation angle can effectively "dilute" the effect of observation noise. This directly guides rule (a) in step S3: the rotation angle for each relative motion should be as large as possible.

[0131] Basic data error and Direct impact: error This is directly proportional to the variance of the pose measurement errors of the camera and the robotic arm itself. This emphasizes that high-precision camera intrinsic parameter calibration, high-accuracy PnP calculation, and improved positioning accuracy of the robotic arm are prerequisites for obtaining high-precision hand-eye calibration results.

[0132] In summary, this step, through the analysis of the S1 data, outputs a quantitative error model. The conclusion of this model, namely the quantitative relationship between key geometric factors (rotation axis angle and rotation angle) and calibration accuracy, is the direct basis for driving the next step, S3, to perform active trajectory planning.

[0133] Step S3: Formulate the optimal calibration trajectory rules based on the error model.

[0134] Detailed Logic: Based on the quantitative error model established in step S2, this step transforms theoretical insights into a set of rigorously enforceable engineering principles. Its core objective is to proactively plan the robotic arm's motion trajectory, ensuring that the collected dataset... It possesses high information content and strong numerical stability from the source, thus laying a solid foundation for subsequent global optimization and maximizing the accuracy potential of the final calibration result.

[0135] The specific rules are as follows: Maximize the rotation angle of a single motion: Under the premise of the robotic arm's workspace and avoiding mutual obstruction between the camera and the calibration plate, plan the robotic arm's motion so that each relative motion used for calibration (i.e., from pose) is maximized. Position All of these include the largest possible rotation angle. This directly corresponds to reducing the error in formula (1). The value of the term is used to effectively suppress the influence of observation noise. In practice, it is recommended that the rotation angle of each movement be no less than 30°. Maximize the angle between rotation axes between movements: When planning multiple sets of calibration poses, ensure that the rotation axes of the different relative motion pairs (e.g., (A1,B1) and (A2,B2)) formed by these poses have the largest possible angle in space, ideally close to 90°. This directly corresponds to increasing the value of the term in the error formula (1). The values ​​of the terms are optimized to avoid equation singularity and improve numerical stability. This can be achieved by planning large-angle rotations of the end effector around its X, Y, and Z axes in the robot's base coordinate system. Minimizing translational motion: While satisfying the above rotational requirements, the translation vector of the robot arm's end effector in each set of relative motions should be minimized. The modulus is close to zero. That is, it mainly involves "rotation around a point". This has two advantages: First, it reduces the additional uncertainties introduced by translational motion (robot translation accuracy is usually lower than rotation accuracy); second, it makes the entire calibration process focus more on solving rotational relationships, which simplifies the problem mathematically. d. Minimize the distance from the camera to the calibration board: While ensuring that the calibration board is fully imaged in the camera's field of view, minimize the distance from the camera's optical center to the plane of the calibration board. According to the principle of perspective projection, a closer distance means that the projection error of image pixels has less impact on the 3D pose calculation, thereby improving the PnP solution. The accuracy, that is, reducing Improving basic pose accuracy: This is the cornerstone of high-precision calibration. For robotics: Use robots with the highest possible absolute positioning accuracy. If this is not possible, ensure that the robot's repeatability or relative motion accuracy is sufficiently high. For cameras: Use sub-pixel methods to finely extract checkerboard corner points and perform high-precision camera intrinsic parameter calibration to minimize... Collect excessive data: Collect significantly more calibration data than the theoretical minimum requirement (2 sets) (Q>>2, Q≥15 to 20 is generally recommended). Utilizing a large amount of data for subsequent optimization can effectively suppress random errors and improve the robustness and accuracy of the results through statistical averaging.

[0136] The output of this step is a set of explicit trajectory planning rules. Based on these rules, the (re)acquisition process of S1 data can be guided, thereby obtaining a set of high-quality input data that directly serves the subsequent global optimization solution.

[0137] Step S4: Parameterize using quaternions and establish a global optimization problem

[0138] Detailed Logic: In steps S2 and S3 of this invention, high-quality data has been prepared for high-precision calibration, and the sources of error have been thoroughly understood. However, directly inputting this data into a traditional "two-step" solver still cannot overcome its inherent defects in error accumulation and decoupled optimization. To systematically solve the above fundamental problems, step S4 of this invention abandons the "two-step" method and instead adopts an advanced "one-step" strategy. Its core is to construct a global optimization problem that can simultaneously optimize all six free parameters of rotation and translation. The overall design idea of ​​this new optimization problem is: first, to use a non-singular, compact parameterization method (quaternions) to represent rotation; then, based on all the collected high-quality data... We construct a cost function that aims to minimize the overall geometric error; finally, we transform the constrained optimization problem into a form that can be reliably solved using modern convex optimization techniques.

[0139] Formula Logic: Quaternion Parameterization: The hand-eye transformation matrix X to be determined is parameterized using unit quaternions. Translation vector Parameterization: (Formula 2) where the rotation matrix is... Quaternions The calculation yielded: (Formula 3);

[0140] Global optimization problem establishment: Establish a nonlinear least squares problem with the objective of minimizing the total geometric error of all S sets of calibration data on the hand-eye equation: (Formula 4) Constraints:

[0141] Explanation of formula symbols:

[0142] The parameterized hand-eye transformation matrix is ​​a representation of... and 4×4 matrix function.

[0143] : Represents the unit quaternion of rotation, ,in It is the scalar part. It is the vector part.

[0144] : Represents a three-dimensional vector of translation. .

[0145] : by quaternions The resulting 3×3 rotation matrix.

[0146] , : The transformation matrix of the k-th group of relative motions.

[0147] The Frobenius norm of a matrix. For a matrix M, its Frobenius norm is: It measures the overall level of difference between all corresponding elements of two matrices.

[0148] The objective function is the sum of squares of the total geometric errors of all S sets of calibration data on the hand-eye equation.

[0149] Constraints on unit quaternions ensure The modulus length is 1, thus ensuring This represents a pure rotation without scaling.

[0150] Non-negativity constraint. Because and Representing the same rotation, this constraint is used to select a unique solution from two symmetric global optimal solutions.

[0151] Analysis of the function of each part of the formula:

[0152] The advantage of quaternion parameterization: the use of formulas (2) and (3) is key. It rotates the matrix. The solution from the received and The complex constraints are transformed into those only subject to The simple spherical constraint greatly simplifies the structure of the optimization problem.

[0153] Uniformity of the global objective function: the objective function in formula (4) This is a major innovation of the present invention. It no longer considers rotation and translation separately, but treats X as a whole and directly minimizes its overall inconsistency across all observation data. This means that the optimization algorithm automatically weighs the rotation and translation errors and finds a solution that best fits the overall hand-eye relationship, thus completely avoiding the error propagation path from rotation to translation in the "two-step method".

[0154] Step S5: Solve the global optimization problem using convex relaxation techniques;

[0155] Detailed logic: In step S4, we established a... and This is a constrained polynomial optimization problem. However, the objective function... It is about (4 variables) and This is a 4th-degree 7-variable polynomial function with 3 variables. This is a non-convex optimization problem, potentially with multiple local minima. Conventional local optimization algorithms heavily rely on the initial value selection; poor initial values ​​can easily lead to local optima and calibration failure. To address the challenge of finding a global solution for this non-convex optimization problem, this step employs a convex relaxation technique based on Lasserre's Moment-SOShierarchy. The core idea is to relax a non-convex polynomial optimization problem into a convex, efficiently and reliably solvable semidefinite programming problem by introducing auxiliary variables (moments) and positive semidefinite constraints.

[0156] Formula logic: Consider the optimization problem established in step S4: ;

[0157] Constructing Lasserre relaxation:

[0158] Variable boosting: Introduce a higher-dimensional vector y, whose elements are the original variables. The expected value of all monomials of order no more than d (relaxation order).

[0159] Constructing a moment matrix: Using vector y, construct a moment matrix. The matrix is ​​positive semi-definite, that is... This condition implies all the nonlinear relationships of the original variables.

[0160] Constraint Expression: Expressing the constraints of the original problem It can also be expressed as a linear constraint on the moment variable y.

[0161] Objective function linearization: original objective function It is a polynomial that, in the space of the moment variable y, can be represented as a linear function of y. .

[0162] Through the above steps, we relax the original non-convex problem P into the following convex semidefinite programming problem: (Formula 5);

[0163] Solving and restoring:

[0164] Efficiently solve semidefinite programming problems using interior-point method and convex optimization algorithms. The optimal moment vector is obtained. .

[0165] from Extract the first moment, i.e. and Since the moment vectors are fixed at the global optimum, these expected values ​​are the optimal solution itself, i.e. .

[0166] Because of numerical computation, the recovered q_x may not strictly satisfy the unit constraint. Therefore, it needs to be normalized: .

[0167] Explanation of Formula Symbols

[0168] P: The original non-convex polynomial optimization problem.

[0169] : A convex relaxation (semi-positive definite programming) problem with relaxation order d.

[0170] y: Moment variable vector.

[0171] : A d-order moment matrix, its positive semidefiniteness It is a convex constraint.

[0172] : The linear representation of the original objective function f in the moment space.

[0173] Equality constraints Linear representation in moment space.

[0174] Localization moment matrix, used to handle inequality constraints. .

[0175] : indicates that the matrix is ​​positive semi-definite (all eigenvalues ​​are non-negative).

[0176] Analysis of the function of each part of the formula: The convex relaxation technique used in step S5 of this invention is the key to ensuring the robustness and optimality of the algorithm.

[0177] Global optimality guarantee: Unlike local optimization methods, convex relaxation techniques can find the global optimal solution to the original non-convex problem, or provide an extremely tight lower bound. This completely eliminates the dependence on initial guesses, making the calibration process fully automated, and the results predictable and reproducible.

[0178] Numerical stability and reliability: Semidefinite programming is a well-established problem in convex optimization, with various efficient and robust interior-point solvers available. These solvers can reliably handle large-scale problems and provide high-precision numerical solutions.

[0179] Step S6: Calibration result output and verification:

[0180] Detailed logic: The optimal quaternion is found through step S5. Translation vector Next, this step involves the final synthesis and output of the results, and verifies the correctness and accuracy of the calibration results through various methods to ensure that they can be used for actual robot vision guidance tasks.

[0181] Formula logic: Result synthesis: and Substituting into formula (2), the final hand-eye transformation matrix can be reconstructed. . (Formula 6);

[0182] Result verification: To evaluate the accuracy of the calibration results, the following methods can be used:

[0183] Reprojection error verification: using Transform the corner points of the chessboard from the chessboard coordinate system to the robot arm's base coordinate system and compare them with their theoretical positions calculated using the robot's kinematics model. Calculate the average reprojection error.

[0184] Leave-one-out cross-validation: From the S sets of data, one set is left unused for calibration, and the remaining data are used for calibration. Then use it to predict A or B in the data set that is left out, and calculate the prediction error. Perform this process multiple times and take the average.

[0185] Real-world application testing: Test its absolute accuracy in real-world visual guidance tasks (such as grasping).

[0186] The above technical solutions only embody the preferred technical solutions of the present invention. Any modifications that may be made by those skilled in the art to certain parts thereof embody the principles of the present invention and fall within the protection scope of the present invention.

Claims

1. A high-precision hand-eye calibration algorithm based on nonlinear optimization, characterized in that, Includes the following steps: Step S1: Data Acquisition; Step S2: Construct the rotation matrix error model and perform accuracy analysis; Step S3: Formulate the optimal calibration trajectory rules based on the error model; Step S4: Parameterize using quaternions and establish a global optimization problem; Step S5: Solve the global optimization problem using convex relaxation techniques; Step S6: Output and verify calibration results.

2. The high-precision hand-eye calibration algorithm based on nonlinear optimization according to claim 1, characterized in that, In step S1: data acquisition includes: coordinate system, data acquisition process and data preprocessing; Coordinate system: Robot base coordinate system Fixed to the robot's base, it serves as a reference frame describing the robot's end effector pose; End flange coordinate system : Fixed to the flange at the end of the robotic arm, and moves with the robotic arm; Camera coordinate system With the camera's optical center as the origin, the Z-axis runs along the optical axis. checkerboard coordinate system : Taking a fixed corner point on the chessboard as a reference, the Z-axis is perpendicular to the plane of the chessboard; The hand-eye transformation matrix to be determined That is, from arrive The transformation matrix; Data acquisition process: Synchronously acquire original pose pairs : Specifically: Setting and fixing: Placing the chessboard calibration plate securely in a fixed position; robotic arm pose data. Acquisition: By controlling the robotic arm, its end effector carries a camera. ( ,recommend To improve robustness, images of a fixed chessboard are acquired from various poses distributed throughout the workspace; each pose is read and recorded in real time via the robot controller's high-precision communication interface. Transformation matrix from the end flange coordinate system to the base coordinate system of the lower robotic arm ( ); Camera pose data Acquisition: In each of the above robotic arm poses Simultaneously, the camera is triggered to acquire images of a fixed chessboard grid; the acquired images... Using a chessboard-patterned image, and employing the mature Zhang Zhengyou camera calibration method combined with a sub-pixel-level corner extraction algorithm, the camera's intrinsic parameter matrix was calculated with high precision. and lens distortion coefficient Using methods for solving PnP problems, combined with calibrated intrinsic parameters and distortions, the pose of each position can be calculated. Transformation matrix from subscript plate coordinate system to camera coordinate system ( ); Ultimately, obtain The original transformation matrix pairs that are strictly synchronized in time and pose are... ; Data preprocessing: Constructing relative motion pairs : Specifically: Classic hand-eye equation This describes the relative motion relationship between two coordinate systems. To apply the acquired static pose data to this equation, preprocessing is required to construct relative motion pairs: from From each pose, select all that satisfy pose combination ; For each combination, calculate separately: Relative motion in the camera coordinate system: ; Relative motion in the end-effector coordinate system of the robotic arm: ; By iterating through all non-repeating combinations in order, from From the original pose, A group of relative motion pairs; for simplicity, these motion pairs are denoted as .

3. The high-precision hand-eye calibration algorithm based on nonlinear optimization according to claim 1, characterized in that, In step S2: using the data provided in S1, an error analysis model for the rotation matrix is ​​constructed; Assuming the robotic arm and camera are in two relative movements In this context, its actual rotation matrix is: , Due to various noise sources, the actual observations are subject to error. Under the small error assumption, the noisy rotation matrix can be represented as the actual rotation plus a small rotational perturbation. The composite: ； ； Substituting the erroneous observations into the ideal hand-eye rotation equation RARx = RxRB, and through linearization approximation and derivation based on Lie group Lie algebras, the solution is obtained. error and , The relationship; derive the hand-eye transformation matrix. Rotation matrix root mean square error An approximate expression for; With three poses, two sets of relative motions can be formed. Taking motion 12 and motion 23 as an example, the rotation matrix... root mean square error Approximately expressed as: (Formula 1); Explanation of formula symbols: Hand-eye transformation matrix Rotation matrix The root mean square error estimate reflects The solution uncertainty; : Represents the unit rotation axis corresponding to two consecutive relative movements of the robotic arm's end effector (e.g., movement 12 from pose 1 to pose 2, and movement 23 from pose 2 to pose 3). and The spatial angle between them; , : These represent the two relative movements of the robotic arm's end effector. The magnitude of the rotation angle; The camera's first relative motion ( In ), its rotation matrix The error variance; it is generally considered that ,in , These are the camera extrinsic parameter rotation matrices at pose 1 and pose 2, respectively. , The standard deviation of the error; The robotic arm in its first relative motion ( In ), the error variance of its rotation matrix R_B12; it is generally considered that ,in , These are the rotation matrices read back by the robotic arm at pose 1 and pose 2, respectively. , The standard deviation of the error.

4. The high-precision hand-eye calibration algorithm based on nonlinear optimization according to claim 1, characterized in that, In step S3: Based on the quantitative error model established in step S2, insights are transformed into a set of rigorously enforceable engineering principles, actively planning the motion trajectory of the robotic arm, and collecting the dataset. It possesses high information content and strong numerical stability from the very source; Specifically: Maximizing the rotation angle of a single motion: Under the premise of the robotic arm's workspace and avoiding mutual obstruction between the camera and the calibration plate, the motion of the robotic arm is planned so that each relative motion used for calibration includes a large rotation angle; this directly corresponds to reducing the error in formula (1). The value of the term; maximizing the angle between rotation axes between motions: when planning multiple sets of calibration poses, ensure that the rotation axes of the different relative motion pairs formed by these poses have the largest possible angle in space, close to 90°; directly corresponding to increasing the value of the term in the error formula (1). The value of the term; achieved by planning the end effector to rotate at large angles around its X, Y, and Z axes in the robot's base coordinate system; minimizing translational motion: while satisfying the above rotational requirements, minimizing the translation vector of the robot arm's end effector in each set of relative motions. The modulus is close to zero; minimize the distance between the camera and the calibration board: while ensuring that the calibration board is fully imaged in the camera's field of view, reduce the distance between the camera's optical center and the plane of the calibration board; improve the accuracy of the basic pose. Robot: Utilizes a robot with high absolute positioning accuracy; Camera: Employs sub-pixel methods to precisely extract checkerboard corner points and performs high-precision camera intrinsic parameter calibration to minimize... Collect excessive data: Collect far more calibration data than the theoretical minimum requirement (2 sets) (Q>>2, it is generally recommended that Q≥15 to 20).

5. The high-precision hand-eye calibration algorithm based on nonlinear optimization according to claim 1, characterized in that, In step S4: a singular, compact parameterization method is used to represent rotation; based on all acquired high-quality data. Construct a cost function that aims to minimize the overall geometric error; The constrained optimization problem is transformed into a form that can be reliably solved using modern convex optimization techniques, including: Quaternion parameterization: Parameterizing the hand-eye transformation matrix X to be determined using unit quaternions. Translation vector Parameterization: (Formula 2) where the rotation matrix is... Quaternions The calculation yielded: (Formula 3); Global optimization problem establishment: Establish a nonlinear least squares problem with the objective of minimizing the total geometric error of all S sets of calibration data on the hand-eye equation: (Formula 4) Constraints: ; Explanation of formula symbols: The parameterized hand-eye transformation matrix is ​​a representation of... and 4×4 matrix function; : Represents the unit quaternion of rotation, ,in It is the scalar part. It is the vector part; : Represents a three-dimensional vector of translation. ; : by quaternions The resulting 3×3 rotation matrix; , : The transformation matrix of the k-th group of relative motions; The Frobenius norm of a matrix; for a matrix M, its Frobenius norm is... It measures the overall level of difference between all corresponding elements of two matrices. The objective function is the sum of squares of the total geometric errors of all S sets of calibration data on the hand-eye equation. Constraints on unit quaternions ensure The modulus length is 1, thus ensuring This represents a pure rotation without scaling. Non-negativity constraint; due to and Representing the same rotation, this constraint is used to select a unique solution from two symmetric global optimal solutions.

6. The high-precision hand-eye calibration algorithm based on nonlinear optimization according to claim 1, characterized in that, In step S5: a non-convex polynomial optimization problem is relaxed into a convex, efficient, and reliable semidefinite programming problem by introducing auxiliary variables and semidefinite constraints. Consider the optimization problem established in step S4: ; Constructing Lasserre relaxation: Variable boosting: Introduce a higher-dimensional vector y, whose elements are the original variables. The expected value of all monomials of order not exceeding d; Constructing a moment matrix: Using vector y, construct a moment matrix. The matrix is ​​positive semi-definite, that is... ; Constraint Expression: Expressing the constraints of the original problem This can also be expressed as a linear constraint on the moment variable y; Objective function linearization: original objective function It is a polynomial, which, in the space of the moment variable y, is represented as a linear function of y. ; Through the above steps, the original non-convex problem P is relaxed into the following convex semidefinite programming problem: (Formula 5); Solving and restoring: Efficiently solve semidefinite programming problems using the interior-point method and convex optimization algorithm. The optimal moment vector is obtained. ; from Extract the first moment, i.e. and Since the moment vector is determined at the global optimum, that is... ; Because of numerical calculations, the recovered q_x may not strictly satisfy the unit constraint; it needs to be normalized. ; Explanation of formula symbols: P: The original non-convex polynomial optimization problem; The convex relaxation problem with relaxation order d; y: Moment variable vector; : A d-order moment matrix, its positive semidefiniteness It is a convex constraint; : The linear representation of the original objective function f in the moment space; : constraint Linear representation in moment space; Localization moment matrix, used to handle non-standard constraints. ; : indicates that the matrix is ​​positive semi-definite.

7. The high-precision hand-eye calibration algorithm based on nonlinear optimization according to claim 1, characterized in that, In step S6: after solving for the optimal quaternion in step S5... Translation vector Next, the results are finally synthesized and output, and the correctness and accuracy of the calibration results are verified to ensure that they can be used for actual robot vision guidance tasks. Specifically: Result synthesis: and Substituting into formula (2), the final hand-eye transformation matrix can be reconstructed. ; (Formula 6); Result verification: To evaluate the accuracy of the calibration results, the following methods were used: Reprojection error verification: using Transform the corner points of the chessboard from the chessboard coordinate system to the robot arm's base coordinate system and compare them with their theoretical positions calculated by the robot's kinematics model; calculate the average reprojection error. Leave-one-out cross-validation: From the S sets of data, one set is left unused for calibration, and the remaining data are used for calibration. Then use it to predict whether A or B will be left out of that set of data, calculate the prediction error; execute multiple times and take the average. Real-world application testing: Its absolute accuracy was tested in real-world visual guidance tasks.

Images