Hand-eye calibration joint optimization method and device, electronic equipment and storage medium

Abstract

The application provides a hand-eye calibration joint optimization method and device, electronic equipment and storage medium, the method comprises the steps that multiple frames of calibration data are acquired; each frame of pose error of an end effector is modeled as an independent rigid body transformation by using a parameterization method, and each frame of pose error obtained by parameterization modeling, a hand-eye transformation matrix, a pose of a calibration object in a world coordinate system and a transformation from the world coordinate system to a base coordinate system are taken as target parameters; a parameterized joint model is established with a first target function of object-base pose consistency constraint, frame-by-frame error constraint of end effector pose error and re-projection error constraint; the target parameters are optimized by minimizing the first target function according to the initialized target parameters, the multiple frames of calibration data and the parameterized joint model, and global error estimation of the end effector pose is performed through global fusion to obtain a hand-eye calibration result. The application improves the hand-eye calibration precision and robustness.

Description

Technical Field

[0001] This invention relates to the field of industrial robot visual positioning and control technology, and in particular to a method, apparatus, electronic device and storage medium for joint optimization of hand-eye calibration. Background Technology

[0002] In modern industrial production, robot vision positioning and manipulation technology has become a crucial support for intelligent manufacturing. Vision systems acquire information about workpieces, the environment, and tools through cameras, enabling robots to perform tasks such as high-precision assembly, flexible grasping, online inspection, and complex path planning. Among these, hand-eye calibration is the core step in achieving precise collaboration between vision and the robot. Its task is to solve the spatial transformation relationship between the robot's end effector coordinate system and the camera coordinate system, thereby accurately mapping visual information onto the robot's control coordinate system.

[0003] The accuracy of hand-eye calibration directly determines the reliability and usability of a robot vision system in industrial settings. Inaccurate calibration will cause deviations between the visual measurement results and the robot's actual operating trajectory, manifesting as grasping and positioning errors, assembly misalignment, trajectory drift, or even task failure. Especially in fields such as high-precision manufacturing, electronic assembly, aerospace, and medical devices, high-precision hand-eye calibration is a prerequisite for achieving automated and intelligent production.

[0004] The accuracy of hand-eye calibration is highly dependent on the accuracy of the end effector's pose relative to the base in each frame. The base pose refers to the position and orientation of the robot's base relative to a fixed reference coordinate system. In practical industrial applications, the end effector pose often exhibits systematic deviations. Ignoring these deviations will lead to a decrease in hand-eye calibration accuracy, especially in single-target or low-redundancy scenarios. Existing hand-eye calibration methods generally assume that the end effector pose is precisely known, lacking explicit modeling and compensation mechanisms for pose deviations, thus limiting accuracy and robustness in practical applications. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, electronic device, and storage medium for joint optimization of hand-eye calibration, so as to improve the accuracy and robustness of hand-eye calibration.

[0006] In a first aspect, the present invention provides a joint optimization method for hand-eye calibration, comprising: Acquire multiple frames of calibration data. Each frame of calibration data includes the pose of the end effector to the base, the pose of the fixed calibration object in the camera coordinate system, and the object point coordinates and image point coordinates of multiple key points on the calibration object. Parametric joint modeling is performed, which involves using a parametric method to model the pose error of the end effector in each frame as an independent rigid body transformation to correct the pose from the end effector to the base. The pose error of each frame, the hand-eye transformation matrix, the pose of the calibration object in the world coordinate system, and the transformation from the world coordinate system to the base coordinate system are used as target parameters. Based on the relative pose invariance between the calibration object and the base, an object-base pose consistency constraint is constructed, and a parametric joint model is established with the object-base pose consistency constraint, the frame-by-frame error constraint of the end effector pose error, and the reprojection error constraint as the first objective function. Initialize the target parameters; Based on the initialized target parameters, multi-frame calibration data, and parameterized joint model, the target parameters are optimized by minimizing the first objective function, and the global error estimation of the end effector pose is performed through global fusion to obtain the hand-eye calibration results.

[0007] In an optional implementation, parametric joint modeling is performed, including: For each frame i The pose error of the end effector is defined as: The pose of the end effector to the base is determined by the following formula. Make corrections: ; Define the hand-eye transformation matrix as X ∈ SE (3), the pose of the calibration object in the world coordinate system is: Y ∈ SE (3) The transformation matrix from the world coordinate system to the base coordinate system is: Z ∈ SE (3); The first objective function is: ; in, This indicates the reprojection error. , Indicates the first i Frame calibration object coordinate system O Next k The coordinates of an object point The result of projecting onto the camera coordinate system and then onto the image plane. Indicates the first i Frame number k Image point coordinates of an object point; This represents the object-base pose consistency constraint residual. , express SE (3) The mapping from Lie groups to Lie algebras This indicates the pose error correction of the corresponding frame. i Frame base coordinate system B Lower end effector E Position of the base Indicates the first i Frame camera coordinate system C Subscript for object O pose in the camera coordinate system, where ZY and All calibration objects are transformed to the base coordinate system through continuous rigid body transformation; Indicates the first i Frame end actuator pose correction error constraint residual, , Indicates the first i Frame pose error; X Represents the hand-eye transformation matrix. Y This indicates the pose of the calibration object in the world coordinate system. Z This represents the transformation matrix from the world coordinate system to the base coordinate system. This represents the pose error for each frame; α , ρ and λ All of these are preset hyperparameters.

[0008] In an optional implementation, the target parameters are initialized, including: The initial hand-eye transformation matrix can be obtained by solving the following equation analytically. : ; in, Indicates the first i Frame base coordinate system B Lower end effector E Position of the base Indicates the first j Frame base coordinate system B The pose of the lower end effector E to the base. Represents the hand-eye transformation matrix. Indicates the first i Frame camera coordinate system C Subscript for object O pose in camera coordinate system Indicates the first j Frame camera coordinate system C Subscript for object O Pose in the camera coordinate system.

[0009] In an optional implementation, the target parameters are initialized, including: Based on the initial hand-eye transformation matrix The initial pose of the calibration object in the world coordinate system is calculated using the following formula. : ; in, N Indicates the number of frames in the calibration data. Indicates the first i Frame base coordinate system B Lower end effector E Position of the base Indicates the first i Frame camera coordinate system C Subscript for object O Pose in the camera coordinate system.

[0010] In an optional implementation, the target parameters are initialized, including: The transformation matrix from the world coordinate system to the base coordinate system and the pose error for each frame are initialized using the identity transformation matrix.

[0011] In an optional implementation, based on the initialized target parameters, multi-frame calibration data, and parameterized joint model, the target parameters are optimized by minimizing the first objective function, and the global error estimation of the end effector pose is performed through global fusion to obtain the hand-eye calibration results, including: Based on the initialized target parameters and multi-frame calibration data, the nonlinear least squares algorithm is used to iteratively solve the target parameters under the first objective function, so as to obtain the target hand-eye transformation matrix and the target pose error for each frame. Based on the target pose error in each frame and the pose from the end effector to the base in the calibration data of each frame, the global error of the end effector pose is estimated to obtain the global target error. The hand-eye calibration result is determined based on the target hand-eye transformation matrix and the target global error.

[0012] In an optional implementation, based on the target pose error of each frame and the pose from the end effector to the base in the calibration data of each frame, a global error estimation of the end effector pose is performed to obtain the target global error, including: Based on the target pose error of each frame and the pose of the end effector to the base in the calibration data of each frame, the global error of the end effector pose is calculated under the second objective function to obtain the target global error; wherein, the second objective function is: ; in, Indicates global error. Indicates the first i Frame base coordinate system B Lower end effectorE Position of the base This indicates the target pose error after correction in the corresponding frame. i Frame base coordinate system B Lower end effector E The position of the base.

[0013] In a second aspect, the present invention provides a hand-eye calibration joint optimization device, comprising: The data acquisition module is used to acquire multiple frames of calibration data. Each frame of calibration data includes the pose of the end effector to the base, the pose of the fixed calibration object in the camera coordinate system, and the object point coordinates and image point coordinates of multiple key points on the calibration object. The model building module is used for parametric joint modeling. Parametric joint modeling includes using a parametric method to model the pose error of the end effector in each frame as an independent rigid body transformation, which is used to correct the pose of the end effector to the base. The pose error of each frame, the hand-eye transformation matrix, the pose of the calibration object in the world coordinate system, and the transformation from the world coordinate system to the base coordinate system obtained by modeling are used as target parameters. Based on the relative pose invariance of the calibration object and the base, the object-base pose consistency constraint is constructed, and a parametric joint model is established with the object-base pose consistency constraint, the frame-by-frame error constraint of the end effector pose error, and the reprojection error constraint as the first objective function. The parameter initialization module is used to initialize the target parameters; The joint optimization module is used to optimize the target parameters by minimizing the first objective function based on the initialized target parameters, multi-frame calibration data and parameterized joint model, and to perform global error estimation of the end effector pose through global fusion to obtain the hand-eye calibration results.

[0014] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the hand-eye calibration joint optimization method of any of the foregoing embodiments.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the hand-eye calibration joint optimization method of any of the foregoing embodiments.

[0016] The present invention provides a method, apparatus, electronic device, and storage medium for joint optimization of hand-eye calibration. The method includes: acquiring multiple frames of calibration data, each frame including the pose of the end effector to the base, the pose of the fixed calibration object in the camera coordinate system, and the object coordinates and image coordinates of multiple key points on the calibration object; performing parametric joint modeling, which includes using a parametric method to model the pose error of the end effector in each frame as an independent rigid body transformation, used to correct the pose of the end effector to the base, and combining the modeled pose error of each frame, the hand-eye transformation matrix, and the pose of the calibration object in the world coordinate system. The pose in the world coordinate system and the transformation from the world coordinate system to the base coordinate system are used as target parameters. Based on the relative pose invariance between the calibration object and the base, an object-base pose consistency constraint is constructed, and a parameterized joint model is established with the object-base pose consistency constraint, the frame-by-frame error constraint of the end effector pose error, and the reprojection error constraint as the first objective function. The target parameters are initialized. Based on the initialized target parameters, multi-frame calibration data, and the parameterized joint model, the target parameters are optimized by minimizing the first objective function, and the global error estimation of the end effector pose is performed through global fusion to obtain the hand-eye calibration result. In this way, the end effector pose deviation of each frame is modeled as an independent rigid body transformation, realizing dynamic correction of the pose deviation. At the same time, by jointly optimizing the reprojection error constraint, the object-base pose consistency constraint, and the frame-by-frame error constraint of the end effector pose error, the impact of single-frame anomalies on calibration accuracy is effectively suppressed on the basis of frame-by-frame estimation and global fusion, improving the overall calibration accuracy and robustness. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Picture 1 This is a schematic diagram illustrating an application scenario of a hand-eye calibration joint optimization method provided in an embodiment of the present invention; Picture 2 This is a flowchart illustrating a hand-eye calibration joint optimization method provided in an embodiment of the present invention; Picture 3 A flowchart illustrating another hand-eye calibration joint optimization method provided in an embodiment of the present invention; Picture 4 This is a schematic diagram of the structure of a hand-eye calibration joint optimization device provided in an embodiment of the present invention; Picture 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] like Picture 1 As shown, in the hand-eye calibration platform, the robot's end effector is connected to the base, and the camera is set on the end effector. Hand-eye calibration is performed using the set calibration objects in conjunction with the world coordinate system.

[0021] Current hand-eye calibration methods typically employ a step-by-step approach: first, the tool coordinate system of the end effector is obtained through tool center point (TCP) calibration; then, analytical methods (such as the Tsai–Lenz algorithm) or least-squares-based optimization methods are used to estimate the hand-eye transformation relationship between the camera and the end effector. Building upon this, the Robot-world-Hand-Eye (RWHE) calibration method has emerged, jointly solving for multiple transformations such as those from the base coordinate system to the world coordinate system, from the camera coordinate system to the end effector coordinate system, and from the target (i.e., the calibration object) to the world coordinate system, to reduce error propagation. However, these methods usually assume that the pose information from the end effector to the base is accurate and reliable. In actual industrial environments, due to human operation and environmental factors, the end effector pose often contains non-negligible errors. 1. Errors caused by operation: Different operators or different times may result in significant differences in calibration results, leading to a shift in the coordinate system of the tool's center point; 2. Errors caused by environmental factors: Illumination, viewing angle or operational deviations may all cause systematic biases in the estimation of the tool's center point.

[0022] These errors result in low stability of the end effector pose and a significant decrease in calibration accuracy. Without modeling and compensation, these deviations will directly lead to a decrease in hand-eye calibration accuracy.

[0023] To address this issue, academia and industry have explored various approaches. For example, some studies have improved the parameters of the Denavit–Hartenberg (DH) model, incorporating end-effector coordinate errors into the geometric error modeling framework, and using least squares or iterative optimization methods to identify and compensate for these errors, significantly improving end-effector positioning accuracy. Other studies have decomposed end-effector errors into base positioning errors, kinematic parameter errors, and tool pose errors, constructing error propagation equations and achieving overall correction through joint angle compensation. These methods rely on the precise acquisition of physical parameters such as robot link length, link torsion angle, and joint angles, which often require expensive external measurement equipment, making the process complex and time-consuming, thus limiting the widespread application of these methods in industrial settings.

[0024] Furthermore, research has revealed that existing methods typically only construct a global error model or rely on geometric parameter correction and offline compensation, lacking a mechanism to directly introduce frame-by-frame dynamic error modeling during hand-eye calibration optimization. This leads to deviations and instability in calibration results when the pose of the end effector in each frame is affected by small perturbations or systematic errors in practical applications.

[0025] Therefore, how to explicitly introduce the modeling of end effector pose error into the hand-eye calibration framework and jointly optimize it with visual constraints in order to achieve dynamic correction of error and suppress the impact of single-frame anomalies has become a key problem that urgently needs to be solved.

[0026] Based on this, embodiments of the present invention provide a hand-eye calibration joint optimization method, device, electronic device and storage medium, which adopts an eye-on-hand hand-eye calibration method based on frame-by-frame end effector pose error modeling and joint optimization, which can solve the problem of insufficient accuracy of existing hand-eye calibration methods when there is a deviation between the end effector and the base pose.

[0027] This invention proposes a hand-eye calibration method based on frame-by-frame end-effector pose error modeling and multi-constraint joint optimization, employing a joint optimization scheme for hand-eye calibration based on frame-by-frame end-effector pose error modeling. In this method, the end-effector pose deviation in each frame is modeled as an independent rigid body transformation, enabling dynamic correction of the pose deviation. Simultaneously, by jointly optimizing and considering reprojection error constraints, object-base pose consistency constraints, and frame-by-frame correction constraints (i.e., frame-by-frame error constraints on the end-effector pose error), the impact of single-frame anomalies on calibration accuracy is effectively suppressed based on frame-by-frame estimation and global fusion, improving overall calibration accuracy and robustness. Compared to existing methods that assume accurate end-effector pose while ignoring systematic error accumulation, this invention, through joint modeling and compensation mechanisms, more accurately compensates for end-effector pose deviations, thereby achieving higher accuracy and stability in industrial robot vision calibration.

[0028] To facilitate understanding of this embodiment, a hand-eye calibration joint optimization method disclosed in this embodiment of the invention will be described in detail below.

[0029] This invention provides a joint optimization method for hand-eye calibration, which can be executed by an electronic device with data processing capabilities, such as a robot controller. See also... Picture 2 The diagram shows a flowchart of a joint optimization method for hand-eye calibration, which mainly includes the following steps S210 to S240: Step S210: Acquire multiple frames of calibration data. Each frame of calibration data includes the pose of the end effector to the base, the pose of the fixed calibration object in the camera coordinate system, and the object point coordinates and image point coordinates of multiple key points on the calibration object.

[0030] The aforementioned end effector can be, but is not limited to, a robotic arm. When acquiring input data, multi-frame calibration data can be obtained by acquiring camera observation data and robot pose data. Acquiring camera observation data can include obtaining data through camera calibration. N The pose of the fixed calibration object in the camera coordinate system object point coordinates Image point coordinates , Indicates the first i Frame camera coordinate system C Subscript for object O pose in camera coordinate system Indicates the first i Frame calibration object coordinate system O Next k The coordinates of an object point Indicates the first i Frame number k Image point coordinates of an object point. Acquiring robot pose data may include: obtaining... N Pose from frame end effector to base , Indicates the first i Frame base coordinate system B Lower end effector E The position of the base. Among them, i Indicates the first i frame, i =1,2,…,N, k Indicates the first k A key point.

[0031] Step S220: Perform parametric joint modeling. Parametric joint modeling includes using a parametric method to model the pose error of the end effector in each frame as an independent rigid body transformation, which is used to correct the pose from the end effector to the base. The pose error of each frame, the hand-eye transformation matrix, the pose of the calibration object in the world coordinate system, and the transformation from the world coordinate system to the base coordinate system obtained by modeling are used as target parameters. Based on the relative pose invariance between the calibration object and the base, object-base pose consistency constraints are constructed, and a parametric joint model is established with object-base pose consistency constraints, frame-by-frame error constraints of the end effector pose error, and reprojection error constraints as the first objective function.

[0032] This embodiment employs parametric joint modeling, including: 1. Hand-eye transformation: Define the hand-eye transformation matrix, which is the transformation from the end effector of the robotic arm to the camera. X ∈ SE (3); 2. Pose of the calibration object in the world coordinate system: Define the pose of the target (i.e., the calibration object) in the world coordinate system as follows: Y ∈ SE (3) This pose also represents the transformation from the calibration object coordinate system to the world coordinate system; 3. Transformation from World Coordinate System to Base Coordinate System: Define the transformation matrix from the world coordinate system to the base coordinate system as follows: Z ∈ SE (3); 4. Frame-by-frame robotic arm pose error modeling: For each frame i Define the end effector pose correction term (i.e., the pose error of the end effector per frame) as: and by the following formula for the original Make corrections: .

[0033] The above SE (3) (Special Euclidean Group in 3 dimensions) is the collection of all rigid body transformations in three-dimensional space, including rotation and translation transformations.

[0034] This invention employs multi-objective joint optimization to ensure pose consistency, including three types of residual terms: 1. Reprojection error: The first i Frame number k The coordinates of an object point (Coordinates in the calibration object coordinate system) are projected onto the camera coordinate system and then onto the image plane via a chain transformation, and then onto the detected image points. Compare: ; ; in, Indicates the camera's image projection. K Let be the camera intrinsic parameter. Therefore, the reprojection error is defined as: .

[0035] 2. Object-base pose consistency constraint residuals: Object-base pose consistency constraints can be constructed based on the relative pose invariance of the calibrator and the base. In this embodiment, the object-base pose consistency constraints are constructed based on the consistency between the calibrator and the base under the Tsai–Lenz model and the robot-world / hand-eye integration model.

[0036] Specifically, based on the ideal hand-eye calibration geometry, the corrected pose of the end effector to the base. Hand-eye transformation matrix X and camera observation The transformation from the base to the world should be considered. Z pose of the target in the world coordinate system Y Maintain consistency. Chain transformation. Starting from the modified end effector, through hand-eye transformation and camera observation, the pose of the target in the base coordinate system can be obtained. ZY This involves transforming the base coordinate system to the world coordinate system. Z pose of the target in the world coordinate system Y The obtained relation is the same. Theoretically, the two should be strictly consistent. Therefore, the object-base pose consistency constraint is defined as: ; in, yes SE (3) Mapping from Lie groups to Lie algebras. Constrained residuals This is used to measure the difference between the two "base-to-target" pose calculation methods. By minimizing this residual, geometric consistency can be forced among different information sources (robot kinematics, camera observations, coordinate system transformations) during optimization. This not only ensures the accuracy requirements of single-frame observations during calibration but also maintains the consistency of multi-frame data globally, thereby improving the robustness and accuracy of hand-eye calibration.

[0037] 3. End effector pose correction error constraint residual (frame-by-frame error constraint of end effector pose error): In order to avoid During the optimization process, unconstrained drift occurs, while ensuring that it is only used to characterize a small range of random deviations. This invention addresses this issue. A zero-mean regularization constraint was introduced: .

[0038] Based on the above three types of residual terms, this embodiment constructs the overall objective function (i.e., the overall optimization objective / first objective function) of the parameterized joint model: ; in, α , ρ and λ All of these are preset hyperparameters.

[0039] Step S230: Initialize the target parameters.

[0040] In some possible embodiments, parameter initialization may include: 1. Hand-eye transformation matrix X The initial hand-eye transformation matrix is ​​obtained by solving the following equation analytically. : .

[0041] 2. Position of the calibration object in the world coordinate system Y Given the initial hand-eye transformation matrix In this case, based on the initial hand-eye transformation matrix The initial pose of the calibration object in the world coordinate system is calculated using the following formula. : .

[0042] 3. Transformation matrix from world coordinate system to base coordinate system Z Using the identity transformation matrix I Perform initialization, that is Z = I , I It is an identity matrix.

[0043] 4. Frame-by-frame correction items (Pose error of the end effector per frame): using the identity transformation matrix I Perform initialization, that is = I .

[0044] Step S240: Based on the initialized target parameters, multi-frame calibration data, and parameterized joint model, optimize the target parameters by minimizing the first objective function, and perform global error estimation of the end effector pose through global fusion to obtain the hand-eye calibration result.

[0045] In some possible embodiments, step S240 may include: iteratively solving the target parameters using a nonlinear least squares algorithm under a first objective function based on the initialized target parameters and multi-frame calibration data to obtain the target hand-eye transformation matrix and the target pose error for each frame; estimating the global error of the end effector pose based on the target pose error for each frame and the pose from the end effector to the base in the calibration data for each frame to obtain the target global error; and determining the hand-eye calibration result based on the target hand-eye transformation matrix and the target global error.

[0046] Furthermore, the aforementioned global target error can be obtained as follows: based on the target pose error of each frame and the pose of the end effector to the base in the calibration data of each frame, the global error of the end effector pose is calculated under the second objective function to obtain the global target error; wherein, the second objective function is: ; in, Indicates global error. Indicates the first i Frame base coordinate system B Lower end effector E Position of the base This indicates the target pose error after correction in the corresponding frame. i Frame base coordinate system B Lower end effector E The position of the base.

[0047] The above hand-eye calibration results can be the product of the target hand-eye transformation matrix and the target global error.

[0048] In practice, the optimization and iteration process may include: 1. Nonlinear optimization: In SE (3) The space is solved iteratively using a nonlinear least squares algorithm until the error (the function value of the first objective function) is less than the first threshold. σ 1. Fixed hand-eye transformation matrix X and frame-by-frame end effector pose error ; 2. Global error estimation of end effector pose: This is achieved through the final values ​​of each frame. The global end effector pose compensation is calculated under the supervision of the second objective function until the error (the function value of the second objective function) is less than the second threshold. σ 2. Fix global error .

[0049] 3. Output results: Output As the final hand-eye calibration result.

[0050] The first and second thresholds mentioned above can be set according to actual needs, and no restrictions are imposed here.

[0051] The hand-eye calibration joint optimization method provided in this invention model the end effector pose deviation of each frame as an independent rigid body transformation, thereby realizing dynamic correction of the pose deviation. At the same time, by jointly optimizing and considering reprojection error constraints, object-base pose consistency constraints, and frame-by-frame error constraints of end effector pose error, the method effectively suppresses the impact of single-frame anomalies on calibration accuracy and improves the overall calibration accuracy and robustness based on frame-by-frame estimation and global fusion.

[0052] The hand-eye calibration joint optimization method provided in this invention includes the following core steps: First, the pose of the end effector to the base and the target pose in the camera coordinate system are obtained for each frame; then, the pose deviation of each frame is modeled as an independent rigid body transformation variable; next, a joint optimization problem is established, simultaneously considering reprojection error constraints, object-base pose consistency constraints, and frame-by-frame correction constraints. Based on frame-by-frame estimation and global fusion, the hand-eye transformation, the target pose in the world coordinate system, and the end effector pose correction are solved; finally, a high-precision and robust hand-eye calibration result is obtained. This method can effectively compensate for the impact of end effector pose deviation on hand-eye calibration accuracy, thereby improving hand-eye calibration accuracy and robustness.

[0053] To better understand the overall execution flow of the method, see [link / reference]. Picture 3 The flowchart shown is another hand-eye calibration joint optimization method, which includes: Step S301, collect N Group input data, including , , , .

[0054] Step S302, parameter initialization.

[0055] Step S303: Calculate the value of the first function according to the first objective function.

[0056] Step S304: Determine whether the first function value is less than or equal to the first threshold. If not, proceed to step S305; if yes, proceed to step S306.

[0057] Step S305: Update the target parameters. Then, re-execute step S303.

[0058] Step S306: Calculate the value of the second function according to the second objective function.

[0059] Step S307: Determine whether the value of the second function is less than or equal to the second threshold. If not, proceed to step S308; if yes, proceed to step S309.

[0060] Step S308: Update the global error parameters. Then, re-execute step S306.

[0061] Step S309: Output the hand-eye calibration result based on the obtained hand-eye transformation matrix and global error.

[0062] It should be noted that, Picture 3 For any parts not described in detail here, please refer to the corresponding content in the foregoing embodiments; they will not be repeated here.

[0063] Compared with existing hand-eye calibration methods, the embodiments of the present invention have the following advantages: Explicit modeling of end effector pose deviation: By performing independent rigid body transformation modeling on the pose deviation from the end effector to the base in each frame, systematic errors can be quantified and compensated, rather than simply assuming that the exact pose of the end effector is known.

[0064] Joint optimization improves calibration accuracy: The reprojection error constraint, object-base pose consistency constraint and frame-by-frame correction constraint are incorporated into the optimization framework to achieve joint optimization of the hand-eye transformation matrix, the pose of the calibration object in the world coordinate system, the transformation matrix from the world coordinate system to the base coordinate system, and the pose error of the end effector in each frame, so as to ensure the overall consistency and accuracy of the calibration results.

[0065] Dynamic correction of frame deviations: Frame-by-frame error parameterization and global fusion mechanisms can effectively suppress the impact of abnormal errors in a single frame on the overall calibration, thereby improving robustness and stability.

[0066] High versatility and adaptability: This method does not depend on specific calibration scenarios or specific sources of deviation, and can be widely applied to hand-eye calibration tasks of industrial robots, thus improving the applicability of the method in practical industrial applications.

[0067] In summary, the embodiments of the present invention effectively solve the problem of decreased accuracy caused by the neglect of end effector pose deviation in existing hand-eye calibration methods by using explicit error modeling and multi-constraint joint optimization, and achieve hand-eye calibration with higher accuracy, stronger robustness and wider applicability.

[0068] Corresponding to the above-described hand-eye calibration joint optimization method, this embodiment of the invention also provides a hand-eye calibration joint optimization device. See also... Picture 4 The diagram shows a structural schematic of a hand-eye calibration joint optimization device, which includes: The data acquisition module 401 is used to acquire multiple frames of calibration data. Each frame of calibration data includes the pose of the end effector to the base, the pose of the fixed calibration object in the camera coordinate system, and the object point coordinates and image point coordinates of multiple key points on the calibration object. The model building module 402 is used for parametric joint modeling. Parametric joint modeling includes using a parametric method to model the pose error of the end effector in each frame as an independent rigid body transformation, which is used to correct the pose of the end effector to the base. The pose error of each frame, the hand-eye transformation matrix, the pose of the calibration object in the world coordinate system and the transformation from the world coordinate system to the base coordinate system obtained by modeling are used as target parameters. Based on the relative pose invariance of the calibration object and the base, the object-base pose consistency constraint is constructed, and a parametric joint model is established with the object-base pose consistency constraint, the frame-by-frame error constraint of the end effector pose error and the reprojection error constraint as the first objective function. The parameter initialization module 403 is used to initialize the target parameters; The joint optimization module 404 is used to optimize the target parameters by minimizing the first objective function based on the initialized target parameters, multi-frame calibration data and parameterized joint model, and to perform global error estimation of the end effector pose through global fusion to obtain the hand-eye calibration result.

[0069] The hand-eye calibration joint optimization device provided in this embodiment of the invention models the end effector pose deviation of each frame as an independent rigid body transformation, realizing dynamic correction of the pose deviation; at the same time, by jointly optimizing and considering reprojection error constraints, object-base pose consistency constraints, and frame-by-frame error constraints of end effector pose error, it effectively suppresses the impact of single-frame anomalies on calibration accuracy and improves the overall calibration accuracy and robustness based on frame-by-frame estimation and global fusion.

[0070] Furthermore, the aforementioned model building module 402 is specifically used for: for each frame i The pose error of the end effector is defined as: The pose of the end effector to the base is determined by the following formula. Make corrections: ; Define the hand-eye transformation matrix as X ∈ SE (3), the pose of the calibration object in the world coordinate system is: Y ∈ SE (3) The transformation matrix from the world coordinate system to the base coordinate system is: Z ∈ SE (3); The first objective function mentioned above is: ; in, This indicates the reprojection error. , Indicates the first i Frame calibration object coordinate system O Nextk The coordinates of an object point The result of projecting onto the camera coordinate system and then onto the image plane. Indicates the first i Frame number k Image point coordinates of an object point; This represents the object-base pose consistency constraint residual. , express SE (3) The mapping from Lie groups to Lie algebras This indicates the pose error correction of the corresponding frame. i Frame base coordinate system B Lower end effector E Position of the base Indicates the first i Frame camera coordinate system C Subscript for object O pose in the camera coordinate system, where ZY and All calibration objects are transformed to the base coordinate system through continuous rigid body transformation; Indicates the first i Frame end actuator pose correction error constraint residual, , Indicates the first i Frame pose error; X Represents the hand-eye transformation matrix. Y This indicates the pose of the calibration object in the world coordinate system. Z This represents the transformation matrix from the world coordinate system to the base coordinate system. This represents the pose error for each frame. α , ρ and λ All of these are preset hyperparameters.

[0071] Furthermore, the parameter initialization module 403 is specifically used to: obtain the initial hand-eye transformation matrix by analytically solving the following formula. : ; in, Indicates the first i Frame base coordinate system B Lower end effector E Position of the base Indicates the first j Frame base coordinate system B The pose of the lower end effector E to the base. Represents the hand-eye transformation matrix. Indicates the firsti Frame camera coordinate system C Subscript for object O pose in camera coordinate system Indicates the first j Frame camera coordinate system C Subscript for object O Pose in the camera coordinate system.

[0072] Furthermore, the parameter initialization module 403 is also used to: initialize the initial hand-eye transformation matrix. The initial pose of the calibration object in the world coordinate system is calculated using the following formula. : ; in, N Indicates the number of frames in the calibration data. Indicates the first i Frame base coordinate system B Lower end effector E Position of the base Indicates the first i Frame camera coordinate system C Subscript for object O Pose in the camera coordinate system.

[0073] Furthermore, the parameter initialization module 403 is also used to: initialize the transformation matrix from the world coordinate system to the base coordinate system and the pose error of each frame using the identity transformation matrix.

[0074] Furthermore, the aforementioned joint optimization module 404 is specifically used for: iteratively solving the target parameters using a nonlinear least squares algorithm under the first objective function based on the initialized target parameters and multi-frame calibration data, to obtain the target hand-eye transformation matrix and the target pose error for each frame; estimating the global error of the end effector pose based on the target pose error for each frame and the pose from the end effector to the base in the calibration data for each frame, to obtain the target global error; and determining the hand-eye calibration result based on the target hand-eye transformation matrix and the target global error.

[0075] Furthermore, the aforementioned joint optimization module 404 is also used to: calculate the global error of the end effector pose under a second objective function based on the target pose error of each frame and the pose from the end effector to the base in the calibration data of each frame, thereby obtaining the target global error; wherein, the second objective function is: ; in, Indicates global error. Indicates the first i Frame base coordinate system B Lower end effector EPosition of the base This indicates the target pose error after correction in the corresponding frame. i Frame base coordinate system B Lower end effector E The position of the base.

[0076] The hand-eye calibration joint optimization device provided in this embodiment has the same implementation principle and technical effect as the aforementioned hand-eye calibration joint optimization method embodiment. For the sake of brevity, any parts not mentioned in the hand-eye calibration joint optimization device embodiment can be referred to the corresponding content in the aforementioned hand-eye calibration joint optimization method embodiment.

[0077] like Picture 5 As shown, an electronic device 500 provided in this embodiment of the invention includes: a processor 501, a memory 502 and a bus. The memory 502 stores a computer program that can run on the processor 501. When the electronic device 500 is running, the processor 501 and the memory 502 communicate through the bus, and the processor 501 executes the computer program to implement the above-mentioned hand-eye calibration joint optimization method.

[0078] Specifically, the memory 502 and processor 501 mentioned above can be general-purpose memory and processor, without any specific limitations here.

[0079] This invention also provides a computer-readable storage medium storing a computer program. When a processor executes the computer program, it performs the hand-eye calibration joint optimization method described in the preceding method embodiments. The computer-readable storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk.

[0080] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0081] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0084] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0085] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A joint optimization method for hand-eye calibration, characterized in that, include: Acquire multiple frames of calibration data. Each frame of calibration data includes the pose of the end effector to the base, the pose of the fixed calibration object in the camera coordinate system, and the object point coordinates and image point coordinates of multiple key points on the calibration object. Parametric joint modeling is performed. The parametric joint modeling includes using a parametric method to model the pose error of the end effector in each frame as an independent rigid body transformation, which is used to correct the pose of the end effector to the base. The pose error of each frame, the hand-eye transformation matrix, the pose of the calibration object in the world coordinate system and the transformation from the world coordinate system to the base coordinate system obtained by modeling are used as target parameters. Based on the relative pose invariance between the calibration object and the base, an object-base pose consistency constraint is constructed, and a parameterized joint model is established with the object-base pose consistency constraint, the frame-by-frame error constraint of the end effector pose error, and the reprojection error constraint as the first objective function. The target parameters are initialized; Based on the initialized target parameters, the multi-frame calibration data, and the parameterized joint model, the target parameters are optimized by minimizing the first objective function, and the global error estimation of the end effector pose is performed through global fusion to obtain the hand-eye calibration result. This includes: iteratively solving the target parameters using a nonlinear least squares algorithm under the first objective function based on the initialized target parameters and the multi-frame calibration data to obtain the target hand-eye transformation matrix and the target pose error for each frame; estimating the global error of the end effector pose based on the target pose error for each frame and the end effector-to-base pose in the calibration data for each frame to obtain the target global error; and determining the hand-eye calibration result based on the target hand-eye transformation matrix and the target global error.

2. The hand-eye calibration joint optimization method according to claim 1, characterized in that, The parametric joint modeling includes: For each frame i The pose error of the end effector is defined as follows: The pose of the end effector to the base is determined by the following formula. Make corrections: ； Define the hand-eye transformation matrix as X ∈ SE (3), the pose of the calibration object in the world coordinate system is: Y ∈ SE (3), and the transformation matrix from the world coordinate system to the base coordinate system is Z ∈ SE (3); The first objective function is: ； in, This indicates the reprojection error. , Indicates the first i Frame calibration object coordinate system O Next k The coordinates of an object point The result of projecting onto the camera coordinate system and then onto the image plane. Indicates the first i Frame number k Image point coordinates of an object point; This represents the object-base pose consistency constraint residual. , express SE (3) The mapping from Lie groups to Lie algebras This indicates the pose error correction of the corresponding frame. i Frame base coordinate system B Lower end effector E Position of the base Indicates the first i Frame camera coordinate system C Subscript for object O pose in the camera coordinate system, where ZY and All calibration objects are transformed to the base coordinate system through continuous rigid body transformation; Indicates the first i Frame-end actuator pose correction error constraint residual. , Indicates the first i Frame pose error; X Represents the hand-eye transformation matrix. Y This indicates the pose of the calibration object in the world coordinate system. Z This represents the transformation matrix from the world coordinate system to the base coordinate system. This represents the pose error for each frame; α , ρ and λ All of these are preset hyperparameters.

3. The hand-eye calibration co-optimization method of claim 1, wherein, The initialization of the target parameters includes: The initial hand-eye transformation matrix can be obtained by solving the following equation analytically. : ； in, Indicates the first i Frame base coordinate system B Lower end effector E Position of the base Indicates the first j Frame base coordinate system B The pose of the lower end effector E to the base. Represents the hand-eye transformation matrix. Indicates the first i Frame camera coordinate system C Subscript for object O pose in camera coordinate system Indicates the first j Frame camera coordinate system C Subscript for object O Pose in the camera coordinate system.

4. The hand-eye calibration co-optimization method of claim 1, wherein, The initialization of the target parameters includes: Based on the initial hand-eye transformation matrix The initial pose of the calibration object in the world coordinate system is calculated using the following formula. : ； in, N This indicates the number of frames in the calibration data. Indicates the first i Frame base coordinate system B Lower end effector E Position of the base Indicates the first i Frame camera coordinate system C Subscript for object O Pose in the camera coordinate system.

5. The hand-eye calibration co-optimization method of claim 1, wherein, The initialization of the target parameters includes: The transformation matrix from the world coordinate system to the base coordinate system and the pose error for each frame are initialized using the identity transformation matrix.

6. The hand-eye calibration co-optimization method of claim 1, wherein, The step of estimating the global error of the end effector pose based on the target pose error in each frame and the pose from the end effector to the base in the calibration data of each frame, to obtain the target global error, includes: Based on the target pose error of each frame and the pose of the end effector to the base in the calibration data of each frame, the global error of the end effector pose is calculated under the second objective function to obtain the target global error; wherein, the second objective function is: ； in, Indicates global error. Indicates the first i Frame base coordinate system B Lower end effector E Position of the base This indicates the target pose error after correction in the corresponding frame. i Frame base coordinate system B Lower end effector E The position of the base.

7. A hand-eye calibration joint optimization apparatus, comprising: include: The data acquisition module is used to acquire multiple frames of calibration data. Each frame of calibration data includes the pose of the end effector to the base, the pose of the fixed calibration object in the camera coordinate system, and the object point coordinates and image point coordinates of multiple key points on the calibration object. The model building module is used for parametric joint modeling. The parametric joint modeling includes using a parametric method to model the pose error of the end effector in each frame as an independent rigid body transformation, which is used to correct the pose of the end effector to the base. The pose error of each frame, the hand-eye transformation matrix, the pose of the calibration object in the world coordinate system and the transformation from the world coordinate system to the base coordinate system obtained by modeling are used as target parameters. Based on the relative pose invariance between the calibration object and the base, an object-base pose consistency constraint is constructed, and a parameterized joint model is established with the object-base pose consistency constraint, the frame-by-frame error constraint of the end effector pose error, and the reprojection error constraint as the first objective function. A parameter initialization module is used to initialize the target parameters; The joint optimization module is used to optimize the target parameters by minimizing the first objective function based on the initialized target parameters, the multi-frame calibration data, and the parameterized joint model, and to perform global error estimation of the end effector pose through global fusion to obtain the hand-eye calibration result. This includes: iteratively solving the target parameters using a nonlinear least squares algorithm under the first objective function based on the initialized target parameters and the multi-frame calibration data to obtain the target hand-eye transformation matrix and the target pose error for each frame; performing global error estimation of the end effector pose based on the target pose error for each frame and the end effector-to-base pose in the calibration data for each frame to obtain the target global error; and determining the hand-eye calibration result based on the target hand-eye transformation matrix and the target global error.

8. An electronic device comprising a memory, a processor, the memory having stored therein a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the hand-eye calibration joint optimization method as described in any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program, when executed by the processor, performs the hand-eye calibration joint optimization method as described in any one of claims 1-6.

Images