Hand-eye calibration of camera-guided devices

By statistically modeling robot inaccuracies and combining methods of minimizing back projection error and error modeling, this approach solves the problems of low absolute posture accuracy and reliance on high-precision measuring instruments in existing hand-eye calibration. It achieves high-accuracy and inexpensive robot calibration, supporting flexible calibration objects and self-calibration.

CN115401685BActive Publication Date: 2026-11-17MVTEC SOFTWARE
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210551230.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-05-26
Filing Date
2022-05-18
Publication Date
2026-11-17
Estimated Expiration
2042-05-18

AI Technical Summary

Technical Problem

Existing hand-eye calibration methods assume that the robot's posture is error-free, resulting in low absolute posture accuracy and requiring high-precision measuring instruments, as well as insufficient flexibility and user-friendliness.

Method used

By statistically modeling robot inaccuracies, and combining minimizing back projection error with error modeling, a calibration-object-based or self-calibration-based approach is adopted, allowing simultaneous calibration of hand-eye pose and internal camera parameters.

Benefits of technology

It improves the accuracy of hand-eye calibration, provides reliable information on robot posture, enables simple and inexpensive calibration, and supports the flexibility and user-friendliness of two variants.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115401685B_ABST
    Figure CN115401685B_ABST
Patent Text Reader

Abstract

This invention describes a general framework for hand-eye calibration of a camera-guided device, wherein a rigid 3D transformation between the device and the camera must be determined. An example of such a device is a camera-guided robot.
Need to check novelty before this filing date? Find Prior Art

Description

Background Technology

[0001] Human eye calibration is essential for camera-guided applications. For robots, it determines a rigid 3D transformation (pose) between the robot and the camera (hand-eye pose). This allows measurements taken in the camera coordinate system to be transformed into the robot coordinate system. For example, in warehouse picking applications with industrial robots, 3D object recognition is used to determine the object's pose in the camera coordinate system (Hofhauser et al., 2009; Ulrich et al., 2012). In order for the robot to grasp an object, the object's pose must be transformed into the robot's coordinate system.

[0002] For camera-guided medical robots, service robots, and humanoid robots, it is also necessary to determine the pose of the robot or robotic hand relative to the camera. For camera-guided autonomously navigating drones, hand-eye calibration determines the pose between the camera and the drone's reference coordinate system. For example, Global Navigation Satellite Systems (GNSS) and Inertial Navigation Systems (INS) can be used to determine the position of the drone's reference coordinate system relative to a higher-level coordinate system. Similar considerations apply to camera-guided ground robots (e.g., reconnaissance or rescue robots). In operating rooms, devices are increasingly used to assist surgeons. In endoscopic examinations, hand-eye calibration must be used to determine the camera's position in the coordinate system of the endoscope or operating room. This also applies to augmented reality applications, where hand-eye calibration must be used to determine the pose between the camera and the position and orientation sensors of the headset. Camera-guided cranes can be used to automate processes in the construction industry. Automated loading and unloading of container ships is also possible. Here, again, the camera's position in the crane's coordinate system must be determined via hand-eye calibration. Although the focus of this invention is on industrial robots, and therefore the following description is based on this case as an example, it is clear that other applications using camera-guided devices can also benefit from this invention. In principle, this applies to all applications where hand-eye calibration is useful. In the following text, the term "robot" therefore refers to all devices that utilize hand-eye calibration.

[0003] Typically, there are two scenarios for camera-guided robots (Steger et al. [Chapter 3.12.2], 2018): In the first scenario, the camera is mounted on the robot's end effector and moves to different positions as the robot is moved. In the second scenario, the camera is fixedly mounted externally to the robot and therefore does not move relative to the robot base. The pose determined by hand-eye calibration is the relative pose of the camera to the robot tool in the first scenario, or the relative pose of the camera to the robot base in the second scenario. In the following description of the invention, reference is made to the first scenario with a moving camera. However, it can also be applied in a similar manner to the second scenario with a stationary camera.

[0004] Conventional methods for hand-eye alignment assume that the pose of the robotic tool relative to the robot base is precisely known. For robot accuracy, it is important to distinguish between repeatability and absolute accuracy (ISO 9283:1998): repeatability describes the robot's ability to repeatedly move the tool to the same pose. Absolute accuracy is the robot's ability to move the tool to a specific pose in 3D space. Modern industrial robots typically offer very high repeatability, ranging from 0.02–0.15 mm (Shiakolas et al., 2002; Abderrahim et al., 2006; Placzek et al., Piszszek, 2018). High repeatability is sufficient for applications where the robotic tool always assumes the exact same pose that has been pre-trained. However, for robots that are programmed offline, and especially for camera-guided robots, high absolute pose accuracy is also important. Unfortunately, the absolute pose accuracy of a robot is often much lower than its repeatability. Absolute attitude accuracy is typically in the range of 0.1 to 10.0 mm, while orientation accuracy is typically in the range of about 0.2 degrees to a few degrees (Abderrahim et al., 2006; Placzek and Piszczek, 2018).

[0005] Robotic calibration can improve accuracy by up to an order of magnitude and, in rare cases, even repeatability (Shiakolas et al., 2002). Unfortunately, robotic calibration typically requires high-precision measuring instruments, such as laser trackers (Abderrahim et al., 2006), and is therefore expensive and time-consuming.

[0006] Almost all previous methods for hand-eye calibration assumed error-free robot pose. The method according to the invention explicitly models robot inaccuracies in a statistically reasonable manner, resulting in higher accuracy of the calculated hand-eye pose. Furthermore, the method according to the invention provides reliable information about robot inaccuracies that would otherwise require high-precision measuring instruments. Since the described method also provides error-corrected (calibrated) robot pose, it also enables simple and inexpensive robot calibration.

[0007] Most existing hand-eye calibration methods require capturing multiple images of the calibration object. Some more flexible and user-friendly solutions avoid using the calibration object. The method according to the invention supports two variants: calibration based on the calibration object and self-calibration, without requiring known 3D points. Furthermore, the method according to the invention optionally allows simultaneous calibration of the internal camera parameters of both variants, thus providing a high degree of user-friendliness. Summary of the Invention

[0008] This invention describes a general framework for hand-eye calibration of camera-guided devices, where a rigid 3D transformation between the device and the camera must be determined. An example of such a device is a camera-guided robot. The robot's inaccuracies are explicitly modeled in a statistically reasonable manner compared to conventional methods. This also applies, for example, to industrial robots. While modern industrial robots are highly precise, their absolute accuracy is often much lower. If these inaccuracies are not explicitly accounted for, they degrade the results of hand-eye calibration. The method proposed in this invention not only results in high accuracy of the calculated hand-eye pose but also provides reliable information about the robot's accuracy. Furthermore, it provides corrected robot poses that can be used for simple and inexpensive robot calibration. The described framework is general in several respects: it supports the use of calibration bodies and self-calibration without requiring known 3D points. It optionally allows for simultaneous calibration of internal camera parameters. The framework is also general with respect to robot type, for example, supporting articulated arms and SCARA robots. Other applications using camera-guided robots or other camera-guided devices, besides those involving industrial robots, can also benefit from this invention. Attached Figure Description

[0009] Figure 1 The coordinate system associated with hand-eye calibration of a robot (device) with a moving camera is shown; and

[0010] Figure 2 The coordinate system associated with the hand-eye calibration of the robot (device) in the case of a stationary camera is shown. Detailed Implementation

[0011] Figure 1The coordinate system associated with hand-eye calibration of a robot (device) with a moving camera is shown. For example, the case of a stationary camera is described in Steger et al. ([Chapter 3.13], 2018) and... Figure 2 As shown in the diagram. Let... s2 H s1 It is a 4x4 homogeneous transformation matrix representing a rigid 3D transformation from a point in coordinate system s1 to s2. One of the most common formulas for hand-eye calibration is based on the closure of the following transformation chain (Strobl and Hirzinger, 2006):

[0012] (1) b H w = b H t t H c c H w

[0013] It has a coordinate system world (w, WCS), a camera (c, CCS), a robot tool (t, TCS), and a robot base (b, BCS), as well as an unknown pose. t H c (Hand-eye posture) and b H w Unknown poses are typically determined by moving the robot to different poses and capturing images of a calibrated object that defines the WCS in each pose. In each robot pose (device pose), b H t The robot controller queries, and c H w Determined by the PnP algorithm or camera calibration (e.g., as described in Zhang, 2000).

[0014] Formula (2) is usually written as follows:

[0015] (2) Y = A i XB i

[0016] For each i (i = 1, ..., n) of n robot poses, there are unknown poses X and Y and an observed pose A. i and B i Since the fundamental unknown is X, Y can be eliminated by considering a pair of different robot poses i and j. By doing so, we can obtain... use and get

[0017] (3) AX = XB,

[0018] Wherein, when the robot moves from pose i to j, the motion of the tool is represented by A, and the motion of the camera is represented by B.

[0019] Several linear methods exist that essentially solve equations (2) or (3), for example, Tsai and Lenz, 1989; Chen, 1991; Horaud and Dornaika, 1995; Dornaika and Horaud, 1998; Danilidis, 1999; Andreff et al., 2001; Schmidt et al., 2003; Ulrich and Steger, 2016. They determine the rotational and translational portions of the hand-eye pose sequentially or simultaneously. The latter has the advantage that rotational errors do not propagate, thus amplifying translational errors. Methods for solving equation (3) require the selection of appropriate robot pose pairs to compute A and B. Criteria for selecting appropriate pose pairs are proposed in Tsai and Lenz, 1989; Schmidt et al., 2003; and Schmidt and Niemann, 2008. However, it remains difficult to ensure that the observed information (A) i and B i The robot's pose is optimally utilized. Furthermore, these methods assume error-free robot pose (Tsai and Lenz, 1989).

[0020] Because linear methods typically minimize algebraic errors, their accuracy is limited. Therefore, they are often used to initialize subsequent nonlinear optimizations to achieve higher accuracy. Most nonlinear methods that minimize algebraic or geometric errors (Horaud and Dornaika 1995; Dornaika and Horaud, 1998; Danilidis 1999; Schmidt et al., 2005; Ulrich and Steger, 2016; Steger [Chapter 3.13.5], 2018) also assume error-free robot pose and often face the additional problem of how to weight the error components of rotation and translation relative to each other. Strobl and Hirazinger (2006) minimize the weighted sum of the rotational and translational error shares, where the weights of the error components are statistically derived. While this allows for consideration of robot pose… b H t Errors in the middle, but no errors in the camera pose. c H w It is used as input. Nguyen and Pham (2018) solved the rotation and translation parts of equation (3) sequentially and propagated the errors in A and B to X.

[0021] Another class of methods minimizes the backprojection error of 3D world points on the calibration object (e.g., Tabb and Yousef, 2017), similar to camera calibration methods such as Zhang's (2000). A key advantage of this class is that it eliminates the need for explicit estimation of the camera position in each image during preprocessing, eliminates the need for pre-selection of pose pairs, and minimizes meaningful geometric errors in the space of the observed error measurements (i.e., image points). Another advantage is that minimizing the backprojection error also allows for simultaneous estimation of intrinsic camera parameters. However, these methods also assume error-free robot pose. In Koide and Menegati (2019), a pose graph optimization framework for hand-eye calibration is presented, where errors in robot pose are minimized in addition to the backprojection error of the calibration object's points. The advantages of considering the uncertainty of robot pose have been demonstrated experimentally. Unfortunately, details regarding the stochastic models, optimizations, and weightings of the different error components relative to each other are not given in this publication.

[0022] Self-calibration methods are used to perform hand-eye calibration without a calibration object. In this method, unknown 3D points are tracked in a sequence of images obtained from a predetermined robot motion. Internal camera parameters and 3D points are estimated simultaneously with the hand-eye pose. Andreff et al. (2001) used a structure-from-motion (SfM) approach, where the unknown scaling factor of the SfM results is integrated into the equations. This idea was derived in Schmidt et al. (2005), where a post-processing step forcing the orthogonality of the rotation matrix is ​​avoided by incorporating the unknown scaling factor into the equations of Horaud and Dornaika (1995) and Danilidis (1999). Since the described self-calibration techniques minimize algebraic errors, the achievable accuracy using these methods is limited.

[0023] The method according to the invention combines the aforementioned advantages of minimizing back projection error, the advantage of randomly modeling the inaccuracies of all measured observations (image points and robot poses), the flexibility of performing calibration or self-calibration based on the calibration object, and the possibility of using known internal camera parameters or estimating them simultaneously.

[0024] According to a first aspect, the present invention provides a hand-eye calibration method for determining parameters of the hand-eye posture of a camera-guided device. This method includes the following steps:

[0025] (a) Using the device to control the attitude of multiple devices;

[0026] (b) Capture camera images at each device pose;

[0027] (c) Extract image features from the captured camera images;

[0028] (d) Determine approximate values ​​for the parameters of the hand-eye pose; and

[0029] (e) Determining the parameters of the hand-eye pose in the case that an assumed incorrect device pose and incorrect extracted image features implicitly lead to an incorrect camera pose, comprising the following steps:

[0030] (e1) Statistical modeling is performed on the accuracy of the parameters describing the device's attitude and the accuracy of the extracted image features, wherein the number of parameters describing the device's attitude is at least equal to the number of the device's degrees of freedom.

[0031] (e2) Taking into account the accuracy from step (e1), the parameters of the hand-eye pose are optimized by simultaneously minimizing the back-projection error of the image features in the captured camera image and the error in the parameters describing the pose of the device.

[0032] (e3) Based on the optimization results from step (e2), variance component estimation is used to calculate the improved accuracy of the parameters describing the device attitude and the extracted image features; and (e4) steps (e1) to (e3) are repeated until the accuracy of the parameters describing the device attitude and the accuracy of the extracted image features converge.

[0033] Preferably, the device is a robot, and the device posture represents the robot posture. Alternatively, the device may be an automated industrial crane, and the device posture represents the crane posture.

[0034] In a first preferred embodiment, step (b) of capturing a camera image in each device pose includes capturing a camera image from a calibration object. More preferably, step (c) of extracting image features from the captured camera image includes extracting calibration marks on the calibration object in the captured camera image. Step (d) of determining approximate values ​​of the hand-eye pose parameters preferably includes the following steps: (d1) using the calibration marks extracted from the calibration object in the camera image to determine the camera pose for each device pose; and (d2) employing a hand-eye calibration method using the camera pose and the device pose to determine the approximate values ​​of the hand-eye pose parameters.

[0035] In an alternative preferred embodiment, step (b) of capturing camera images in each device pose includes capturing camera images of a scene suitable for extracting salient image points. More preferably, step (c) of extracting image features from the captured camera images includes extracting salient image points from the captured camera images. Preferably, step (d) of determining approximate values ​​for the hand-eye pose parameters includes the following steps: (d1) using the salient image points extracted in the camera images to determine a scaled camera pose for each device pose; (d2) using the scaled camera pose and the device pose to determine a normalization factor; (d3) normalizing the scaled camera pose using the normalization factor to determine the camera pose; and (d4) using a hand-eye calibration method using the camera pose and the device pose to determine the approximate values ​​for the parameters of the hand-eye pose.

[0036] Preferably, the method further includes determining an improved (calibrated) device posture from the results of optimizing the parameters of the hand-eye posture.

[0037] A further preferred step is to determine the accuracy of the device based on the results of optimizing the parameters of hand-eye posture.

[0038] According to another aspect, the present invention provides a method wherein steps (d) and (e2) are replaced by: (d) determining approximate values ​​of parameters of hand-eye pose and parameters of camera interior orientation; and (e2) taking into account the accuracy from step (e1), optimizing the parameters of hand-eye pose and parameters of camera interior orientation by simultaneously minimizing the backprojection error of image features in the captured camera image and the error in the parameters describing the device pose. In other words, the second aspect provides a method as follows:

[0039] (a) Using the device to control the attitude of multiple devices;

[0040] (b) Capture camera images at each device pose;

[0041] (c) Extract image features from the captured camera images;

[0042] (d) Determine approximate values ​​for the parameters of the hand-eye pose and the parameters of the camera's internal orientation; and

[0043] (e) Determining the parameters of the hand-eye pose in the case that an assumed incorrect device pose and incorrect extracted image features implicitly lead to an incorrect camera pose, comprising the following steps:

[0044] (e1) Statistical modeling is performed on the accuracy of the parameters describing the device's attitude and the accuracy of the extracted image features, wherein the number of parameters describing the device's attitude is at least equal to the number of the device's degrees of freedom.

[0045] (e2) Taking into account the accuracy from step (e1), optimize the parameters of the hand-eye pose and the parameters of the camera's internal orientation by simultaneously minimizing the back-projection error of the image features in the captured camera image and the error in the parameters describing the device's pose.

[0046] (e3) Based on the optimization results from step (e2), variance component estimation is used to calculate the improved accuracy of the parameters describing the device attitude and the extracted image features; and

[0047] (e4) Repeat steps (e1) to (e3) until the accuracy of the parameters describing the device attitude and the accuracy of the extracted image features converge.

[0048] First, the camera model and calibration model that form the basis of this invention are described, namely, the relationship between 3D world points and their projections in the camera. For ease of description, it is assumed that the camera-guided device is a camera-guided robot. This description can be readily applied by those skilled in the art to other camera-guided devices as described above. Furthermore, it should be understood that the camera is mounted on the end effector of the robot. Therefore, the description of this invention primarily relates to the case of a moving camera. It is known from the literature that the case of a stationary camera is equivalent to the case of a moving camera. Therefore, in the following, the case of a stationary camera will only be discussed where the equivalence is not obvious. Subsequently, three alternative optimization methods for hand-eye calibration are described. Finally, methods for providing the necessary initial values ​​for unknowns during the optimization process are described.

[0049] Camera model

[0050] In a preferred embodiment of the invention, the camera is described by a perspective camera model as described by Steger et al. ([Chapter 3.9.1], 2018): when using homogeneous coordinates, the 3D point p in the WCS is... w The point p in the CCS is transformed by the following formula. c

[0051] (4)

[0052] Where t = (t x , t y , t z ) T It is a translation vector and R is a rotation matrix parameterized by Euler angles: R = R x (α)R y (β)R z (γ). These parameters (t) x , t y , t zThe external orientation (α, β, γ) describes the camera's external orientation and thus represents a rigid transformation in 3D. In alternative embodiments of the invention, the external orientation is represented by another suitable parameterization. For example, the rotational portion of the external orientation may be represented by quaternions, or the entire external orientation may be represented by double quaternions (Daniilidis, 1999).

[0053] Then, point p is expressed by the following formula. c =(x c y c , z c ) T Projected onto the image plane:

[0054] (5)

[0055] Wherein, the chamber constant of the lens is c. In alternative embodiments of the invention, other camera models may be used instead (e.g., for telecentric lenses, Steger et al. [Section 3.9], 2018). Subsequently, to account for lens distortion, the undistorted point (x) u y u ) T Distorted into (x d y d ) T In a preferred embodiment of the invention, the division model (Lenz and Fritsch, 1990) is used to model lens distortion:

[0056] (6)

[0057] It simulates radially symmetric distortion using parameter k. Therefore, it utilizes... In another preferred embodiment of the invention, a polynomial model (Brown, 1971) is used, which simulates radial symmetry and tangential distortion. In alternative embodiments of the invention, other alternative distortion models may be used in an obvious manner without departing from the scope of the invention.

[0058] Finally, the distortion point (x) d y d ) T Transform to image coordinate system:

[0059] (7)

[0060] Among them (c x c y ) T Represents the principal point of the image and s x and s y Describe the horizontal and vertical distances between adjacent sensor elements.

[0061] The six parameters of the division model are i = (c, k, s) x s y c x c y ) T (or ten parameters of a polynomial model) describe the camera's internal orientation i.

[0062] Calibration Model

[0063] In hand-eye calibration, the device is moved to different device poses. In the case of the mobile camera robot example used in the present invention, the robot's tool is moved to n. r Different poses were captured, and camera images were captured in each of these robot poses. In the case of calibration based on a calibration object, the calibration object was placed at a fixed position in the robot's workspace (see...). Figure 1 In the case of self-calibration, camera images are captured instead from scenes suitable for extracting salient image points. This could be, for example, capturing a fully structured object or an arbitrary but structured background scene. Let... t H b,j The robot controller is in pose j (j = 1, ..., n) r The robot's pose returned at position p. Further, let p... k (k = 1, ..., n) w ) is a 3D world point (given in WCS) in the calibration object or scene, and its 2D projection onto the image of the robot pose j is p. j,k Then, the projection of 3D points onto the image is described by the following formula:

[0064] (8)p j,k =π( c H t t H b,j b H w p k ,i),

[0065] Where π(p) c i) represents the point p given in the CCS. c The projection onto the image. For example, in the case of the division model, π(p c ,i) corresponds to the continuous execution of equation (5), using the inverses of equations (6) and (7) of the internal orientation i.

[0066] Finally, vector e t,j Including robot posture t H b,j The transformation parameters, vector e c Including unknown hand-eye posturesc H t The transformation parameters, and vector e b Including unknown postures b H w The transformation parameters (and the external orientation parameters mentioned above) c H w (Similarly). In a preferred embodiment of the invention, the transformation parameters are three components of translation and three Euler rotation angles. In an alternative embodiment, the rotational components of the transformation are described by quaternions (4 parameters) or the transformation is described by double quaternions (8 parameters). The invention further allows for other alternative parameterizations using these transformations, thus making the invention easily integrateable into existing methods. However, describing robot pose... t H b,j The number of parameters must at least correspond to the number of degrees of freedom of the robot. For example, while most industrial robots cover 6 degrees of freedom, there are special robot designs that allow for fewer degrees of freedom. An example is the SCARA robot, which has only 4 degrees of freedom (3 translations and 1 rotation) (Ulrich and Steger, 2016). Conversely, overparameterization, such as that that exists when using quaternions or double quaternions, is possible. Other camera-guided devices can also have limited degrees of freedom. For example, some ground robots can move inside an airplane but cannot leave it. This is because it means they can only rotate about one axis and move in two directions, so they only have 3 degrees of freedom (1 rotation and 2 translations).

[0067] In the case of a stationary camera, the calibration object (for calibration based on the calibration object) or a fully structured object (for self-calibration) is mounted on the robot's end effector and thus moves with the robot. The stationary camera then captures camera images of the co-moving object in each approaching robot pose. In the case of self-calibration, ensuring the background is as uniform and unstructured as possible can be helpful for robust and automatic detection of the object's position in the camera images. (Poses in moving situations) c H t attitude corresponding to the static state c H b Attitude during movement b H w attitude corresponding to the static state t H W However, posture c H w and t H b,j The role remains unchanged.

[0068] Parameter estimation in Gauss-Markov models

[0069] In the following text, a distinction will be made between functional models and stochastic models. (Wrobel and Wrobel, 2016). Functional models describe the relationship between observed and unknown parameters. In stochastic models, observed values ​​and unknowns are treated as random variables with uncertainty, which is described by (co)variance.

[0070] Assuming the robot initially has an error-free posture, the hand-eye calibration problem can be solved using a so-called Gaussian-Markov model ( And Wrobel [Chapter 4.4], 2016) to formulate, since all erroneous observations l can be expressed as a function l = f(x) of unknown parameters x, where f describes the functional model and corresponds to equation (8).

[0071] Vector l contains the measured image points That is, image features extracted from captured camera images. In the case of calibration based on a calibration object, in a preferred embodiment of the invention, the extracted image features represent the projection of the center of the circular marker onto the calibration ontology (Steger and Ulich, 2018). In an alternative embodiment of the invention, they represent the projection of the intersections of a checkerboard pattern onto the calibration ontology (OpenCV, 2021). In the case of self-calibration, the extracted image features represent salient image points, which are computed in the image using appropriate image processing operators and mapped to each other on different images. Examples of such image processing operators are... Point extractor ( 1994), Harris point extractor (Harris and Stephens, 1988), and SIFT point extractor (Lowe, 2004).

[0072] If a 3D point is not visible in a particular image, the corresponding entry is omitted in l. The sum of the measured image points across all images is determined by n. i This represents the number of observations, n. l For 2n i For each observed image point, two equations of form (8) are given. Therefore, 2n is obtained. i Several equations. The structure of vector x depends on the application scenario and can be determined according to the following set of rules:

[0073] For all scenarios, x includes hand-eye pose: x := e c One characteristic to consider for SCARA robots is that not all parameters of the hand-eye pose can be determined. For example, if the hand-eye pose is parameterized by three translation angles and three Euler angles, the robot pose parameter t cannot be determined when calibrating the SCARA robot. zIn this case, t z It is set to 0 and excluded from optimization (Ulrich and Steger, 2016).

[0074] For calibration based on the calibration object, x must be determined by the unknown e. b Extended: Conversely, in the case of self-calibration, 3D points can be directly reconstructed in the BCS. Therefore, in this case, e b We can eliminate the unknowns and for e b The case described by 3 rotation parameters and 3 translation parameters is internally set to, for example, e. b =(0,0,0,0,0,0) T For other parameterizations, parameters can be selected to describe the cell image in the selected parameterization.

[0075] If the camera's interior orientation is unknown and should be co-estimated simultaneously, then the vector with unknowns is extended by the parameters of the interior orientation: x := (x T i T ) T Note that s y Typically, it must be excluded from calibration to obtain a unique solution (Steger et al., [Chapter 3.9.4.2], 2018).

[0076] • In the case of self-calibration, x must be extended from the coordinates of the 3D point:

[0077]

[0078] Let n x It is the final number of unknown parameters in x.

[0079] The stochastic model specifies the statistical properties of the observation process. In the current case, we assume erroneously extracted image features. We also assume that the image points are uncorrelated and measured with the same accuracy, then the n of the observations... l ×n l The weighting coefficient matrix is ​​set as the identity matrix: Q ll =I.

[0080] For linearization, establish 2n i ×n x A Jacobian matrix A, which includes the first derivative of f with respect to the unknown x, at the position of the initial value or the unknown x. (0) Evaluation at the current approximation (the determination of the initial value is described below):

[0081] (9)

[0082] Using Δl=lf(x (0) and weight matrix The correction value for the unknown can now be calculated. This is accomplished by solving the following equation:

[0083] (10)

[0084] Then, for example, by using the Cholesky decomposition of the sparse matrix (the top operator ^ the estimate in descriptive statistics). Therefore, the improved unknowns lead to... Finally, equations (9) and (10) are repeated until convergence. This corresponds to minimizing the backprojection error of the 3D points into the camera image.

[0085] After convergence, the covariance matrix of the original observations is obtained through... The variance factor is obtained. residual And redundancy r=n l -n x Corrected observations are obtained through... Obtained. The covariance matrix of the estimated unknowns is obtained through variance propagation. get.

[0086] Below, we now describe two alternative procedural models (parameter estimation in the Gauss-Helmert model and parameter estimation in the Gauss-Markov model with dummy unknowns) for determining the parameters of hand-eye pose under the assumption of incorrect robot pose and incorrect extraction of image features for implicitly leading to incorrect camera pose:

[0087] Parameter estimation in Gauss-Helmert models

[0088] To correct erroneous robot poses, in addition to the image coordinates of the extracted image features, these coordinates must also be incorporated as observations. Therefore, the observations can no longer be expressed as functions of unknowns. Consequently, parameter estimation is no longer easily achieved in Gauss-Markov models. In one embodiment of the invention, therefore, in the Gauss-Helmholtz model (… Parameter estimation is performed in Wrobel [Chapter 4.8], 2016. The functional model becomes f(x, l) = 0.

[0089] In this model, vector l contains erroneous observed image points and incorrect robot poses from the extracted image features. The number of equations observed is still 2n iHowever, for the case where the robot's pose is represented by 3 translation parameters and 3 rotation parameters, the number of observations is now n. l =2n i +6n r In the case of an alternative representation of robot pose, n l The value will change accordingly; for example, when using double quaternions, it will be n. l =2n i +8n r This is because a double quaternion contains 8 parameters. The vector x of the unknowns is equal to the vector of the Gauss-Markov model.

[0090] Compared to Gaussian-Markov models, stochastic models must additionally account for the uncertainty of robot posture. Tests on real systems have shown that the error in robot posture is meanless and Gaussian distributed (Strobl and Hirzinger, 2006).

[0091] Therefore, the following statistical modeling includes the accuracy of the parameters describing the robot's pose and the accuracy of the extracted image features, wherein the number of parameters describing the robot's pose is at least equal to the number of the robot's degrees of freedom.

[0092] Even assuming unrelated observations, the relative accuracy between observations from different groups must be known in order to establish Q. ll For the case where the robot pose is represented by three translation angles and three Euler angles, the three sets of observations are the image coordinates of the extracted image features, the Euler angles of the robot pose, and the translation components of the robot pose. Since this relative accuracy is unknown in practice, approximations are used to initialize Q. ll Then, the concept of variance components of the observed values ​​is used to estimate the actual variance. (Wrobel and Wrobel, 2016). In one embodiment of the invention, the identity matrix is ​​used to initialize Q. ll Q ll =I. However, this could lead to an unnecessarily large number of iterations to estimate the variance components. Therefore, in a preferred embodiment of the invention, a more reasonable value is used to initialize the variance. For example, one possibility when using Euler angles is to set the standard deviation of the image points to σ. i =0.1 pixels, set the standard deviation of Euler angles to σ a =0.1 degrees, and set the standard deviation of the translation component to σ. t =1mm. Then The function rep(y, n) generates a vector containing n copies of y.

[0093] The Jacobian matrix A is the same as in the Gauss-Markov model. In the Gauss-Helmholtz model, f must also be linearized with respect to the observations. The derivative with respect to the observations is stored in 2n. i ×n l In the Jacobian matrix B.

[0094] (11)

[0095] B's top left 2n i ×2n i The submatrix is ​​a diagonal matrix in which all diagonal elements are set to -1.

[0096] Using w = f(x) (0) Now we can calculate the correction value for the unknown. For this purpose...

[0097] (12)

[0098] Must be solved ( (and Wrobel, 2016).

[0099] Therefore, considering the accuracy of the parameters describing the robot's pose and the accuracy of the extracted image features, this model optimizes the hand-eye pose parameters by simultaneously minimizing the backprojection error of the image features in the captured camera images and the error of the parameters describing the robot's pose.

[0100] After optimization convergence, the results are used to estimate the variance components for each group of observations. When Euler angles are used, the variance component estimation produces σ. i , σ a and σ t The correction value is used to generate the correction matrix Q. ll .exist The procedure for estimating variance components is described in Wrobel ([Section 4.2.4], 2016). It can be found in... In Wrobel (2016), the search for Gauss-Helmert models and The calculation of variance component estimation yields parameters describing the robot's pose and improves the accuracy of the extracted image features.

[0101] Finally, the above optimization is performed again with the improved accuracy. Then, statistical modeling, optimization, and variance component estimation are repeated until the accuracy of the parameters converges. In practice, this typically takes 3 to 5 iterations.

[0102] Optionally, the covariance matrix of the observations is composed of a matrix with redundancy r = 2n. i -nx of Obtained. Through Obtain the covariance matrix of the estimated unknowns.

[0103] Another advantage of this method is that the camera pose does not need to be assumed to be known and error-free. This is achieved by allowing the camera pose to be transformed through erroneous observations or estimates. c H w = c H t t H b,j b H w Expressed as a cascade, this method implicitly leads to incorrect camera pose.

[0104] Parameter estimation in Gauss-Markov models with dummy unknowns

[0105] Due to the computationally intensive matrix operations in the Gauss-Helmert model, in a preferred embodiment of the invention, parameter estimation is performed in a more efficient variant of the Gauss-Markov model. This is equivalent to parameter estimation in the Gauss-Helmert model (Koch, 1999; Koch, 2007). Here, the robot pose is introduced as a so-called virtual unknown.

[0106] The basic idea here is to introduce the uncertain robot pose simultaneously as an observation and as an unknown. The first part of the functional model remains l = f(x). However, unlike the estimation in the Gauss-Markov model above, for the case where the robot pose is represented by 3 translation parameters and 3 rotation parameters, now 6 additional forms e are added for each robot pose j. t,j =f2(e t,j The observation equations are given by 2n. For example, using the quantum number of rotation adds 7 additional observation equations for each robot pose, while using two quantum numbers adds, for example, 8. Here, the function f2 corresponds to identity. This is for the case where the robot pose is represented by 3 translation parameters and 3 rotation parameters, and now 2n is given. i +66n r There are several equations. Therefore, vector l contains the observed image points and the robot's pose. Similar to the Gauss-Helmholtz model described above, robot pose is now also introduced as an unknown by correspondingly expanding the unknown vector:

[0107] Since l is the same as in the Gauss-Helmert model, the same stochastic model can be applied to the statistical modeling of accuracy, especially for cases where the robot pose is represented by 3 translation parameters and 3 rotation parameters. It can be initialized.

[0108] Thus, even in Gauss-Markov models with virtual unknowns, statistical modeling includes the accuracy of the parameters describing the robot's posture and the accuracy of the extracted image features, where the number of parameters describing the robot's posture is at least equal to the number of the robot's degrees of freedom.

[0109] Using equations (9) and (10) to perform (2n) i +6n r )×n x Jacobian matrix A (for robot pose represented by 6 parameters) and correction values The calculation. In this case, the lower left part of A contains 0 and the lower right part contains...

[0110] Therefore, this model also optimizes the hand-eye pose parameters by simultaneously minimizing the backprojection error of image features in the captured camera images and the error in the parameters describing the robot pose, while taking into account the accuracy of the parameters describing the robot pose and the accuracy of the extracted image features.

[0111] After convergence, use as follows The optimization results described in Wrobel (2016) or Niemeier ([Chapter 9.3], 2008) are used to estimate the variance components of the observation set. Variance component estimation produces parameters describing the robot's pose and improved accuracy of the extracted image features.

[0112] Finally, the above optimization is performed again with the improved accuracy. Then, statistical modeling, optimization, and variance component estimation are repeated until the accuracy of the parameters converges. In practice, this typically takes 3 to 5 iterations.

[0113] Optionally, as well as Calculate as in the parameter estimation of the Gauss-Markov model described above.

[0114] Similar to the previously described methods in the Gauss-Helmert model, the parameter estimation method in the Gauss-Markov model with dummy unknowns also has the advantage of not having to assume that the camera pose is known and error-free. This is achieved by allowing the camera pose to be transformed through erroneous observations or estimates. c H w = c H t t H b,j b H w Expressed as a cascade, this method implicitly leads to incorrect camera pose.

[0115] Determining the approximate value of the unknown parameter

[0116] In a preferred embodiment of the invention, the initial values ​​of the unknowns, and in particular approximate values ​​of the parameters of hand-eye posture, are set by the following procedure:

[0117] • Used for hand-eye postures e c and used for e b The approximations are obtained from any hand-eye calibration method known in the literature and suitable for these purposes. In a preferred embodiment of the invention, a linear method of hand-eye calibration, such as that of Danilidis (1999), is used for this purpose. In Danilidis's (1999) method, and in some other methods, this requires pre-determining the camera pose for each robot pose based on calibration marks extracted from the camera image. The determination of approximate values ​​for the hand-eye pose parameters is then performed using a hand-eye calibration method for both the camera pose and the robot pose.

[0118] • From the camera (s x and s y The data tables for the image principal point (c) and lens (c) are used to obtain initial values ​​for interior orientation. x c y ) T It is set to the center of the image and the distortion coefficient is set to 0.

[0119] In the case of self-calibration, the determination of approximate values ​​for these parameters is performed as follows: In a preferred embodiment of the invention, the SfM method is performed on the captured camera images. Possible suitable SfM implementations include, for example, COLMAP (Schenberger's method). (Frahm et al., 2016; Schönberg et al., 2016). The SfM method provides each image j with an internally oriented, scaled reconstructed 3D point p. k Image features p j,k Extracting 2D points and scaling the external orientation (i.e., camera pose). c H w,j The parameters.

[0120] In alternative embodiments of the invention, any other suitable method known in the literature for determining initial values ​​or additional knowledge from a particular application may be used to initialize the unknowns.

[0121] In the case of self-calibration, the scaling factor inherently unknown in the SfM method must be p k and c H w,jThe initial value is determined. For this purpose, the normalization factor is first determined using scaled camera pose and robot pose. In a preferred embodiment of the invention, camera motion is calculated for all robot pose pairs j1 and j2. and tool movement The two rigid 3D transformations are then converted into the parameters of the screw (Daniilidis, 1999). The congruence theorem (Chen, 1991) states that the translational parameter d of the screw during tool movement... t Translation parameter d of the screw moving with the camera c The same. Therefore, the ratio d t / d c This reflects the unknown scaling factor of the SfM method and therefore the sought normalization factor. In one embodiment of the invention, the average is calculated over the ratios of all pairs. In a preferred embodiment of the invention, for greater robustness, the median of the ratios of all pairs is calculated, ignoring its d. t or d c The ratio of noise levels less than the noise threshold.

[0122] Finally, the determined normalization factor was used to normalize the scaled reconstructed 3D point p. k And the external orientation of the scaling (i.e., camera pose). cH w,j .

[0123] Determining the accuracy of a robot

[0124] In both the Gauss-Helmholtz model and the Gauss-Markov model with dummy unknowns, the results of optimizing the parameters of hand-eye pose can be used to determine the robot's accuracy. Estimation of the variance components allows for meaningful statements about the robot's accuracy, which typically requires robot calibration. The robot's accuracy can be directly derived from the matrix C produced after variance component estimation. ll The robot posture is described by three translation parameters and three rotation parameters. For this purpose, the variances of the translation and rotation parameters of the robot posture are averaged over all robot postures, so that the accuracy of the robot can be presented in the form of two values.

[0125] Determination of robot posture after calibration

[0126] In both the Gauss-Helmholtz model and the Gauss-Markov model with virtual unknowns, the improved (calibrated) robot pose can be determined based on the results of optimizing the parameters of the hand-eye pose. By introducing the robot pose as an observation, the vector... In addition to balanced image coordinates, balanced robot poses are also included. These can be considered as corrected or calibrated robot poses. Therefore, they can be used as a basis for simple and inexpensive robot calibration.

[0127] Advantages of the present invention

[0128] Explicitly modeling the uncertainties of a device is advantageous for hand-eye calibration. It improves accuracy, provides the calibrated device orientation, and offers information about the device's uncertainties. This is important, for example, in industrial robots used for tasks requiring high accuracy. Parameter estimation in a Gaussian-Markov model with virtual unknowns, combined with estimation of the variance components proposed in this invention, provides a statistically reasonable representation of the problem. Different hand-eye calibration scenarios (e.g., calibration based on the calibration object, self-calibration; calibration of different devices (e.g., articulated industrial robots, SCARA industrial robots, ground exploration robots); unknown or known internal orientation) can be readily represented by adding or removing appropriate parameters from the parameter vector. Therefore, a wide range of applications can benefit from this invention.

[0129] References

[0130] M. Abderrahim, A. Khamis, S. Garrido and L. Moreno, "Accuracy and calibration issues of industrial manipulators", Industrial Robots: Programming, Simulation and Applications, IntechOpen, 2006, pp. 131–146.

[0131] N. Andreff, R. Horaud and B. Espiau, “Robot hand-eye calibration using structure-from-motion”, International Journal of Robotics Research, Vol. 20, No. 3, pp. 228–248, 2001.

[0132] DCBrown, “Close-range camera calibration”, Photogrammetric Engineering, Vol. 37, No. 8, pp. 855–866, August 1971.

[0133] HHChen, “A screw motion approach to uniqueness analysis of headeye geometry”, Computer Vision and Pattern Recognition, 1991, pp. 145–151.

[0134] K. Danilidis, “Hand-eye calibration using dual quaternions”, International Journal of Robotics Research, Vol. 18, No. 3, pp. 286–298, 1999.

[0135] F. Dornaika and R. Horaud, “Simultaneous robot-world and hand-eye calibration”, IEEE Transactions on Robotics and Automation, Vol. 14, No. 4, pp. 617–622, 1998.

[0136] W. "A Framework for Low Level Feature Extraction", see Jan-Olof Eklundh, Springer, 1994, ed., Lectures on Computer Science, Volume 801, Third European Conference on Computer Vision, pp. 383–394.

[0137] W. With BP Wrobel, Photogrammetry Computer Vision: Statistics, Geometry, Orientation and Reconstruction, Springer International, 2016.

[0138] C. Harris, M. Stephens, "A Combined Corner and Edge Detector", Proceedings of the Fourth Alvi Vision Conference, 1988, pp. 147–151.

[0139] A. Hofhauser, C. Steger, and N. Navab, “Perspective planarshape matching”, in Image Processing: Machine Vision Applications 2, edited by K.S. Niel and D. Fofi, ser. Proc. SPIE 7251, 2009.

[0140] R. Horaud and F. Dornaika, “Hand-eye calibration”, International Journal of Robotics Research, 1995, vol.14, no.3, pp.195–210.

[0141] ISO 9283:1998, "Manipulating industrial robots—performance criteria and related test methods", 1998.

[0142] K.-R. Koch, Parameter Estimation and Hypothesis Testing in Linear Models, Berlin-Heidelberg: Springer, 1999.

[0143] K.-R. Koch, Introduction to Bayesian Statistics, 2nd Edition, Heidelberg: Springer, 2007.

[0144] K. Koide and E. Menegatti, “General hand-eye calibration based on reprojection error minimization”, IEEE Robotics and Automation Communications, April 2019, vol.4, no.2, pp.1021–1028.

[0145] R. Lenz and D. Fritsch, “Accuracy of videometry with CCD sensors”, Journal of Photogrammetry and Remote Sensing, 1990, vol.45, no.2, pp.90–110.

[0146] DGLowe, “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision 60(2): pp.91–110, 2004.

[0147] W. Niemeier, "Ausgleichungsrechung – Statistics (Statistische) Auswertemethoden", 2.überarbeitete und erweiterte Auflage, de Gruyter, Berlin, 2008.

[0148] H. Nguyen and Q. Pham, “On the covariance of X in AX = XB”, IEEE Transactions on Robotics, vol.34, no.6, pp.1651–1658, 2018.

[0149] OpenCV, "Open Computer Vision Library", version 4.5.2, 2021 open source CV module calib3d, https: / / docs.opencv.org / 4.5.2 / d9 / d0c / group__calib3d.html, accessed 2021.7.4.

[0150] M.Placzek and Piszczek, “Testing of an industrial robot's accuracy and repeatability in off and online environments”, Eksploatacja i Niezawodnosc—Maintenance and Reliability, vol.20, no.3, pp.455–464, 2018.

[0151] J. Schmidt and H. Niemann, “Data selection for hand-eye calibration: A vector quantization approach”, International Journal of Robotics Research, vol. 27, no. 9, pp. 1027–1053, 2008.

[0152] J. Schmidt, F. Vogt, and H. Niemann, “Robust hand–eye calibration of an endoscopic surgery robot using dual quaternions,” Pattern Recognition, B. Michaelis and G. Krell, Eds, Springer Berlin-Heidelberg, 2003, pp. 548–556.

[0153] J. Schmidt, F. Vogt and H. Niemann, “Calibration-free hand-eye calibration: A structure-from-motion approach”, Pattern Recognition, WGKropatsch, R. Sablatnig and A. Hanbury, Eds. Springer Berlin-Heidelberg, 2005, pp. 67–74.

[0154] JL And J.M. Frahm, “Structure-from-motion revisited”, Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

[0155] JL E. Zheng, M. Pollefeys and J.-M. Frahm, “Pixelwise view selection for unstructured multi-view stereo”, European Conference on Computer Vision (ECCV), 2016.

[0156] P. Shiakolas, K. Conrad, and T. Yih, “On the accuracy, repeatability, and degree of influence of kinematics parameters for industrial robots,” International Journal of Modeling and Simulation, vol. 22, no. 4, pp. 245–254, 2002.

[0157] K. H. Strobl and G. Hirzinger, “Optimal hand-eye calibration”, IEEE / RSJ International Conference on Intelligent Robots and Systems, 2006, pp. 4647–4653.

[0158] C. Steger, M. Ulrich and C. Wiedemann, Machine Vision Algorithms and Applications, 2nd Edition, Weinheim: Wiley-VCH, 2018.

[0159] A. Tabb and K.M. Yousef, “Solving the robot-world hand-eye(s) calibration problem with iterative methods”, Machine Vision and Its Applications, August 2017, vol.28, no.5, pp.569–590.

[0160] RYTsai and RKLenz, “A new technique for fully autonomous and efficient 3D robotics hand / eye calibration,” IEEE Transactions on Robotics and Automation, vol.5, no.3, pp.345–358, June 1989.

[0161] M. Ulrich and C. Steger, “Hand-eye calibration of SCARA robots using dual quaternions”, Pattern Recognition and Image Analysis, vol. 26, no. 1, pp. 231–239, 2016.

[0162] M. Ulrich, C. Wiedemann and C. Steger, “Combining scale-space and similarity-based aspect graphs for fast 3D object recognition”, Journal of Pattern Analysis and Machine Intelligence, October 2012, vol.34, no.10, pp.1902–1914.

[0163] Z. Zhang, “A flexible new technique for camera calibration”, Journal of Pattern Analysis and Machine Intelligence, November 2000, vol.22, no.11, pp.1330–1334.

Claims

1. A hand-eye calibration method for determining parameters of the hand-eye posture of a camera-guided device, the method comprising the following steps: (a) Using the device to control the attitude of multiple devices; (b) Capture camera images at each device pose; (c) Extract image features from the captured camera images; as well as (d) Determine approximate values ​​for the parameters of the hand-eye posture; characterized in that: (e) Determining the parameters of the hand-eye pose in the case that an assumed incorrect device pose and incorrect extracted image features implicitly lead to an incorrect camera pose, comprising the following steps: (e1) Statistical modeling is performed on the accuracy of the parameters describing the device attitude and the accuracy of the extracted image features using a weight matrix, a weight coefficient matrix or a covariance matrix, wherein the number of parameters describing the device attitude is at least equal to the number of degrees of freedom of the device. (e2) Taking into account the accuracy from step (e1), the parameters of the hand-eye pose are optimized by simultaneously minimizing the back-projection error of the image features in the captured camera image and the error in the parameters describing the pose of the device. (e3) Based on the optimization results from step (e2), variance component estimation is used to calculate the improved accuracy of the parameters describing the device attitude and the extracted image features; and (e4) Repeat steps (e1) to (e3) until the accuracy of the parameters describing the device attitude and the accuracy of the extracted image features converge.

2. The method according to claim 1, wherein the device is a robot and the device posture represents the robot posture.

3. The method according to claim 1 or 2, wherein capturing camera images in each device pose in step (b) comprises: Capture a camera image from the calibration object, wherein step (c) of extracting image features from the captured camera image includes: extracting calibration marks on the calibration object in the captured camera image.

4. The method according to claim 3, wherein determining the approximate value of the hand-eye posture parameters in step (d) includes the following steps: (d1) The camera pose of each device pose is determined using the calibration mark on the calibration object extracted from the camera image; as well as (d2) The approximate values ​​of the hand-eye posture parameters are determined using a hand-eye calibration method that uses the camera posture and the device posture.

5. The method according to claim 1 or 2, wherein capturing camera images in each device pose in step (b) comprises: Capture a camera image of a scene suitable for extracting salient image points, and wherein the extraction of image features in the captured camera image in step (c) includes: extracting salient image points in the captured camera image.

6. The method according to claim 5, wherein determining the approximate value of the hand-eye posture parameter in step (d) includes the following steps: (d1) Using the salient image points extracted from the camera image to determine a scaled camera pose for each device pose, wherein the scaled camera pose is a camera pose affected by an unknown scaling factor. (d2) The normalization factor is determined as the relationship between the translation parameters between the device pose and the scaled camera pose; (d3) The camera pose is determined by normalizing the scaled camera pose using the normalization factor; as well as (d4) The approximate value of the parameter of the hand-eye posture is determined by using a hand-eye calibration method that uses the camera posture and the device posture.

7. The method according to claim 1, further comprising the following step: (f) Determine an improved device posture from the results of optimizing the parameters of the hand-eye posture.

8. The method according to claim 1, further comprising the following step: (f) Determine the accuracy of the device from the results of optimizing the parameters of the hand-eye posture.

9. The method of claim 1, wherein steps (d) and (e2) are replaced by the following steps: (d) Determine approximate values ​​for the parameters of the hand-eye pose and the parameters of the camera's internal orientation; (e2) Taking into account the accuracy from step (e1), the parameters of the hand-eye pose and the parameters of the camera's internal orientation are optimized by simultaneously minimizing the back-projection error of the image features in the captured camera image and the error in the parameters describing the device pose.

Citation Information

Patent Citations

  • Machine vision system and calibration method implemented by the machine vision system

    CN107871328A

  • System and method for robust calibration between a machine vision system and a robot

    US20110280472A1