Hand-eye calibration of camera guide devices

Abstract

To provide a method for hand-eye calibration for supporting both variants of deformation of calibration-object-based calibration and self-calibration without requiring known 3D points.SOLUTION: A method comprises the steps of: controlling a plurality of device attitudes using a camera guide device; capturing camera images in respective device attitudes; extracting image features in the camera images; determining approximate values of parameters of the hand-eye attitude; statistically modeling accuracy of the parameters and accuracy of the image features; optimizing the parameters by simultaneously minimizing back projection errors of the image features and parameter errors by taking account of the accuracy; using variance component estimation to calculate improved accuracy of the parameters and image features based on the optimization results; and repeating the aforementioned steps until the accuracy of the parameters and image features converges.SELECTED DRAWING: None

Description

[Technical Field]

[0001] Summary of the Invention This invention describes a general framework for hand-eye calibration of camera-guided devices, which requires determining a rigid 3D transformation between the device and the camera. An example of such a device is a camera-guided robot. In contrast to conventional methods, the robot's imprecision is explicitly modeled in a statistically sound manner. This is relevant, for example, to industrial robots. Although modern industrial robots have high accuracy, their absolute accuracy is usually much lower. This imprecision can deteriorate hand-eye calibration results if the imprecision is not explicitly considered. The proposed method not only results in high accuracy of the calculated hand-eye poses, but also provides reliable information about the robot's accuracy. Furthermore, it provides corrected robot poses that can be used for simple and inexpensive robot calibration. The described framework is general in several respects, supporting both the use of calibration bodies and self-calibration without the need for known 3D points. This optionally allows for simultaneous calibration of internal camera parameters. The framework is also general with respect to robot type, supporting, for example, both articulated arms and SCARA robots. In addition to applications involving industrial robots, other applications using camera-guided robots or other camera-guided devices can benefit from this invention. [Background technology]

[0002] Background technology Hand-eye calibration is essential for applications that use camera-guided systems. In the case of robots, this determines the rigid 3D transformation (pose) between the robot and the camera (hand-eye pose). This allows measurements taken in the camera coordinate system to be transformed into the robot coordinate system. For example, in industrial robotic bin-picking applications, the object pose in the camera coordinate system is determined using 3D object recognition (Hofhauser et al., 2009; Ulrich et al., 2012). To enable the robot to grasp the object, the object pose must be transformed into the robot's coordinate system.

[0003] For camera-guided medical robots, service robots, and humanoid robots, it is also necessary to determine the pose of the robot or robotic hand relative to the camera. For camera-guided autonomously navigating drones, hand-eye calibration determines the pose between the camera and the drone's reference frame. For example, the position of the drone's reference frame relative to a higher-level frame can be determined using a global navigation satellite system (GNSS) and an inertial navigation system (INS). Similar considerations apply to camera-guided ground robots (e.g., surveillance robots or rescue robots). In operating rooms, devices are increasingly being used to assist surgeons. In endoscopy, hand-eye calibration must be used to determine the camera's position in the endoscope's or operating room's coordinate system. This also applies to augmented reality applications, where hand-eye calibration must be used to determine the pose between the camera and the headset's position and orientation sensors. Camera-guided cranes can be used to automate processes in the construction industry, including automated loading and unloading of container ships. Here, too, the camera's position in the crane's coordinate system must be determined via hand-eye calibration. The focus of the present invention is on industrial robots, and therefore the following description is based on this scenario as an example, but obviously further applications in which camera-guided devices are used can benefit from the present invention. In principle, this applies to all applications in which hand-eye calibration is useful. Therefore, in the following, the term robot refers to all devices in which hand-eye calibration is used.

[0004] Generally, there are two scenarios for camera-guided robots (Steger et al., [Chapter 3.12.2], 2018). In the first scenario, the camera is attached to the robot's end effector and moves to different positions with the robot. In the second scenario, the camera is fixedly attached to the outside of the robot and therefore does not move relative to the robot base. The pose determined by hand-eye calibration is the relative pose of the camera with respect to the robot tool in the first scenario, or the relative pose of the camera with respect to the robot base in the second scenario. In the following, the description of the invention will refer to the first scenario with a moving camera. However, it can also be applied to the second scenario with a fixed camera in a similar manner.

[0005] Traditional hand-eye calibration techniques assume that the pose of a robot tool is precisely known relative to the robot base. Regarding robot accuracy, it is important to distinguish between robot repeatability and absolute accuracy (ISO 9283:1998). Repeatability describes a robot's ability to repeatedly move a tool to the same pose. Absolute accuracy is the robot's ability to move a tool to a specific pose in 3D space. Modern industrial robots typically offer very high repeatability, ranging from 0.02 to 0.15 mm (Shiakolas et al., 2002; Abderrahim et al., 2006; Placzek and Piszszek, 2018). High repeatability is sufficient for applications where the robot tool always assumes the exact same pre-trained pose. However, for offline-programmed robots, especially camera-guided robots, high absolute pose accuracy is also important. Unfortunately, a robot's absolute pose accuracy is often much lower than its repeatability. Absolute attitude accuracy typically ranges from 0.1 mm to 10.0 mm, and orientation accuracy often ranges from about 0.2 degrees to a few degrees ( Abderrahim et al., 2006 ; Placzek and Piszczek, 2018 ).

[0006] Robot calibration can improve accuracy by up to an order of magnitude, and in rare cases even to replication (Shiakolas et al., 2002). Unfortunately, robot calibration often requires high-precision measurement equipment such as laser trackers (Abderrahim et al., 2006) and is therefore expensive and time-consuming.

[0007] Almost all conventional methods for hand-eye calibration assume error-free robot poses. Our approach explicitly models robot inaccuracies in a statistically sound way, resulting in higher accuracy of the calculated hand-eye poses. Furthermore, our approach provides reliable information about robot inaccuracies that would otherwise require high-precision measurement equipment. Because the described method also provides error-corrected (calibrated) robot poses, the method also enables simple and inexpensive robot calibration.

[0008] Most existing approaches to hand-eye calibration require the capture of multiple images of a calibration object. Some more flexible and user-friendly solutions avoid the use of a calibration object. The method according to the invention supports both calibration-object-based and self-calibration variants without requiring known 3D points. Furthermore, the method according to the invention optionally allows simultaneous calibration of internal camera parameters for both variants, thus providing a high degree of user-friendliness.

[0009] prior art FIG. 1 shows the coordinate systems relevant to hand-eye calibration of a robot (device) with a moving camera.

[0010]

number

[0011] Equation (2) is often written as:

[0012]

number

[0013] There are several linear methods that essentially solve equations (2) or (3), e.g., Tsai and Lenz, 1989; Chen, 1991; Horaud and Dornaika, 1995; Dornaika and Horaud, 1998; Daniilidis, 1999; Andreff et al., 2001; Schmidt et al., 2003; Ulrich and Steger, 2016. These determine the rotation and translation distribution of the hand-eye pose sequentially or simultaneously. The latter has the advantage that rotation errors do not propagate, thereby amplifying translation errors. The methods that solve equation (3) require the selection of an appropriate pair of robot poses to calculate A and B. Criteria for selecting an appropriate pair of poses have been proposed in Tsai and Lenz, 1989; Schmidt et al., 2003; Schmidt and Niemann, 2008. However, the observed information (A i and B i ) is optimally utilized. Furthermore, these methods assume error-free robot poses (Tsai and Lenz, 1989).

[0014] Linear methods typically minimize algebraic errors, limiting their accuracy. Therefore, they are often used to initialize subsequent nonlinear optimizations to achieve higher accuracy. Most nonlinear methods that minimize either algebraic or geometric errors (Horaud and Dornaika, 1995; Dornaika and Horaud, 1998; Daniilidis, 1999; Schmidt et al., 2005; Ulrich and Steger, 2016; Steger [Chapter 3.13.5], 2018) also assume an error-free robot pose and often face the additional problem of how to weight the rotational and translational error components relative to each other. Strobl and Hirzinger (2006) minimize a weighted sum of rotational and translational error distributions, where the weights of the error components are statistically derived. This minimizes the error of the robot pose. b Ht can be taken into account, but the error-free camera pose c H W are considered as inputs. Nguyen and Pham (2018) solve the rotation and translation parts of equation (3) sequentially and propagate the errors in A and B to X.

[0015] Another class of methods minimizes the backprojection error of 3D world points onto a calibration object, similar to camera calibration methods such as Zhang (2000) (e.g., Tabb and Yousef, 2017). Key advantages of this class of methods are that they eliminate the need for explicit estimation of the camera position in each image during preprocessing, do not require pose pair preselection, and minimize observed erroneous measurements, i.e., significant geometric errors in the space of image points. Another advantage is that minimizing the backprojection error also enables simultaneous estimation of the inner camera parameters. However, these aforementioned methods also assume an error-free robot pose. Koide and Menegatti (2019) presented a pose graph optimization framework for hand-eye calibration in which the robot pose error is minimized in addition to the backprojection error of the calibration object points. The benefits of considering the uncertainty in the robot pose have been demonstrated through experiments. Unfortunately, the publication does not provide details regarding the probabilistic model, optimization, and weighting of different error components relative to each other.

[0016] A self-calibration technique is used to perform hand-eye calibration without a calibration object. In this technique, unknown 3D points are tracked in an image sequence acquired from a given robot motion. The internal camera parameters and 3D points are estimated simultaneously with the hand-eye pose. In Andreff et al. (2001), a structure-from-motion (SfM) approach is used, and unknown scaling factors of the SfM results are integrated into the equations. This idea is taken up in Schmidt et al. (2005), where a post-processing step to enforce orthogonality of the rotation matrix is ​​avoided by introducing unknown scaling factors into the equations of Horaud and Dornaika (1995) and Daniilidis (1999). The described self-calibration technique minimizes algebraic errors, which limits the accuracy achievable with these techniques. Summary of the Invention [Means for solving the problem]

[0017] The method according to the invention combines the aforementioned advantages of minimizing the backprojection error, the advantages of probabilistically modeling the inaccuracies of all measured observations (image points and robot poses), the flexibility to perform either a calibration object-based calibration or a self-calibration, and the possibility to use known internal camera parameters or to estimate them simultaneously.

[0018] According to a first aspect, the present invention provides a hand-eye calibration method for determining hand-eye pose parameters of a camera guide device, in this regard the method comprising: (a) controlling a plurality of device attitudes using the device; (b) capturing camera images at each device orientation; (c) extracting image features in the captured camera image; (d) determining approximate values ​​of the hand-eye pose parameters; (e) determining hand-eye pose parameters assuming incorrect device poses and incorrectly extracted image features to implicitly account for incorrect camera poses, (e1) statistically modeling the accuracy of parameters describing the device pose and the accuracy of extracted image features, wherein the number of parameters describing the device pose is at least equal to the number of degrees of freedom of the device; (e2) optimizing the parameters of the hand-eye pose by simultaneously minimizing the back-projection error of the image features in the captured camera images and the error of the parameters describing the device pose, taking into account the accuracy from step (e1); (e3) calculating parameters describing the device pose and improved accuracy of the extracted image features based on the optimization results from step (e2) using variance component estimation; (e4) repeating steps (e1) to (e3) until the accuracy of the parameters describing the device pose and the accuracy of the extracted image features converge.

[0019] Preferably, the device is a robot and the device pose represents the robot pose. Alternatively, the device may be an automated industrial crane and the device pose represents the crane pose.

[0020] In a first preferred embodiment, capturing camera images at each device pose in step (b) includes capturing camera images from a calibration object. More preferably, extracting image features in the captured camera images in step (c) includes extracting calibration marks on the calibration object in the captured camera images. Determining approximate hand-eye pose parameters in step (d) preferably includes the steps of: (d1) determining a camera pose for each device pose using the calibration marks on the calibration object extracted in the camera images; and (d2) determining approximate hand-eye pose parameters using a hand-eye calibration technique that uses the camera pose and the device pose.

[0021] In an alternative preferred embodiment, capturing camera images at each device pose in step (b) includes capturing camera images of a scene suitable for extracting salient image points. More preferably, extracting image features in the captured camera images in step (c) includes extracting salient image points in the captured camera images. Preferably, determining approximate values ​​of hand-eye pose parameters in step (d) includes: (d1) determining a scaled camera pose for each device pose using the extracted salient image points in the camera images; (d2) determining a normalization factor using the scaled camera pose and the device pose; (d3) determining the camera pose by normalizing the scaled camera pose using the normalization factor; and (d4) determining approximate values ​​of the hand-eye pose parameters using a hand-eye calibration technique using the camera pose and the device pose.

[0022] Preferably, the method further comprises determining an improved (calibrated) device pose from the results of optimizing the parameters of the hand-eye pose.

[0023] A further step of determining the accuracy of the system from the results of optimizing the parameters of the hand-eye pose is further preferred.

[0024] According to a further aspect, the present invention provides a method in which steps (d) and (e2) are replaced by: (d) determining approximations of the hand-eye pose parameters and the camera interior orientation parameters; and (e2) optimizing the hand-eye pose parameters and the camera interior orientation parameters by simultaneously minimizing the backprojection error of image features in the captured camera images and the error of the parameters describing the device pose, taking into account the accuracy from step (e1). In other words, the second aspect provides the following method.

[0025] (a) controlling a plurality of device attitudes using the device; (b) capturing camera images at each device position; (c) extracting image features within the captured camera image; (d) determining hand-eye pose parameters and approximations of said parameters of the camera interior orientation; (e) determining hand-eye pose parameters assuming incorrect device poses and incorrectly extracted image features to implicitly account for incorrect camera poses; (e1) statistically modeling the accuracy of parameters describing the device pose and the accuracy of extracted image features, wherein the number of parameters describing the device pose is at least equal to the number of degrees of freedom of the device; (e2) optimizing the hand-eye pose parameters and the camera interior orientation parameters by simultaneously minimizing the backprojection error of image features in the captured camera images and the error of the parameters describing the device pose, taking into account the accuracy from step (e1); (e3) calculating parameters describing the device pose and improved accuracy of the extracted image features based on the optimization results from step (e2) using variance component estimation; (e4) repeating steps (e1) to (e3) until the accuracy of the parameters describing the device pose and the accuracy of the extracted image features converge. [Brief explanation of the drawings]

[0026] [Figure 1] 1 shows the coordinate systems relevant to hand-eye calibration of a robot (device) with a moving camera. [Figure 2] 1 shows the coordinate systems relevant to robot (device) hand-eye calibration for the fixed camera case. DETAILED DESCRIPTION OF THE INVENTION

[0027] MODE FOR CARRYING OUT THE INVENTION First, we describe the camera model and calibration model underlying our invention, i.e., the relationship between 3D world points and their projection onto the camera. To facilitate the description of our invention, we assume that the camera-guided device is a camera-guided robot. The description can be easily applied to other camera-guided devices, such as those described above by those skilled in the art. Furthermore, it is understood that the camera is attached to the end effector of the robot. Therefore, our description primarily focuses on the moving camera case. It is known from the literature that the fixed camera case is equivalent to the moving camera case. Therefore, we will only discuss the fixed camera case below where equivalence is not clear. We then describe three alternative optimization methods for hand-eye calibration. Finally, we describe how to provide the initial values ​​needed for the unknowns in the optimization procedure.

[0028] Camera Model In a preferred embodiment of the present invention, the camera is described by a perspective camera as described in Steger et al. ([Chapter 3.9.1], 2018), and using homogeneous coordinates,

[0029]

number

[0030] Next,

[0031]

number

[0032] lastly,

[0033]

number

[0034] Calibration Model In hand-eye calibration, the device is moved to different device poses. In the example of a moving camera robot used to explain the invention, the robot's tool isr The robot is moved to different poses and camera images are captured at each of these robot poses. In the case of calibration object-based calibration, a calibration object is placed at a fixed position in the robot's workspace (see Figure 1). In the case of self-calibration, camera images are instead captured from a scene suitable for extracting salient image points. This could be, for example, the capture of a well-structured object or an arbitrary but structured background scene.

[0035]

number

[0036] lastly,

[0037]

number

[0038] In the case of a fixed camera, a calibration object (for calibration object-based calibration) or a well-structured object (for self-calibration) is attached to the robot's end effector and thus moves with the robot. The fixed camera then captures camera images of the co-moving object at each approximate robot pose. In the case of self-calibration, it may be useful to ensure that the background is as homogeneous and unstructured as possible so that the object's position can be detected as robustly and automatically as possible in the camera images.

[0039]

number

[0040] Parameter estimation in Gauss-Markov models In the following, we distinguish between functional models and probabilistic models (Forstner and Wrobel, 2016). Functional models describe the relationship between observations and unknown parameters. In probabilistic models, observations and unknowns are treated as random variables with uncertainty, and the uncertainty is described by (co)variance.

[0041] Assuming an error-free robot pose at the outset, the hand-eye calibration problem can be formulated in terms of the so-called Gauss-Markov model (Forstner and Wrobel [Chapter 4.4], 2016), where

[0042]

number

[0043] That is, they include extracted image features in the captured camera image. In the case of calibration object-based calibration, in a preferred embodiment of the present invention, the extracted image features represent the projection of the centers of circular marks on the calibration body (Steger and Ulrich, 2018). In an alternative embodiment of the present invention, they represent the projection of the intersection points of a checkerboard pattern on the calibration body (OpenCV, 2021). In the case of self-calibration, the extracted image features represent salient image points that are computed using appropriate image processing operators in the image and mapped to each other across different images. Examples of such image processing operators are the Förstner point extractor (Forstner, 1994), the Harris point extractor (Harris and Stephens, 1988), and the SIFT point extractor (Lowe, 2004).

[0044] If the 3D points are not visible in a particular image,

[0045]

number

[0046] For calibration object-based calibration,

[0047]

number

[0048] For self-calibration,

[0049]

number

[0050]

number

[0051] After convergence,

[0052]

number

[0053] In the following, we describe two alternative procedural models for determining the parameters of the hand-eye pose under the assumption of incorrect extracted image features for the incorrect robot pose and implicitly taking into account the incorrect camera pose: parameter estimation in the Gauss-Helmert model and parameter estimation in the Gauss-Markov model with fictitious unknowns.

[0054] Parameter estimation in the Gauss-Helmert model To account for incorrect robot poses, they must be introduced as observations in addition to the image coordinates of the extracted image features. Therefore, the observations can no longer be expressed as functions of unknowns. This makes parameter estimation difficult with the Gauss-Markov model. Therefore, in one embodiment of the present invention, parameter estimation is performed with the Gauss-Helmert model (Forstner and Wrobel [Chapter 4.8], 2016).

[0055]

number

[0056] However, the number of current observations is n where the robot pose is represented by three translational parameters and three rotational parameters. l =2n i +6n r For alternative representations of the robot pose, n l changes accordingly, for example, when using a double quaternion, n l =2n i +8n r The vector of unknowns x is the same as that in the Gauss-Markov model.

[0057] Compared to the Gauss-Markov model, the probabilistic model must additionally take into account the uncertainty of the robot pose. Tests on real systems have shown that the robot pose errors are mean-free and Gaussian distributed (Strobl and Hirzinger, 2006).

[0058] Therefore, the following statistical modeling includes the accuracy of the parameters describing the robot's pose and the accuracy of the extracted image features, where the number of parameters describing the robot's pose is at least equal to the number of degrees of freedom of the robot.

[0059] Even assuming uncorrelated observations,

[0060]

number

[0061]

number

[0062] Therefore, this model optimizes the hand-eye pose parameters by simultaneously minimizing the backprojection error of image features in the captured camera images and the error in the parameters describing the robot pose, taking into account the accuracy of the parameters describing the robot pose and the accuracy of the extracted image features.

[0063] After the optimization has converged, the optimization results are used to estimate the variance components for each observation group. If Euler angles are used, the variance component estimates are

[0064]

number

[0065] The calculation of can be found in Forstner and Wrobel (2016). Variance component estimation leads to improved accuracy of the parameters describing the robot pose and the extracted image features.

[0066] Finally, the above optimization is run again with improved accuracy. Finally, statistical modeling, optimization, and variance component estimation are repeated until parameter accuracy converges. In practice, this typically occurs after 3-5 iterations.

[0067] Optionally, the covariance matrix of the observations is

[0068]

number

[0069] Parameter estimation in Gauss-Markov models using fictitious unknowns Due to the computationally intensive matrix operations in the Gauss-Helmert model, in a preferred embodiment of the present invention, parameter estimation is performed with a more efficient variant of the Gauss-Markov model, which is equivalent to parameter estimation in the Gauss-Helmert model (Koch, 1999; Koch, 2007), where the robot pose is introduced as a so-called fictitious unknown.

[0070] The basic idea here is to introduce the uncertain robot pose as both an observation and an unknown. The first part of the functional model is still l=f(x). However, unlike the Gauss-Markov model estimation mentioned above, here we use

[0071]

number

[0072] Since l is the same as the Gauss-Helmert model, the same probability model can be applied to statistical modeling of accuracy, and if the robot pose is described by three translational parameters and three rotational parameters, then:

[0073]

number

[0074] can be initialized. Therefore, even for a Gauss-Markov model with fictitious unknowns, the statistical modeling includes the accuracy of the parameters describing the robot's pose and the accuracy of the extracted image features, and the number of parameters describing the robot's pose is at least equal to the number of degrees of freedom of the robot.

[0075]

number

[0076] Therefore, this model also optimizes the hand-eye pose parameters by simultaneously minimizing the backprojection error of image features in the captured camera images and the error in the parameters describing the robot pose, while taking into account the accuracy of the parameters describing the robot pose and the accuracy of the extracted image features.

[0077] After convergence, the variance components of the observations are estimated using the optimization results as described in Forstner and Wrobel (2016) or Niemeier ([Chapter 9.3], 2008). The variance component estimation leads to improved accuracy of the parameters describing the robot pose and the extracted image features.

[0078] Finally, the above optimization is run again with improved accuracy. Finally, statistical modeling, optimization, and variance component estimation are repeated until parameter accuracy converges. In practice, this typically occurs after 3-5 iterations.

[0079] Optionally,

[0080]

number

[0081] Determining approximate values ​​of unknown parameters In a preferred embodiment of the present invention, the initial values ​​of the unknowns, particularly the approximate values ​​of the hand-eye pose parameters, are set by the following procedure.

[0082] Hand-eye posture c and e b Approximations of θ can be obtained from any hand-eye calibration method known from the literature and suitable for these purposes. In a preferred embodiment of the present invention, a linear approach to hand-eye calibration, such as the method of Daniilidis (1999), is used for this purpose. The Daniilidis (1999) method, but also some other methods, requires that the camera pose be determined in advance for each robot pose based on calibration marks on a calibration object extracted in the camera image. The determination of approximations of the hand-eye pose parameters is then performed by the hand-eye calibration method using the camera pose and the robot pose.

[0083] The initial value of the interior orientation is the camera (s x and s y) and the lens (c) data sheet. x ,c y ) T is set to the center of the image and the distortion coefficients are set to 0.

[0084] In the case of self-calibration, the determination of the approximate values ​​of the parameters is carried out as follows: In a preferred embodiment of the invention, a SfM method is performed on the captured camera images. A possible suitable SfM implementation is for example COLMAP (Schonberger and Frahm, 2016; Schonberger et al., 2016). The SfM method:

[0085]

number

[0086] In alternative embodiments of the present invention, the unknowns may be initialized using any other suitable technique known in the literature for determining initial values ​​or additional knowledge from a particular application.

[0087] In the case of self-calibration, the essentially unknown scaling factor in the SfM method is

[0088]

number

[0089] Determining robot accuracy The results of optimizing the hand-eye pose parameters in both the Gauss-Helmert and Gauss-Markov models with fictitious unknowns can be used to determine the accuracy of the robot. The estimation of the variance components allows making meaningful statements about the accuracy of the robot, which usually requires an associated robot calibration. The accuracy of the robot is calculated by the matrix obtained after the variance components estimation:

[0090]

number

[0091] In a preferred embodiment of the invention, where a robot pose is described by three translational and three rotational parameters, the variations of the translational and rotational parameters of a robot pose are averaged separately over all robot poses for this purpose, so that the accuracy of the robot can be presented in the form of two values.

[0092] Determining calibrated robot poses In both the Gauss-Helmert model and the Gauss-Markov model with fictitious unknowns, it is possible to determine an improved (calibrated) robot pose based on the results of optimizing the parameters of the hand-eye pose. By introducing the robot pose as an observation, the vector

[0093]

number

[0094] In addition to the equilibrium image coordinates, the equations (i.e., the coordinate system) contain the equilibrium robot poses. These can be thought of as corrected or calibrated robot poses. They can therefore be used as the basis for simple and inexpensive robot calibration.

[0095] Effect of the invention Explicitly modeling device uncertainty is advantageous for hand-eye calibration. This improves accuracy, provides calibrated device poses, and provides information about device uncertainty. This is important, for example, in industrial robots used for tasks requiring high precision. Parameter estimation in a Gauss-Markov model with fictitious unknowns combined with the variance component estimation proposed in this invention provides a statistically sound representation of the problem. Various hand-eye calibration scenarios (e.g., calibration object-based calibration, self-calibration, calibration of different devices, such as articulated arm industrial robots, SCARA industrial robots, and ground survey robots, and unknown or known interior orientation) can be easily represented by adding or removing appropriate parameters from the parameter vector. Therefore, numerous applications can benefit from this invention.

[0096] References

[0097] [Table 1] TIFF0007811144000029.tif248170TIFF0007811144000030.tif100170

Claims

1. 1. A hand-eye calibration method for determining hand-eye pose parameters of a camera-guided device, comprising: (a) controlling a plurality of device attitudes using the device; (b) capturing camera images at each device orientation; (c) extracting image features within the captured camera image; (d) determining an approximation of the parameters of the hand-eye pose; (e) determining the parameters of the hand-eye pose assuming erroneous device poses and erroneously extracted image features to implicitly take into account erroneous camera poses, (e1) statistically modeling the accuracy of the parameters describing the device pose and the accuracy of the extracted image features by a weight matrix, a weight coefficient matrix, or a covariance matrix, wherein the number of parameters describing the device pose is equal to or greater than the number of degrees of freedom of the device; (e2) optimizing the parameters of the hand-eye pose by simultaneously minimizing backprojection errors of the image features in the captured camera images and errors in the parameters describing the device pose, taking into account the accuracy from step (e1); (e3) calculating improved accuracy of the parameters describing the device pose and the extracted image features based on the optimization results from step (e2) using variance component estimation; (e4) repeating steps (e1) to (e3) until the accuracy of the parameters describing the device pose and the accuracy of the extracted image features converge.

2. The method of claim 1 , wherein the device is a robot and the device pose represents a robot pose.

3. 3. The method of claim 1, wherein capturing a camera image at each device pose in step (b) includes capturing a camera image from a calibration object, and wherein extracting image features in the captured camera images in step (c) includes extracting calibration marks on the calibration object in the captured camera images.

4. 3. The method of claim 1 or 2, wherein capturing camera images at each device pose in step (b) includes capturing camera images of a scene suitable for extracting salient image points, and wherein extracting image features in the captured camera images in step (c) includes extracting salient image points in the captured camera images.

5. Determining the approximation of the parameter of the hand-eye pose in step (d) includes: (d1) determining a camera pose for each device pose using the calibration marks on the calibration object extracted in the camera image; (d2) determining the approximation of the parameters of the hand-eye pose using a hand-eye calibration technique that uses the camera pose and the device pose; The method of claim 3.

6. Determining the approximation of the parameter of the hand-eye pose in step (d) includes: (d1) determining a scaled camera pose for each device pose using the salient image points extracted in the camera images; (d2) determining a normalization factor using the scaled camera pose and the device pose; (d3) determining the camera pose by normalizing the scaled camera pose using the normalization factor; (d4) determining the approximations of the parameters of the hand-eye pose using a hand-eye calibration technique that uses the camera pose and the device pose; The method of claim 4.

7. (f) determining an improved (calibrated) device pose from the results of optimizing the parameters of the hand-eye pose; The method of claim 1.

8. (f) determining the accuracy of the device from the results of optimizing the parameters of the hand-eye pose; The method of claim 1.

9. Steps (d) and (e2) (d) determining approximations of the parameters of the hand-eye pose and the parameters of the interior orientation of the camera; (e2) optimizing the parameters of the hand-eye pose and the parameters of the interior orientation of the camera by simultaneously minimizing backprojection errors of the image features in the captured camera images and the errors in the parameters describing the device pose, taking into account the accuracy from step (e1); The method of claim 1.

Images