Hand-eye calibration of camera-guided devices
Patent Information
- Application Number
- JP2022078183
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-05-26
- Filing Date
- 2022-05-11
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2042-05-11
AI Technical Summary
Conventional hand-eye calibration methods for camera-guided robots assume error-free robot poses, leading to inaccuracies and requiring expensive, time-consuming calibration processes, while modern industrial robots have lower absolute accuracy than repeatability, exacerbating these issues.
A method that statistically models robot imprecision, allowing for high-accuracy hand-eye pose calculation and providing reliable information about robot accuracy, supporting both calibration object-based and self-calibration without known 3D points, and optionally calibrating internal camera parameters.
Improves hand-eye calibration accuracy, provides corrected robot poses for simple and inexpensive calibration, and offers reliable robot accuracy information, applicable to various camera-guided devices including industrial robots.
Smart Images

Figure 00000024_0000 
Figure 00000024_0001
Abstract
Description
[Technical Field]
[0001] Summary of the Invention This invention describes a general framework for hand-eye calibration of camera guiding devices, which requires determining the rigid 3D transformation between the device and the camera. An example of such a device is a camera-guided robot. In contrast to conventional methods, robot inaccuracies are explicitly modeled in a statistically sound manner. This is also relevant, for example, to industrial robots. While modern industrial robots have high precision, their absolute precision is usually much lower. This inaccuracy degrades the results of hand-eye calibration if it is not explicitly considered. The method proposed in this invention not only yields high precision in the calculated hand-eye pose but also provides reliable information regarding the robot's precision. Furthermore, it provides a corrected robot pose that can be used for simple and inexpensive robot calibration. The described framework is general in several respects and supports the use of both calibrators and self-calibration without requiring known 3D points. This optionally allows for simultaneous calibration of internal camera parameters. The framework is also general with respect to robot type, supporting, for example, both articulated arms and SCARA robots. In addition to applications including industrial robots, other applications using camera-guided robots or other camera guiding devices can also benefit from this invention. [Background technology]
[0002] Background technology Hand-eye calibration is essential for applications using camera-guided devices. In the case of robots, this determines the rigid 3D transformation (pose) between the robot and the camera (hand-eye pose). This allows measurements obtained in the camera coordinate system to be converted to the robot coordinate system. For example, in bin-picking applications using industrial robots, the object's pose in the camera coordinate system is determined using 3D object recognition (Hofhauser et al., 2009; Ulrich et al., 2012). In order for the robot to grasp the object, the object's pose must be converted to the robot's coordinate system.
[0003] In the case of camera-guided medical robots, service robots, and humanoid robots, it is also necessary to determine the orientation of the robot or robot hand relative to the camera. In the case of camera-guided autonomously navigating drones, hand-eye calibration determines the orientation between the camera and the drone's reference coordinate system. For example, the position of the drone's reference coordinate system relative to a higher coordinate system can be determined using Global Navigation Satellite Systems (GNSS) and Inertial Navigation Systems (INS). Similar considerations apply to camera-guided ground robots (e.g., surveillance robots or rescue robots). In operating rooms, the devices are increasingly used to assist surgeons. In endoscopic examinations, hand-eye calibration must be used to determine the position of the camera in the coordinate system of the endoscope or operating room. This is also true for augmented reality applications where hand-eye calibration must be used to determine the orientation between the camera and the headset's position and orientation sensors. Camera-guided cranes can be used to automate processes in the construction industry. Automated loading and unloading of container ships is also possible. Here again, the position of the camera in the crane's coordinate system must be determined via hand-eye calibration. The focus of this invention lies on industrial robots, and therefore the following description is based on this scenario as an example; however, obviously further applications in which camera guiding devices are used can benefit from this invention. In principle, this applies to all applications where hand-eye calibration is useful. Therefore, hereafter, the term "robot" refers to all devices in which hand-eye calibration is used.
[0004] Generally, there are two scenarios for camera-guided robots (Steger et al., [Chapter 3.12.2], 2018). In the first scenario, the camera is mounted on the robot's end effector and moves with the robot to different positions. In the second scenario, the camera is fixedly mounted on the outside of the robot and therefore does not move relative to the robot base. The orientation determined by hand-eye calibration is the camera's relative orientation to the robot tool in the first scenario, or the camera's relative orientation to the robot base in the second scenario. Hereafter, the description of the present invention refers to the first scenario using a moving camera. However, this can also be applied to the second scenario using a fixed camera in a similar manner.
[0005] Traditional methods for hand-eye calibration assume that the robot tool's orientation is precisely known relative to the robot base. Regarding robot accuracy, it is important to distinguish between repeatability and absolute accuracy (ISO 9283:1998). Repeatability represents the robot's ability to repeatedly move the tool to the same orientation. Absolute accuracy is the robot's ability to move the tool to a specific orientation in 3D space. Modern industrial robots typically offer very high repeatability, ranging from 0.02 to 0.15 mm (Shiakolas et al., 2002; Abderrahim et al., 2006; Placzek and Piszszek, 2018). High repeatability is sufficient for applications where the robot tool always assumes the exact same pre-trained orientation. However, for offline programmed robots, especially camera-guided robots, high absolute orientation accuracy is also important. Unfortunately, the absolute orientation accuracy of robots is often much lower than their repeatability. Absolute orientation accuracy is typically in the range of 0.1 mm to 10.0 mm, and orientation accuracy is often in the range of approximately 0.2 degrees to a few degrees (Abderrahim et al., 2006; Placzek and Piszczek, 2018).
[0006] Robot calibration can improve accuracy by up to one order of magnitude, and rarely, to the point of complete reproducibility (Shiakolas et al., 2002). Unfortunately, robot calibration often requires high-precision measuring equipment such as laser trackers (Abderrahim et al., 2006), and is therefore expensive and time-consuming.
[0007] Almost all conventional methods for hand-eye calibration assume an error-free robot posture. The method according to the present invention explicitly models robot inaccuracies in a statistically sound manner, resulting in higher accuracy of the calculated hand-eye posture. Furthermore, the method according to the present invention provides reliable information about robot inaccuracies, which other methods would require high-precision measuring instruments. The described method also provides an error-corrected (calibrated) robot posture, thus enabling simple and inexpensive robot calibration.
[0008] Most existing methods for hand-eye calibration require the acquisition of multiple images of the calibration object. Some more flexible and user-friendly solutions avoid the use of calibration objects. The method according to the present invention supports both calibration object-based and self-calibration variants without requiring known 3D points. Furthermore, the method according to the present invention optionally allows simultaneous calibration of internal camera parameters for both variants, thus providing a high degree of user-friendliness.
[0009] prior art Figure 1 shows the coordinate system related to hand-eye calibration of a robot (device) with a mobile camera.
[0010]
number
[0011] Equation (2) is often written as follows:
[0012]
number
[0013] Several linear methods essentially solve equation (2) or (3), e.g., Tsai and Lenz, 1989; Chen, 1991; Horaud and Dornaika, 1995; Dornaika and Horaud, 1998; Daniilidis, 1999; Andreff et al., 2001; Schmidt et al., 2003; Ulrich and Steger, 2016. These determine the rotational and translational distribution of the hand-eye pose sequentially or simultaneously. The latter has the advantage that rotational errors do not propagate, thereby amplifying translational errors. Methods for solving equation (3) require selecting an appropriate pair of robot poses to calculate A and B. Criteria for selecting an appropriate pair of poses have been proposed in Tsai and Lenz, 1989; Schmidt et al., 2003; and Schmidt and Niemann, 2008. However, the observed information (A i and B i It remains difficult to guarantee that these methods are optimally utilized. Furthermore, these methods assume an error-free robot posture (Tsai and Lenz, 1989).
[0014] Linear methods typically minimize algebraic errors, thus limiting their accuracy. Therefore, they are often used to initialize subsequent nonlinear optimizations to achieve higher accuracy. Most nonlinear methods that minimize either algebraic or geometric errors (Horaud and Dornaika, 1995; Dornaika and Horaud, 1998; Daniilidis, 1999; Schmidt et al., 2005; Ulrich and Steger, 2016; Steger [Chapter 3.13.5], 2018) also often face the additional problem of how to weight the rotational and translational error components relative to each other, assuming an error-free robot posture. Strobl and Hirszinger (2006) minimize the weighted sum of the rotational and translational error distributions, where the weights of the error components are statistically derived. This minimizes the error in the robot posture. b Ht This can be taken into consideration, but the camera posture is error-free. c H W This is considered as the input. Nguyen and Pham (2018) sequentially solve the rotation and translation parts of equation (3) and propagate the errors of A and B to X.
[0015] Another class of methods minimizes the backprojection error of 3D world points on the calibration object, similar to camera calibration methods such as Zhang (2000) (e.g., Tabb and Yousef, 2017). A key advantage of this class of methods is that it eliminates the need for explicit estimation of camera positions within each image during preprocessing, does not require pre-selection of pose pairs, and minimizes observed erroneous measurements, i.e., significant geometric errors in the space of image points. Another advantage is that minimizing backprojection errors also enables simultaneous estimation of inner camera parameters. However, these aforementioned methods also assume error-free robot poses. Koide and Menegatti (2019) presented a pose graph optimization framework for hand-eye calibration that minimizes errors in robot poses in addition to backprojection errors of points on the calibration object. The advantages of considering the uncertainty of robot poses have been experimentally demonstrated. Unfortunately, details regarding the probabilistic model, optimization, and weighting of different error components relative to each other are not presented in the publication.
[0016] The self-calibration method is used to perform hand-eye calibration without a calibration object. In this method, unknown 3D points are tracked within an image sequence obtained from a predetermined robot motion. The internal camera parameters and the 3D points are estimated simultaneously with the hand-eye pose. In Andreff et al. (2001), a Structure from Motion (SfM) approach is used, and the unknown scaling factor of the SfM result is integrated into the equation. This idea was taken up by Schmidt et al. (2005), and the post-processing step for enforcing the orthogonality of the rotation matrix is avoided by introducing an unknown scaling factor into the equations of Horaud and Dornaika (1995) and Daniilidis (1999). Since the described self-calibration techniques minimize algebraic errors, the accuracy achievable with these methods is limited. Summary of the Invention Means for Solving the Problems
[0017] The method according to the present invention combines the aforementioned advantages of minimizing the reprojection error, the advantage of probabilistically modeling the inaccuracies of all measured observations (image points and robot poses), the flexibility to perform either calibration object-based calibration or self-calibration, and the possibility of using known internal camera parameters or estimating them simultaneously.
[0018] According to a first aspect, the present invention provides a hand-eye calibration method for determining the parameters of the hand-eye pose of a camera-guided device. In this regard, the method comprises: (a) controlling a plurality of device poses using the device; (b) capturing a camera image at each device pose; (c) extracting image features within the captured camera image; (d) determining an approximation of the parameters of the hand-eye pose; (e) determining the parameters of the hand-eye pose assuming incorrect device poses and incorrectly extracted image features in order to implicitly account for incorrect camera poses. (e1) A step of statistically modeling the accuracy of parameters describing the apparatus orientation and the accuracy of extracted image features, wherein the number of parameters describing the apparatus orientation is at least equal to the number of degrees of freedom of the apparatus. (e2) Taking into account the accuracy from step (e1), the step of optimizing the hand-eye attitude parameters by simultaneously minimizing the back projection error of the image features in the acquired camera image and the error of the parameters describing the apparatus attitude, (e3) Using variance component estimation, the steps include calculating the improved accuracy of parameters describing the apparatus posture and extracted image features based on the optimization results from step (e2), (e4) The step of repeating steps (e1) to (e3) until the accuracy of the parameters describing the apparatus orientation and the accuracy of the extracted image features converges.
[0019] Preferably, the device is a robot, and the device posture represents the robot's posture. Alternatively, the device may be an automated industrial crane, and the device posture represents the crane's posture.
[0020] In a first preferred embodiment, acquiring a camera image in each device orientation in step (b) includes acquiring a camera image from a calibration object. More preferably, extracting image features in the acquired camera image in step (c) includes extracting calibration marks on the calibration object in the acquired camera image. Determining approximate values of hand-eye orientation parameters in step (d) preferably includes (d1) determining the camera orientation for each device orientation using the calibration marks on the calibration object extracted in the camera image, and (d2) determining approximate values of hand-eye orientation parameters using a hand-eye calibration method that uses the camera orientation and the device orientation.
[0021] In an alternative preferred embodiment, acquiring camera images in each device posture in step (b) includes acquiring camera images of scenes suitable for extracting prominent image points. More preferably, extracting image features in the acquired camera images in step (c) includes extracting prominent image points in the acquired camera images. Preferably, determining approximate values of hand-eye posture parameters in step (d) includes (d1) determining a scaled camera posture for each device posture using prominent image points extracted in the camera images; (d2) determining a normalization coefficient using the scaled camera posture and device posture; (d3) determining a camera posture by normalizing the scaled camera posture using the normalization coefficient; and (d4) determining approximate values of hand-eye posture parameters using a hand-eye calibration technique that uses the camera posture and device posture.
[0022] Preferably, the method further includes determining an improved (calibrated) apparatus posture from the results of optimizing the hand-eye posture parameters.
[0023] A further step of determining the accuracy of the device based on the results of optimizing the hand-eye posture parameters is even more preferable.
[0024] In a further embodiment, the present invention provides a method in which steps (d) and (e2) are replaced by (d) determining approximate values for the hand-eye pose parameters and the camera's internal orientation parameters, and (e2) optimizing the hand-eye pose parameters and the camera's internal orientation parameters by simultaneously minimizing the backprojection error of image features in the acquired camera image and the error of the parameters describing the apparatus pose, taking into account the accuracy from step (e1). In other words, a second embodiment provides the following method:
[0025] (a) Controlling the orientation of multiple devices using the device, (b) Capture camera images in each device orientation, (c) Extracting image features from the captured camera image, (d) Determining approximate values of the hand-eye posture parameters and the internal camera orientation parameters, (e) Determining hand-eye orientation parameters by assuming an incorrect device orientation and incorrectly extracted image features in order to implicitly account for an incorrect camera orientation, (e1) A step of statistically modeling the accuracy of parameters describing the apparatus orientation and the accuracy of extracted image features, wherein the number of parameters describing the apparatus orientation is at least equal to the number of degrees of freedom of the apparatus. (e2) Taking into account the accuracy from step (e1), the step of optimizing the hand-eye orientation parameters and the camera's internal orientation parameters by simultaneously minimizing the back projection error of the image features in the acquired camera image and the error of the parameters describing the apparatus orientation, (e3) Using variance component estimation, the steps include calculating the improved accuracy of parameters describing the apparatus posture and extracted image features based on the optimization results from step (e2), (e4) The step of repeating steps (e1) to (e3) until the accuracy of the parameters describing the apparatus orientation and the accuracy of the extracted image features converge. [Brief explanation of the drawing]
[0026] [Figure 1] This shows the coordinate system related to hand-eye calibration of a robot (device) equipped with a mobile camera. [Figure 2] This shows the coordinate system related to hand-eye calibration of a robot (device) in the case of a fixed camera. [Modes for carrying out the invention]
[0027] Modes for carrying out the invention First, the camera model and calibration model that form the basis of the present invention, i.e., the relationship between 3D world points and their projection onto the camera, will be described. To facilitate the explanation of the present invention, it will be assumed that the camera guiding device is a camera guiding robot. The explanation can be readily applied to other camera guiding devices as described above by those skilled in the art. Furthermore, it will be understood that the camera is mounted on the end effector of the robot. Thus, the explanation of the present invention will mainly concern the case of a moving camera. It is known from the literature that the case of a fixed camera is equivalent to the case of a moving camera. Therefore, the case of a fixed camera will be described below only where the equivalence is not clear. Next, three alternative optimization methods for hand-eye calibration will be described. Finally, a method for providing the necessary initial values for unknowns in the optimization procedure will be described.
[0028] Camera model In a preferred embodiment of the present invention, the camera is described by a perspective projection camera as described by Steger et al. ([Chapter 3.9.1], 2018), and when homogeneous coordinates are used,
[0029]
number
[0030] Next,
[0031]
number
[0032] lastly,
[0033]
number
[0034] Calibration Model In hand-eye calibration, the device is moved to different device orientations. In the example of the mobile camera robot used in the description of the present invention, the robot's tool is nr The robot is moved to individual different poses, and camera images are captured in each of these robot poses. In the case of calibration object-based calibration, the calibration object is placed in a fixed position within the robot's workspace (see Figure 1). In the case of self-calibration, instead, camera images are captured from a scene suitable for extracting prominent image points. This could be, for example, a well-structured object or an arbitrary but structured background scene.
[0035]
number
[0036] lastly,
[0037]
number
[0038] In the case of a fixed camera, a calibration object (for calibration object-based calibration) or a well-structured object (for self-calibration) is attached to the robot's end effector and thus moves with the robot. The fixed camera then captures a camera image of the co-moving object in each approaching robot posture. In the case of self-calibration, it may be useful to ensure that the background is as homogeneous and unstructured as possible so that the object's position in the camera image can be detected as robustly and automatically as possible.
[0039]
number
[0040] Parameter estimation in Gauss-Markov models In the following, we distinguish between functional models and probabilistic models (Forstner and Wrobel, 2016). Functional models describe the relationship between observed values and unknown parameters. In probabilistic models, observed values and unknowns are treated as random variables with uncertainty, and this uncertainty is described by (co)variance.
[0041] Assuming an error-free robot posture initially, the hand-eye calibration problem can be formulated using the so-called Gauss-Markov model (Forstner and Wrobel [Chapter 4.4], 2016).
[0042]
number
[0043] That is, it includes extracted image features within the captured camera image. In the case of calibration object-based calibration, in a preferred embodiment of the present invention, the extracted image features represent projections of the centers of circular marks on the calibrator (Steger and Ulrich, 2018). In an alternative embodiment of the present invention, they represent projections of the intersections of a grid pattern on the calibrator (OpenCV, 2021). In the case of self-calibration, the extracted image features represent prominent image points that are computed using appropriate image processing operators within the image and mapped to each other across different images. Examples of such image processing operators are the Forstner point extractor (Forstner, 1994), the Harris point extractor (Harris and Stephens, 1988), and the SIFT point extractor (Lowe, 2004).
[0044] If the 3D point is not visible in a particular image,
[0045]
number
[0046] • For calibration object-based calibration,
[0047]
number
[0048] • In the case of self-calibration,
[0049]
number
[0050]
number
[0051] After the situation subsides,
[0052]
number
[0053] Below, we describe two alternative procedural models for determining hand-eye pose parameters under the assumption of incorrect extracted image features due to the implicit consideration of incorrect robot and camera poses: parameter estimation in a Gauss-Hermat model and parameter estimation in a Gauss-Markov model with hypothetical unknowns.
[0054] Parameter estimation in the Gauss-Hermat model To explain incorrect robot postures, they must be introduced as observations in addition to the image coordinates of the extracted image features. Therefore, the observations can no longer be expressed as a function of unknowns. For this reason, parameter estimation becomes difficult in the Gauss-Markov model. Thus, in one embodiment of the present invention, parameter estimation is performed using the Gauss-Hermert model (Forstner and Wrobel [Chapter 4.8], 2016).
[0055]
number
[0056] However, the number of current observed values is n when the robot pose is represented by three translational parameters and three rotational parameters l =2n i +6n r In the case of an alternative representation of the robot pose, n l changes accordingly. For example, since a dual quaternion contains eight parameters, when using a dual quaternion, n[[ID=I2]] l =2n i +8n r The vector x of unknowns is the same as that of the Gauss-Markov model.
[0057] Compared with the Gauss-Markov model, the probability model must further consider the uncertainty of the robot pose. Tests in an actual system have shown that the error of the robot pose is mean-free and Gaussian distributed (Strobl and Hirzinger, 2006).
[0058] Therefore, the following statistical modeling includes the accuracy of the parameters describing the robot pose and the accuracy of the extracted image features, and the number of parameters describing the robot pose is at least equal to the number of degrees of freedom of the robot. <000C296>
[0059] Even assuming uncorrelated observed values,
[0060]
Number
[0061]
Number
[0062] Therefore, this model optimizes the hand-eye pose parameters by simultaneously minimizing the backprojection error of image features in the captured camera image and the error of the parameters describing the robot pose, taking into account the accuracy of the parameters describing the robot pose and the accuracy of the extracted image features.
[0063] After optimization convergence, the variance components for each observation group are estimated using the optimization results. When Euler angles are used, the variance component estimation is:
[0064]
number
[0065] The calculation can be found in Forstner and Wrobel (2016). Variance component estimation results in improved accuracy for the parameters describing the robot's pose and the extracted image features.
[0066] Finally, the optimization described above is performed again with improved accuracy. Finally, statistical modeling, optimization, and variance component estimation are repeated until the parameter accuracy converges. In practice, this typically takes 3 to 5 iterations.
[0067] Optionally, the covariance matrix of the observed values is:
[0068]
number
[0069] Parameter estimation in a Gauss-Markov model using hypothetical unknowns Due to the computationally intensive matrix operations in the Gauss-Helmert model, in a preferred embodiment of the present invention, parameter estimation is performed using a more efficient variation of the Gauss-Markov model. This is equivalent to parameter estimation in the Gauss-Helmert model (Koch, 1999; Koch, 2007), where the robot pose is introduced as a so-called hypothetical unknown.
[0070] The fundamental idea here is to simultaneously introduce the observed value and the uncertain robot pose as an unknown variable. The first part of the function model is still l=f(x). However, unlike the estimation in the Gauss-Markov model described above, here,
[0071]
number
[0072] Since l is the same as the Gauss-Hermat model, the same probability model can be applied to the statistical modeling of accuracy, and when the robot posture is represented by three translational parameters and three rotational parameters,
[0073]
number
[0074] It can be initialized. Therefore, even in a Gauss-Markov model with hypothetical unknowns, the statistical modeling includes the precision of the parameters describing the robot's pose and the precision of the extracted image features, and the number of parameters describing the robot's pose is at least equal to the number of degrees of freedom of the robot.
[0075]
number
[0076] Therefore, this model also optimizes the hand-eye pose parameters by simultaneously minimizing the backprojection error of image features in the captured camera image and the error of the parameters describing the robot pose, while taking into account the accuracy of the parameters describing the robot pose and the accuracy of the extracted image features.
[0077] After convergence, the variance component of the observation group is estimated using optimization results as described in Forstner and Wrobel (2016) or Niemeier ([Chapter 9.3], 2008). Variance component estimation results in improved accuracy for parameters describing the robot pose and for extracted image features.
[0078] Finally, the optimization described above is performed again with improved accuracy. Finally, statistical modeling, optimization, and variance component estimation are repeated until the parameter accuracy converges. In practice, this typically takes 3 to 5 iterations.
[0079] Optional,
[0080]
number
[0081] Determining approximate values for unknown parameters In a preferred embodiment of the present invention, the initial values of unknowns, particularly the approximate values of the hand-eye posture parameters, are set by the following procedure.
[0082] • Hand and eye position e c and e b Approximations of these parameters are known from the literature and can be obtained from any hand-eye calibration method suitable for these purposes. In a preferred embodiment of the present invention, a linear method to hand-eye calibration, such as the method of Daniilidis (1999), is used for this purpose. Although it is the method of Daniilidis (1999), several other methods also require pre-determining the camera pose for each robot pose based on calibration marks on calibration objects extracted in the camera image. The determination of approximate values for the hand-eye pose parameters is then performed by a hand-eye calibration method using the camera pose and robot pose.
[0083] • The initial value of the internal orientation is the camera (s x and s y) and obtained from the datasheet of lens (c). Image principal point (c x ,c y ) T The center of the image is set, and the distortion coefficient is set to 0.
[0084] • In the case of self-calibration, the determination of approximate parameter values is performed as follows. In a preferred embodiment of the present invention, the SfM method is performed on the acquired camera image. A suitable possible SfM implementation is, for example, COLMAP (Schonberger and Frahm, 2016; Schonberger et al., 2016). The SfM method is,
[0085]
number
[0086] In alternative embodiments of the present invention, the unknowns can be initialized using any other suitable technique known in the literature for determining initial values or additional knowledge from a particular application.
[0087] In the case of self-calibration, the scaling factor in the SfM method is essentially unknown.
[0088]
number
[0089] Determining the accuracy of a robot In both Gauss-Hermat and Gauss-Markov models with hypothetical unknowns, the robot's accuracy can be determined using the results of optimizing the hand-eye pose parameters. Estimating the variance components allows for a meaningful description of the robot's accuracy, which typically requires associated robot calibration. The robot's accuracy is expressed in the matrix obtained after variance component estimation.
[0090]
number
[0091] This can be read directly from. In a preferred embodiment of the present invention, where the robot posture is described by three translational parameters and three rotational parameters, the variations in the translational and rotational parameters of the robot posture are averaged separately across all robot postures for this purpose, and as a result, the accuracy of the robot may be presented in the form of two values.
[0092] Determination of calibrated robot posture In both Gauss-Hermat and Gauss-Markov models with hypothetical unknowns, it is possible to determine an improved (calibrated) robot pose based on the results of optimizing the hand-eye pose parameters. By introducing the robot pose as an observed value, a vector can be generated.
[0093]
number
[0094] This includes the equilibrium robot pose in addition to the equilibrium image coordinates. These can be considered as modified or calibrated robot poses. Therefore, they can be used as the basis for simple and inexpensive robot calibration.
[0095] Effects of the invention Explicitly modeling the uncertainty of the device is advantageous for hand-eye calibration. This improves accuracy, provides a calibrated device orientation, and offers information about the device's uncertainty. This is important, for example, in industrial robots used in tasks requiring high precision. Parameter estimation in a Gauss-Markov model with hypothetical unknowns, combined with the estimation of the variance component proposed in this invention, provides a statistically sound representation of the problem. Various hand-eye calibration scenarios (e.g., calibration object-based calibration, self-calibration, different devices, e.g., calibration of articulated arm industrial robots, SCARA industrial robots, ground exploration robots, unknown or known internal localization) can be easily represented by adding or removing appropriate parameters to the parameter vector. Thus, numerous applications can benefit from this invention.
[0096] References
[0097] [Table 1] TIFF2022183032000029.tif248170TIFF2022183032000030.tif100170
Claims
1. 1. A hand-eye calibration method for determining parameters of a hand-eye pose of a camera guide device, comprising: (a) controlling a plurality of device orientations using the device; (b) capturing camera images at each device orientation; (c) extracting image features within the captured camera image; (d) determining an approximation of the parameters of the hand-eye pose; (e) determining the parameters of the hand-eye pose assuming erroneous device poses and erroneously extracted image features to implicitly take into account erroneous camera poses, (e1) statistically modeling the accuracy of the parameters describing the device pose and the accuracy of the extracted image features, the number of parameters describing the device pose being at least equal to the number of degrees of freedom of the device; (e2) optimizing the parameters of the hand-eye pose by simultaneously minimizing a backprojection error of the image features in the captured camera images and the error of the parameters describing the device pose, taking into account the accuracy from step (e1); (e3) calculating an improved accuracy of the parameters describing the device pose and the extracted image features based on the optimization results from step (e2) using variance component estimation; (e4) repeating steps (e1) to (e3) until the accuracy of the parameters describing the device pose and the accuracy of the extracted image features converge.
2. The method of claim 1 , wherein the device is a robot, and the device pose represents a robot pose.
3. 3. The method of claim 1 or 2, wherein capturing a camera image at each device pose in step (b) includes capturing a camera image from a calibration object, and wherein extracting image features in the captured camera images in step (c) includes extracting calibration marks on the calibration object in the captured camera images.
4. 3. The method of claim 1 or 2, wherein capturing a camera image at each device pose in step (b) comprises capturing a camera image of a scene suitable for extracting salient image points, and wherein extracting image features in the captured camera images in step (c) comprises extracting salient image points in the captured camera images.
5. Determining the approximation of the hand-eye pose parameters in step (d) comprises: (d1) determining a camera pose for each device pose using the calibration marks on the calibration object extracted in the camera image; (d2) determining the approximations of the hand-eye pose parameters using a hand-eye calibration technique that uses the camera pose and the device pose; The method according to claim 3.
6. Determining the approximation of the hand-eye pose parameters in step (d) comprises: (d1) determining a scaled camera pose for each device pose using the salient image points extracted in the camera images; (d2) determining a normalization factor using the scaled camera pose and the device pose; (d3) determining the camera pose by normalizing the scaled camera pose using the normalization factor; (d4) determining the approximations of the parameters of the hand-eye pose using a hand-eye calibration technique that uses the camera pose and the device pose; The method according to claim 4.
7. (f) determining an improved (calibrated) device pose from the results of optimizing the parameters of the hand-eye pose; The method of claim 1.
8. (f) determining the accuracy of the device from the results of optimizing the parameters of the hand-eye pose. The method of claim 1.
9. Steps (d) and (e2) (d) determining approximations of the parameters of the hand-eye pose and the parameters of the interior orientation of the camera; (e2) optimizing the parameters of the hand-eye pose and the parameters of the interior orientation of the camera by simultaneously minimizing the backprojection error of the image features in the captured camera images and the error of the parameters describing the device pose, taking into account the accuracy from step (e1); The method of claim 1.
