Hand-eye calibration method and hand-eye calibration device for robot arm

Abstract

A hand-eye calibration method and a hand-eye calibration device are provided. The method includes obtaining a first mapping relationship between a base of a robot arm and an end of the robot arm and a second mapping relationship between a camera and a target object; updating a third mapping relationship between the end of the robot arm and a tool mounted on the end and a fourth mapping relationship between the camera and the base in each dimension in order based on a scale to minimize an error between a position of the target object in an image extracted by the camera and a position of the tool; and in response to the error converging and the scale being less than or equal to a scale threshold, outputting the third mapping relationship and the fourth mapping relationship that are calibrated by the scale. Thus, the scale can be continuously reduced to gradually minimize the error of the rotation matrix, so that the correction result of the rotation matrix is more accurate.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a hand-eye calibration method and a hand-eye calibration device for a robot arm, and more particularly to a hand-eye calibration method and a hand-eye calibration device for an eye-to-hand robot arm. BACKGROUND

[0002] An eye-to-hand robot arm system can acquire images of a monitored area by a camera installed at a fixed position, and then control the robot arm to process a target object in the monitored area according to the images. In order to make the operation of the robot arm more accurate, a user can perform a hand-eye calibration on the robot arm system to make the coordinate transformation relationship among the robot arm, the camera and the target object more accurate. However, the hand-eye calibration is a calibration method that requires human intervention. If the user does not have a professional background in operating the robot arm or does not spend a lot of time in performing the hand-eye calibration, the result of the hand-eye calibration is often unsatisfactory. Accordingly, how to propose an accurate hand-eye calibration method is one of the goals of the people in the field. SUMMARY

[0003] The present disclosure aims to provide a hand-eye calibration method and a hand-eye calibration device, which can automatically calibrate the coordinate transformation relationship among an eye-to-hand robot arm, a camera and a target object.

[0004] A hand-eye calibration method of the present disclosure is suitable for an eye-to-hand robot arm, and includes: acquiring a first mapping relationship between a base of the robot arm and an end of the robot arm, and a second mapping relationship between a camera and a target object; updating a third mapping relationship between the end of the robot arm and a tool installed at the end, and a fourth mapping relationship between the camera and the base in each dimension in order based on a scale to minimize an error between a position of the target object in an image extracted by the camera and a position of the tool; and in response to the error converging and the scale being less than or equal to a scale threshold, outputting the third mapping relationship and the fourth mapping relationship that are corrected by the scale.

[0005] According to one embodiment of the present disclosure, it further includes: in response to the error converging and the scale being greater than the scale threshold, reducing the scale to update the third mapping relationship and the fourth mapping relationship.

[0006] According to one of the embodiments of the present disclosure, the step of updating the third mapping relationship and the fourth mapping relationship in each dimension based on the scale to minimize the error comprises: obtaining a coordinate value corresponding to the third mapping relationship, and generating a plurality of offset coordinate values according to the scale and the coordinate value; selecting a selected offset coordinate value corresponding to the minimum error from the plurality of offset coordinate values according to the first mapping relationship and the second mapping relationship; and updating the third mapping relationship according to the selected offset coordinate value.

[0007] According to one of the embodiments of the present disclosure, in response to the error converging and the scale being less than or equal to the scale threshold, the step of outputting the third mapping relationship and the fourth mapping relationship corrected by the scale comprises: increasing the scale, and determining whether updating the third mapping relationship and the fourth mapping relationship according to the increased scale increases the error; if yes, outputting the third mapping relationship and the fourth mapping relationship corrected by the scale; and if no, updating the third mapping relationship and the fourth mapping relationship according to the increased scale to minimize the error.

[0008] According to one of the embodiments of the present disclosure, further comprising: updating the third mapping relationship based on the scale to generate a first rotation matrix, and updating the fourth mapping relationship based on the scale to generate a second rotation matrix; calculating a first error between the third mapping relationship and the fourth mapping relationship and between the first rotation matrix and the second rotation matrix; updating the first rotation matrix based on the scale to generate a third rotation matrix, and updating the second rotation matrix based on the scale to generate a fourth rotation matrix; calculating a second error between the first rotation matrix and the second rotation matrix and between the third rotation matrix and the fourth rotation matrix; and determining that the error converges in response to an absolute difference between the first error and the second error being less than or equal to a difference threshold.

[0009] According to one of the embodiments of the present disclosure, the each dimension comprises an X-axis, a Y-axis, a Z-axis, a twist angle along the X-axis, a twist angle along the Y-axis, and a twist angle along the Z-axis.

[0010] According to one of the embodiments of the present disclosure, the first mapping relationship comprises a coordinate conversion relationship between the base and the end, the second mapping relationship comprises a coordinate conversion relationship between the camera and the target object, the third mapping relationship comprises a coordinate conversion relationship between the end and the tool, and the fourth mapping relationship comprises a coordinate conversion relationship between the camera and the base.

[0011] A hand-eye calibration device of one embodiment of the present disclosure is applicable to an eye-to-hand based robot arm, and includes a transceiver and a processor. The processor is coupled to the transceiver, and is configured to perform: obtaining, by the transceiver, a first mapping relationship between a base of the robot arm and an end of the robot arm, and a second mapping relationship between a camera and a target object; updating, based on a scale, a third mapping relationship between the end of the robot arm and a tool mounted on the end, and a fourth mapping relationship between the camera and the base in each dimension in sequence to minimize an error between a position of the target object in an image extracted by the camera and a position of the tool; and in response to the error converging and the scale being less than or equal to a scale threshold, outputting, by the transceiver, the third mapping relationship and the fourth mapping relationship that are corrected by the scale.

[0012] According to one embodiment of the present disclosure, the processor is further configured to perform: in response to the error converging and the scale being greater than the scale threshold, reducing the scale to update the third mapping relationship and the fourth mapping relationship.

[0013] According to one embodiment of the present disclosure, the processor is further configured to perform: obtaining a coordinate value corresponding to the third mapping relationship, and generating a plurality of offset coordinate values according to the scale and the coordinate value; selecting, according to the first mapping relationship and the second mapping relationship, a selected offset coordinate value corresponding to a minimum error from the plurality of offset coordinate values; and updating the third mapping relationship according to the selected offset coordinate value.

[0014] According to one embodiment of the present disclosure, the processor is further configured to perform: increasing the scale, and determining whether updating the third mapping relationship and the fourth mapping relationship according to the increased scale increases the error; if yes, outputting the third mapping relationship and the fourth mapping relationship that are corrected by the scale; and if no, updating the third mapping relationship and the fourth mapping relationship according to the increased scale to minimize the error.

[0015] According to one of the embodiments of the present disclosure, the processor is further configured to perform: updating the third mapping relationship based on the scale to generate a first rotation matrix, and updating the fourth mapping relationship based on the scale to generate a second rotation matrix; calculating a first error between both the third mapping relationship and the fourth mapping relationship and both the first rotation matrix and the second rotation matrix; updating the first rotation matrix based on the scale to generate a third rotation matrix, and updating the second rotation matrix based on the scale to generate a fourth rotation matrix; calculating a second error between both the first rotation matrix and the second rotation matrix and both the third rotation matrix and the fourth rotation matrix; and determining that the error converges in response to an absolute difference between the first error and the second error being less than or equal to a difference threshold.

[0016] According to one of the embodiments of the present disclosure, the dimensions include an X axis, a Y axis, a Z axis, a twist angle along the X axis, a twist angle along the Y axis, and a twist angle along the Z axis.

[0017] According to one of the embodiments of the present disclosure, the first mapping relationship includes a coordinate conversion relationship between the base and the end, the second mapping relationship includes a coordinate conversion relationship between the camera and the target object, the third mapping relationship includes a coordinate conversion relationship between the end and the tool, and the fourth mapping relationship includes a coordinate conversion relationship between the camera and the base.

[0018] Based on the above, for the coordinate conversion relationship of each element in the robot arm system, the hand-eye calibration device of the present disclosure can use a specific scale to correct the rotation matrix that may exist in the error of the coordinate conversion relationship. In the process of correction, the hand-eye calibration device can continuously reduce the scale to gradually minimize the error of the rotation matrix, so that the correction result of the rotation matrix is more accurate. In addition, the correction process sequentially corrects each dimension, that is, using the single-axis convergence method, the operation complexity can be greatly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 According to an embodiment of the present disclosure, a schematic diagram of a hand-eye calibration device is shown.

[0020] Figure 2 According to an embodiment of the present disclosure, a schematic diagram of a robot arm system based on eye-to-hand is shown.

[0021] Figure 3 According to an embodiment of the present disclosure, a flowchart of a hand-eye calibration method is shown.

[0022] Figure 4 According to an embodiment of the present disclosure, a flowchart of performing hand-eye calibration is shown in more detail.

[0023] Figure 5 A flowchart showing updating a rotation matrix is shown according to an embodiment of the disclosure.

[0024] Reference signs are as follows:

[0025] 100: hand-eye calibration device

[0026] 110: processor

[0027] 120: storage medium

[0028] 130: transceiver

[0029] 20: robot arm system

[0030] 200: robot arm

[0031] 210: base

[0032] 220: end

[0033] 230: tool

[0034] 300: camera

[0035] 400: target object

[0036] S301, S302, S303, S401, S402, S403, S404, S405, S406, S407, S408, S409, S410, S411, S501, S502, S503, S504, S505, S506: steps DETAILED DESCRIPTION

[0037] In order that the disclosure can be more readily understood, the following exemplary embodiments are given, as examples of how the disclosure can be carried out. Additionally, where possible, the same reference numerals are used in the drawings and the embodiments to represent the same or similar components.

[0038] Figure 1 A schematic diagram of a hand-eye calibration device 100 is shown according to an embodiment of the disclosure, wherein the hand-eye calibration device 100 is adapted for calibrating an eye-to-hand based robot arm. The hand-eye calibration device 100 can comprise a processor 110, a storage medium 120, and a transceiver 130.

[0039] The processor 110 is, for example, a central processing unit (CPU), or other programmable general purpose or special purpose micro control unit (MCU), microprocessor, digital signal processor (DSP), programmable controller, application specific integrated circuit (ASIC), graphics processing unit (GPU), image signal processor (ISP), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field programmable gate array (FPGA), or other similar element or combination thereof. The processor 110 can be coupled to the storage medium 120 and the transceiver 130, and access and execute a plurality of modules and various application programs stored in the storage medium 120.

[0040] The storage medium 120 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD), or similar element or combination thereof, and is used to store a plurality of modules or various application programs executable by the processor 110.

[0041] The transceiver 130 transmits and receives signals in a wireless or wired manner. The transceiver 130 can also perform operations such as low noise amplification, impedance matching, mixing, up or down frequency conversion, filtering, amplification, and the like.

[0042] Figure 2An embodiment according to the present disclosure shows a schematic diagram of an eye-to-hand based robot arm system 20. The robot arm system 20 can include a robot arm 200, a camera 300, and a target object 400 to be processed by the robot arm 200. The camera 300 can be installed at a fixed position and can take pictures of a fixed monitored area. The target object 400 can be placed in the monitored area.

[0043] The robot arm 200 can include a base 210, an end 220, and a tool 230. The base 210 is used to set the robot arm 200 at a fixed position. The robot arm 200 can include a plurality of joint nodes, wherein the end 220 is a node of the plurality of joint nodes located at the end of the robot arm 200. The end 220 can have a flange to connect the tool 230. The tool 230 can have different types for different tasks. For example, when the task of the robot arm 200 is to move the target object 400 composed of metal, the tool 230 can be an iron plate with magnetic force. When the task of the robot arm 200 is to process the target object 400, the tool 230 can be a gripper.

[0044] The robot arm 200 can be communicatively connected with the camera 300. After the camera 300 takes pictures of the monitored area, the controller of the robot arm 200 can obtain the pictures from the camera 300 and control the robot arm 200 to act on the target object 400 according to the pictures.

[0045] In a three-dimensional space, the position and pose of an object can be represented by six parameters [x y z rx ry rz], wherein x represents the coordinate value of the object on the X-axis of the Cartesian coordinate system, y represents the coordinate value of the object on the Y-axis of the Cartesian coordinate system, z represents the coordinate value of the object on the Z-axis of the Cartesian coordinate system, rx represents the angle of rotation of the object around the X-axis (or the Eulerian angle corresponding to the X-axis), ry represents the angle of rotation of the object around the Y-axis (or the Eulerian angle corresponding to the Y-axis), and rz represents the angle of rotation of the object around the Y-axis (or the Eulerian angle corresponding to the Z-axis). The parameters [x y z] can be used to represent the position of the object, and the parameters [rx ry rz] can be used to represent the pose of the object. The mapping relationship between the coordinate system of the origin in the three-dimensional space and the coordinate system of the object can be represented by a rotation matrix as shown in equation (1).

[0046]

[0047] In three-dimensional space, when the tool 230 of the robot arm 200 is attached to the target object 400, the ideal coordinate transformation relationship between the elements in the robot arm system 20 is shown in equation (2), where the rotation matrix A corresponds to the mapping relationship between the coordinate system of the base 210 and the coordinate system of the end 220, the rotation matrix B corresponds to the mapping relationship between the coordinate system of the end 220 and the coordinate system of the tool 230, the rotation matrix C corresponds to the mapping relationship between the coordinate system of the base 210 and the coordinate system of the camera 300, and the rotation matrix D corresponds to the mapping relationship between the coordinate system of the camera 300 and the coordinate system of the target object 400. In this embodiment, the mapping relationship includes the coordinate transformation relationship between two coordinate systems.

[0048] A·B=C·D...(2)

[0049]

[0050]

[0051] In the conventional calibration process, the rotation matrix B of the tool 230 is defined by the user in advance by using the tool angle R b and manually moving the robot arm 200 to make the tool 230 touch a fixed position element in space in different postures (as shown in equation (3), A1, A2,..., An rotation matrices can be obtained from the posture position of the end 220 of the robot arm 200 in space N ; the N rotation matrices have the same coordinate values x, y and z, and can be used to calculate the coordinate values x b , y b and z b) is obtained (N can be set as 4). The mapping relationship (i.e. rotation matrix D) between the target object 400 and the camera 300 is obtained by placing a template with a known size to obtain the image correspondence extracted by the camera 300. Then, after the tool 230 is attached to the target object 400, the actual coordinate values of each element in the robot arm system 20 can be substituted into the rotation matrix A, the rotation matrix B and the rotation matrix D of the equation (2). In the case where the rotation matrix A, the rotation matrix B and the rotation matrix D are known, the rotation matrix C can be derived based on the equation (2). However, since the correction of the rotation matrix B is performed by a person operating the pose of the robot arm 200 to touch the fixed point, the correctness of the rotation matrix B can be affected by the operation of the person (whether the same position is actually touched). In addition, since the rotation matrix C is derived based on the rotation matrix B, the rotation matrix B and the rotation matrix C usually have errors. Therefore, the rotation matrix B and the rotation matrix C need to be corrected at the same time, so that the coordinate conversion relationship between each element of the robot arm system 20 is consistent with the equation (2). In order to solve the above problems, the hand-eye calibration device 100 is disclosed. The hand-eye calibration device 100 can simultaneously correct the rotation matrix B and the rotation matrix C of the robot arm system 20.

[0052] Each rotation matrix in the equation (2) can be associated with 6 coordinate values. Taking the rotation matrix B as an example, the value of the rotation matrix B can be determined by the following 6 coordinate values: the relative displacement coordinate value x of the end 220 and the tool 230, the relative displacement coordinate value y of the end 220 and the tool 230, the relative displacement coordinate value z of the end 220 and the tool 230, the relative angle coordinate value rx of the end 220 and the tool 230, the relative angle coordinate value ry of the end 220 and the tool 230, and the relative angle coordinate value rz of the end 220 and the tool 230. Taking the rotation matrix C as an example, the value of the rotation matrix C can be determined by the following 6 coordinate values: the relative displacement coordinate values x, y and z of the camera 300 relative to the base 210, and the relative angle coordinate values rx, ry and rz of the camera 300 relative to the base 210.

[0053] Figure 3 According to an embodiment of the present disclosure, a flowchart of a hand-eye calibration method is shown, wherein the hand-eye calibration can be implemented by the hand-eye calibration device 100 as shown in Figure 1 . In this embodiment, it is assumed that the robot arm system 20 is in a state to be calibrated. In step S301, the hand-eye calibration device 100 obtains the first mapping relationship between the base of the robot arm and the end of the robot arm, and the second mapping relationship between the camera and the target object. Specifically, refer to Figure 4 . Figure 4A flowchart of performing hand-eye calibration is shown in more detail according to an embodiment of the present disclosure. In step S401, the processor 110 can obtain N rotation matrices A x from the robot arm system 20 through the transceiver 130, and can obtain N rotation matrices D x from the images extracted by the camera 200, where x is the index of the rotation matrix (x = 1 ~ N), and N is a positive integer.

[0054] The user can manually move the robot arm 200 so that the tool 230 is attached to the target object 400. After the attachment is completed, the robot arm system 20 can output a set of rotation matrices according to the positions of the elements. Then, the user can again manually move the robot arm 200 so that the tool 230 is attached to the target object 400 in another way. After the attachment is completed, the robot arm system 20 can output another set of rotation matrices according to the positions of the elements. This is repeated N times, and the robot arm system 20 can output N sets of rotation matrices as shown in Table 1. The processor 110 can be communicatively connected to the robot arm system 20 through the transceiver 130 to obtain the information of Table 1 from the robot arm system 20.

[0055] Table 1

[0056]

[0057] After obtaining the information of Table 1, the processor 110 can sequentially update the rotation matrices B and C in each dimension (i.e., X-axis, Y-axis, Z-axis, twist angle RX along X-axis, twist angle RY along Y-axis, and twist angle RZ along Z-axis) based on a scale to minimize the error (hereinafter referred to as "error e") between the position of the target object 400 in the image extracted by the camera 300 and the position of the tool 230. In detail, the processor 110 sequentially updates the rotation matrices B and C in each dimension using a single-axis convergence operation to reduce the complexity of the operation.

[0058] Returning to Figure 3 In step S302, the processor 110 can sequentially update the third mapping relationship between the end of the robot arm and the tool mounted on the end and the fourth mapping relationship between the camera and the base based on a scale to minimize the error between the position of the target object in the image extracted by the camera and the position of the tool. In detail, refer to Figure 4 In step S402, the processor 110 can update the initial value of the rotation matrix B to obtain the rotation matrix B' according to a scale Δ(i, and can update the initial value of the rotation matrix C to obtain the rotation matrix C' according to a scale Δ(i. The processor 110 can update the initial values of the rotation matrices B and C according to the flow as shown in Figure 4 ​

[0059] Figure 5 A flowchart illustrating the process of updating the rotation matrix, i.e., the detailed process of step S402, step S403, or step S409, is shown according to an embodiment of the present disclosure. In step S501, the processor 110 can obtain initial rotation coordinates B i and initial rotation coordinates C i-6 . If the processor 110 is performing step S402, the initial rotation coordinates B i and the initial rotation coordinates C i-6 may be initial correction values (e.g., B and C in Table 1). If the processor 110 is performing step S403, the initial rotation coordinates B i may be the rotation coordinates B’ generated in step S402, and the initial rotation coordinates C i-6 may be the rotation coordinates C’ generated in step S302. If the processor 110 is performing step S409, the initial rotation coordinates B i may be the rotation coordinates B” generated in step S403, and the initial rotation coordinates C i-6 may be the rotation coordinates C” generated in step S403.

[0060] In step S502, the processor 110 can obtain coordinate values corresponding to the rotation matrix B and coordinate values corresponding to the rotation matrix C, and generate M (M is a positive integer) offset coordinate values according to a current scale and the coordinate values, as shown in equation (4), where P BC (i,j) can represent the jth (j = 1 ~ M) offset coordinate value corresponding to the ith (i = 1 ~ 6, i is an index associated with 6 coordinate values of the rotation matrix, and the initial value of i can be 1) coordinate value of the initial rotation matrix B i . When i > 6, P BC (i,j) can represent the jth (j = 1 ~ M) offset coordinate value corresponding to the (i-6)th (i = 7 ~ 12, i is an index associated with 6 coordinate values of the rotation matrix, and the initial value of i can be 7) coordinate value of the initial rotation matrix C i-6 . Δ(i can represent the current scale, and a(j can represent a weight corresponding to the jth offset coordinate value. When i ≤ 6, P BC (i) can represent the ith coordinate value of the initial rotation matrix B i . When i > 6, P BC (i) can represent the (i-6)th coordinate value of the initial rotation matrix C i-6 . For example, when i = 1, P BC (i) can represent the relative displacement coordinate value x associated with the end 220 and the tool 230 in the initial rotation matrix B i . When i = 2, P BC(i) can represent the initial rotation matrix B i (i) can represent the relative displacement coordinate value x between the end 220 and the tool 230. When i = 1, P BC (i) can represent the initial rotation matrix B i (i) can represent the relative displacement coordinate value z between the end 220 and the tool 230. When i = 4, P BC (i) can represent the initial rotation matrix B i (i) can represent the relative angular coordinate value rx between the end 220 and the tool 230. When i = 5, P BC (i) can represent the initial rotation matrix B i (i) can represent the relative angular coordinate value ry between the end 220 and the tool 230. When i = 6, P BC (i) can represent the initial rotation matrix B i (i) can represent the relative angular coordinate value rz between the end 220 and the tool 230. When i = 7, P BC (i) can represent the initial rotation matrix C i-6 (i) can represent the relative displacement coordinate value x between the camera 300 and the base 210. When i = 8, P BC (i) can represent the initial rotation matrix C i-6 (i) can represent the relative displacement coordinate value y between the camera 300 and the base 210. When i = 9, P BC (i) can represent the initial rotation matrix C i-6 (i) can represent the relative displacement coordinate value z between the camera 300 and the base 210. When i = 10, P BC (i) can represent the initial rotation matrix C i-6 (i) can represent the relative angular coordinate value rx between the camera 300 and the base 210. When i = 11, P BC (i) can represent the initial rotation matrix C i-6 (i) can represent the relative angular coordinate value ry between the camera 300 and the base 210. When i = 12, P BC (i) can represent the initial rotation matrix C i-6 (i) can represent the relative angular coordinate value rz between the camera 300 and the base 210.

[0061] P BC (i,j) = P BC (i) ± α(j) · Δ(i)... (4)

[0062] Notably, the scale Δ(i) can include scales of different units. Specifically, the scale Δ(i) can include scales corresponding to the position of the robot arm 200 (i.e., scales related to coordinate values x, y, and z) and scales corresponding to the pose of the robot arm 200 (i.e., scales related to coordinate values rx, ry, and rz). Taking the scale Δ(4) as an example, assume that i = 7, the offset coordinate value P BC (i) is related to the relative displacement coordinate value x of the camera 300 and the base 210. Accordingly, the unit of the scale Δ(i) can be a displacement amount. Assume that i = 10, the offset coordinate value P BC (i) is related to the relative angle coordinate value rx of the camera 300 and the base 210. Accordingly, the unit of the scale Δ(i) can be an angle.

[0063] In step S503, the processor 110 can select an elected offset coordinate value P BC (i,j) (j = 1 ~ M) that satisfies the equation (5) (i.e., select the elected offset coordinate value P corresponding to the smallest error e) from the M offset coordinate values P i (i,j) and can update the initial rotation coordinate B i-6 (i,j) (when i > 6). When i ≤ 6, B i (P BC (i,j) can be the initial rotation coordinate B BC (i,j) that is replaced by P i (i,j). Dis(A(k) · B i (P BC (i,j), C i-6 · D(k)) can be the distance (e.g., Manhattan distance or Euclidean distance) between A(k) · B i (P BC (i,j) and C i-6 · D(k). For example, if the elected offset coordinate value P is the coordinate value P BC (1,1), the processor 110 can use the coordinate value P BC (1,1) to replace the relative displacement coordinate value x of the end 220 and the tool 230 in the initial rotation coordinate B i (i,j) and update the initial rotation coordinate B i (i,j). When the rotation matrix A(k), the initial rotation matrix B i (P Bc (i,j), the initial rotation matrix C i-6 , and the rotation matrix D(k) are substituted into the identity equation (2), the elected offset coordinate value P can minimize the error between the left-hand side and the right-hand side of the identity equation (2).

[0064]

[0065] Similarly, when i > 6 (i.e., after updating the initial rotation coordinates B) i Afterwards, processor 110 can obtain M offset coordinate values ​​P BC Select the selected offset coordinate value P from (i,j)(j=1~M) that satisfies the formula (5), and update the initial rotation coordinate C according to the selected offset coordinate value P. i-6 C i-6 (P BC (i,j) can be the (i-6)th coordinate value that P can be used for. BC (i,j) replaces the initial rotation coordinates C i-6 Dis(A(k)· i ,i-6(P BC (i,j))·D(k)) can be A(k)· i With C i-6 (P BC The distance between (i,j))·D(k). For example, if the selected offset coordinate value P is the coordinate value P BC (7,1, then processor 110 can use coordinate value P) BC (7,1 is used to replace the initial rotation coordinates C) i-6 The relative displacement coordinates x between the camera 300 and the base 210 are used to update the initial rotation coordinates C. i-6 When the rotation matrix A(k) and the initial rotation matrix B i Initial rotation matrix C i-6 (P BC When (i,j)) and the rotation matrix D(k) are substituted into identity (2), the selected offset coordinate value P can minimize the error between the left and right sides of identity (2).

[0066] In step S504, the processor 110 determines whether the value of i is equal to 12. If the value of i is less than 12, the process proceeds to step S505, where the processor 110 increments the value of i by 1. If the value of i is equal to 12, the process proceeds to step S506.

[0067] In step S506, the processor 110 can complete the initial rotation matrix B i and C i-6 Update. If processor 110 is executing step S402, then processor 110 may generate rotation matrix B' and rotation matrix C' in step S506. If processor 110 is executing step S403, then processor 110 may generate rotation matrix B"" and rotation matrix C"" in step S506. If processor 110 is executing step S409, then processor 110 may generate rotation matrix B""' and rotation matrix C"' in step S506.

[0068] Returning to Figure 4 In step S403, the processor 110 can update the rotation matrix B' to obtain a rotation matrix B" according to the scale Δ(i), and can update the rotation matrix C' to obtain a rotation matrix C" according to the scale Δ(i). The processor 110 can update the rotation matrix B' and the rotation matrix C' according to the flowchart as shown in FIG. 4B, which will not be described herein again. In the present embodiment, the purposes of the operation steps S402 and S403 are to repeatedly verify to avoid the error e caused by the update of one dimension (e.g., x, y, z, rx, ry, or rz) affecting another dimension. Figure 4

[0069] In step S404, the processor 110 can calculate the error E' corresponding to the rotation matrix B' and the rotation matrix C' according to the equation (6), and can calculate the error E" corresponding to the rotation matrix B" and the rotation matrix C" according to the equation (7). Then, the processor 110 can determine whether the absolute difference between the error E' and the error E" is less than or equal to the difference threshold T1, that is, the processor 110 can determine whether |E''-E'|≤T1 is correct. If the absolute difference between the error E' and the error E" is less than or equal to the difference threshold T1 (representing that the error e converges), the step S406 is entered. If the absolute difference between the error E' and the error E" is greater than the difference threshold T1 (representing that the error e does not converge), the step S405 is entered.

[0070]

[0071]

[0072] In step S405, the processor 110 can set the initial value of the rotation matrix B for the step S402 as the current rotation matrix B', and can set the initial value of the rotation matrix C for the step S402 as the current rotation matrix C'. After completing the step S405, the processor 110 can re-execute the step S402 to obtain the updated rotation matrix B' and the rotation matrix C'.

[0073] In step S406, the processor 110 can determine whether the current scale Δ(i) is less than or equal to the scale threshold T2, that is, the processor 110 can determine whether Δ(i)≤T2 is correct. If the scale Δ(i) is less than or equal to the scale threshold T2, it represents that the correction of the rotation matrix B and the rotation matrix C by the processor 110 is very accurate. Accordingly, the processor 110 can execute the step S409. On the other hand, if the scale Δ(i) is greater than or equal to the scale threshold T2, the step S407 is entered.

[0074] ​In step S407, the processor 110 can decrease the scale Δ(i). After the scale Δ(i) is decreased, the processor 110 can then perform step S405.

[0075] When the result of the determination in step S406 is YES, it means that the scale Δ(i) currently used is already accurate enough to correct the rotation matrix B and the rotation matrix C. However, even if the accuracy of the correction result is already very high, the correction result can still be trapped in a local optimal solution instead of a global optimal solution. To solve the above problem, in step S408, the processor 110 can increase the scale Δ(i). Then, in step S409, the processor 110 can update the rotation matrix B" according to the scale Δ(i) to obtain a rotation matrix B'" and can update the rotation matrix C" according to the scale Δ(i) to obtain a rotation matrix C"'. The processor 110 can update the rotation matrix B" and the rotation matrix C" according to the flowchart as shown in FIG. 8, and thus will not be described again. Figure 4

[0076] The processor 110 can determine whether updating the rotation matrix B and the rotation matrix C according to the increased scale Δ(i) increases the error e. If YES, step S411 is entered. If NO, step S405 is entered. In detail, in step S410, the processor 110 can calculate an error E'" corresponding to the rotation matrix B'" and the rotation matrix C'" according to equation (8). Then, the processor 110 can determine whether the error E'" is smaller than the error E", that is, the processor 110 can determine whether E'" < E" is correct. If the error E'" is smaller than the error E", it means that the error of equation (2) is reduced after the scale Δ(i) is increased. That is, the rotation matrix B' and the rotation matrix C' obtained in step S402 (or the rotation matrix B" and the rotation matrix C" obtained in step S303) are a local optimal solution instead of a global optimal solution. Accordingly, the processor 110 can perform step S405 to further update the rotation matrix B' and the rotation matrix C' according to the current scale Δ(i) to minimize the error e. On the other hand, if the error E'" is greater than or equal to the error E", it means that the rotation matrix B' and the rotation matrix C' obtained in step S402 (or the rotation matrix B" and the rotation matrix C" obtained in step S303) are a global optimal solution. Accordingly, the processor 110 can perform step S411.

[0077] Returning to Figure 3 In step 303, in response to the error e converging and the scale Δ(i) being smaller than or equal to the scale threshold T2, the processor 110 can output the mapping relationship between the end 220 and the tool 230 and the mapping relationship between the base 210 and the camera 300 corrected by the scale Δ(i). In detail, referring to FIG. 7, the processor 110 can output the mapping relationship between the end 220 and the tool 230 and the mapping relationship between the base 210 and the camera 300 corrected by the scale Δ(i) to the display 120. Figure 4 ​In step S411, the processor 110 can generate a corrected rotation matrix B according to the rotation matrix B' (or the rotation matrix B''), and can generate a corrected rotation matrix C according to the rotation matrix C' (or the rotation matrix C''). Then, the processor 110 can output the corrected rotation matrix B and the corrected rotation matrix C through the transceiver 130.

[0078] In an embodiment, the processor 110 can select one of the rotation matrix B' and the rotation matrix B'' as the corrected rotation matrix B. In addition, the processor 110 can select one of the rotation matrix C' and the rotation matrix C'' as the corrected rotation matrix C.

[0079] In summary, for the coordinate conversion relationship of each element in the robot arm system, the hand-eye calibration device of the present disclosure can correct the rotation matrix that may have errors in the coordinate conversion relationship by using a specific scale. After the correction of the rotation matrix is completed, the hand-eye calibration device can reduce the scale and correct the rotation matrix again. In this way, the hand-eye calibration device can continuously correct the rotation matrix with a smaller scale to make the rotation matrix more accurate. In addition, after the correction of the rotation matrix is completed, the hand-eye calibration device can increase the scale to update the rotation matrix, so as to confirm whether the corrected rotation matrix is a global optimal solution. Therefore, the hand-eye calibration device can avoid the calibration result of the robot arm falling into a local optimal solution. In addition, the correction process sequentially corrects each dimension, that is, uses a single-axis convergence method, which can greatly reduce the computational complexity.

Claims

1. A hand-eye calibration method for a hand-eye based robot arm, the method comprising: comprising: obtaining a first mapping relationship between a base of the robot arm and an end of the robot arm and a second mapping relationship between a camera and a target object; updating a third mapping relationship between the end of the robot arm and a tool mounted on the end of the robot arm and a fourth mapping relationship between the camera and the base in each dimension based on a scale in sequence to minimize an error of a position of the target object in an image extracted by the camera and a position of the tool; and in response to the error converging and the scale being less than or equal to a scale threshold, outputting the third mapping relationship and the fourth mapping relationship corrected by the scale. further comprising:

2. The hand-eye calibration method of claim 1, wherein, in response to the error converging and the scale being greater than the scale threshold, reducing the scale to update the third mapping relationship and the fourth mapping relationship. the step of updating the third mapping relationship and the fourth mapping relationship in each dimension based on the scale to minimize the error comprises:

3. The hand-eye calibration method of claim 1, wherein, obtaining a coordinate value corresponding to the third mapping relationship, and generating a plurality of offset coordinate values according to the scale and the coordinate value; selecting an elected offset coordinate value corresponding to the smallest error from the plurality of offset coordinate values according to the first mapping relationship and the second mapping relationship; and updating the third mapping relationship according to the elected offset coordinate value. the step of outputting the third mapping relationship and the fourth mapping relationship corrected by the scale in response to the error converging and the scale being less than or equal to the scale threshold comprises:

4. The hand-eye calibration method of claim 1, wherein, increasing the scale, and determining whether updating the third mapping relationship and the fourth mapping relationship according to the increased scale increases the error; if yes, outputting the third mapping relationship and the fourth mapping relationship corrected by the scale; and if no, updating the third mapping relationship and the fourth mapping relationship according to the increased scale to minimize the error. further comprising:

5. The hand-eye calibration method of claim 1, wherein, updating the third mapping relationship based on the scale to generate a first rotation matrix, and updating the fourth mapping relationship based on the scale to generate a second rotation matrix; calculating a first error between both the third mapping relationship and the fourth mapping relationship and both the first rotation matrix and the second rotation matrix; updating the first rotation matrix based on the scale to generate a third rotation matrix, and updating the second rotation matrix based on the scale to generate a fourth rotation matrix; calculating a second error between both the first rotation matrix and the second rotation matrix and both the third rotation matrix and the fourth rotation matrix; and in response to an absolute difference between the first error and the second error being less than or equal to a difference threshold, determining that the error converges. the each dimension comprises an X-axis, a Y-axis, a Z-axis, a twist angle along the X-axis, a twist angle along the Y-axis, and a twist angle along the Z-axis. ​ 6. The hand-eye calibration method of claim 1, wherein, ​ 7. The hand-eye calibration method of claim 1, wherein, The first mapping relationship includes a coordinate conversion relationship between the base and the end, the second mapping relationship includes a coordinate conversion relationship between the camera and the target object, the third mapping relationship includes a coordinate conversion relationship between the end and the tool, and the fourth mapping relationship includes a coordinate conversion relationship between the camera and the base.

8. A hand-eye correction device, suitable for an eye-to-hand robotic arm, characterized in that, Comprise: a transceiver; and a processor coupled to the transceiver, wherein the processor is configured to perform: obtaining, by the transceiver, a first mapping relationship between a base of the robot arm and an end of the robot arm and a second mapping relationship between a camera and a target object; updating, based on a scale, a third mapping relationship between the end of the robot arm and a tool mounted on the end and a fourth mapping relationship between the camera and the base in each dimension in sequence to minimize an error of a position of the target object in an image extracted by the camera and a position of the tool; and in response to the error converging and the scale being less than or equal to a scale threshold, outputting, by the transceiver, the third mapping relationship and the fourth mapping relationship corrected by the scale.

9. The hand-eye correction device of claim 8, wherein, The processor is further configured to perform: in response to the error converging and the scale being greater than the scale threshold, reducing the scale to update the third mapping relationship and the fourth mapping relationship.

10. The hand-eye correction device of claim 8, wherein, The processor is further configured to perform: obtaining a coordinate value corresponding to the third mapping relationship, and generating a plurality of offset coordinate values according to the scale and the coordinate value; selecting, according to the first mapping relationship and the second mapping relationship, a selected offset coordinate value corresponding to the smallest error from the plurality of offset coordinate values; and updating the third mapping relationship according to the selected offset coordinate value.

11. The hand-eye correction device of claim 8, wherein, The processor is further configured to perform: increasing the scale, and determining whether updating the third mapping relationship and the fourth mapping relationship according to the increased scale increases the error; if yes, outputting the third mapping relationship and the fourth mapping relationship corrected by the scale; and if no, updating the third mapping relationship and the fourth mapping relationship according to the increased scale to minimize the error.

12. The hand-eye correction device of claim 8, wherein, The processor is further configured to perform: updating the third mapping relationship based on the scale to generate a first rotation matrix, and updating the fourth mapping relationship based on the scale to generate a second rotation matrix; calculating a first error between both the third mapping relationship and the fourth mapping relationship and both the first rotation matrix and the second rotation matrix; updating the first rotation matrix based on the scale to generate a third rotation matrix, and updating the second rotation matrix based on the scale to generate a fourth rotation matrix; calculating a second error between both the first rotation matrix and the second rotation matrix and both the third rotation matrix and the fourth rotation matrix; and in response to an absolute difference between the first error and the second error being less than or equal to a difference threshold, determining that the error converges.

13. The hand-eye correction device of claim 8, wherein, The dimensions include an X-axis, a Y-axis, a Z-axis, a twist angle along the X-axis, a twist angle along the Y-axis, and a twist angle along the Z-axis.

14. The hand-eye correction device of claim 8, wherein, The first mapping relationship includes a coordinate conversion relationship between the base and the end, the second mapping relationship includes a coordinate conversion relationship between the camera and the target object, the third mapping relationship includes a coordinate conversion relationship between the end and the tool, and the fourth mapping relationship includes a coordinate conversion relationship between the camera and the base.

Images