A real-time hand-eye calibration method for multi-task scenarios
By calculating the homogeneous transformation matrix of the robot's end effector and the camera, and combining it with the least squares method, the problem of complexity and time-consuming traditional hand-eye calibration methods is solved, achieving fast and accurate hand-eye calibration and improving the robot's operating efficiency and flexibility.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUNAN UNIV
- Filing Date
- 2024-03-25
- Publication Date
- 2026-07-21
Smart Images
Figure CN118052890B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot calibration technology, and in particular to a real-time hand-eye calibration method for multi-task scenarios. Background Technology
[0002] Hand-eye calibration is a critical task in robotics and computer vision, used to determine the accurate relationship between a robot's hand and its vision system (eye). In traditional robot operation, a "eye on hand" approach is generally used for sampling, meaning that the camera or vision system is mounted on the robot arm, usually near the robot's end effector.
[0003] This method has significant limitations. The relative positions of the arm and the camera must remain constant. In actual operation, the robotic arm may not need to guide the operation in some cases, so the camera mounting device must be removed. However, if the camera system is required for visual guidance in another operation scenario, tedious hand-eye calibration is required, which is very troublesome and time-consuming.
[0004] In the traditional hand-eye calibration process, if visual guidance is required in a certain scenario, hand-eye calibration needs to be performed once. However, for complex scenarios or robot operation scenarios without visual guidance, the camera device needs to be removed and operated multiple times. Furthermore, the traditional hand-eye calibration needs to be performed again in the next operation that requires visual guidance. This is very cumbersome and wastes a lot of time in a fixed and routine calibration process.
[0005] Furthermore, the final visual guidance requires determining the pose relationship between the camera coordinate system and the robotic arm base coordinate system. Hand-eye calibration only determines the pose relationship between the robotic arm's end effector and the camera. Furthermore, it is necessary to multiply by the pose transformation between the robot base and the robot end effector. .
[0006] For different complex scenarios, hand-eye calibration must be performed each time, and then the final pose transformation between the camera coordinate system and the base coordinate system must be calculated based on the pose transformation between the robot base and the robot end effector. This process is very complex, time-consuming, and labor-intensive, and greatly reduces the efficiency of the entire robot operation. Summary of the Invention
[0007] This invention provides a real-time hand-eye calibration method for multi-task scenarios, which solves the technical problems of existing robot hand-eye calibration methods, such as complex calibration process, time and labor consumption, and reduced overall robot operation efficiency.
[0008] To achieve the above objectives, the technical solution of the present invention is implemented as follows: This invention provides a real-time hand-eye calibration method for multi-task scenarios, comprising the following steps: S1. Based on the position and pose of the robot end-effector coordinate system after moving relative to the base coordinate system, calculate the homogeneous transformation matrix of the base coordinate system relative to the robot end-effector coordinate system after each movement; then calculate the homogeneous transformation matrix of the camera coordinate system relative to the calibration pattern coordinate system after each movement. S2. Based on the data obtained in S1, complete the calibration of the relationship between the robot base coordinate system and the calibration pattern coordinate system, and obtain the homogeneous transformation matrix of the calibration pattern coordinate system relative to the robot base coordinate system. S3. Use the data obtained in S2 to complete the real-time calibration of the relationship between the robot base coordinate system and the camera coordinate system, as well as the real-time calibration of the relationship between the robot end effector coordinate system and the camera coordinate system. S4. Use the data obtained in S3 to complete the robot calibration task.
[0009] Furthermore, step S1 specifically includes the following steps: S11. First, define α, β, It is the Euler angle of the rotation of the robot's end-effector coordinate system relative to the base coordinate system; and it satisfies the following relationship: ; in, Rotate the initial coordinate system around the x-axis The rotation matrix after °; Let be the rotation matrix after rotating the initial coordinate system around the y-axis by β°; The rotation matrix is the result of rotating the initial coordinate system by α° around the z-axis. , , All are angles of rotation; S12. Then, use the relation in S11 to obtain the rotation matrix of the robot's end effector coordinate system relative to the base coordinate system. ; S13. Utilize the rotation matrix of the robot's end effector coordinate system relative to the base coordinate system. Given the relative position coordinates t, the homogeneous transformation matrix of the robot's end effector coordinate system relative to the base coordinate system is obtained. ; S14. Using the homogeneous transformation matrix of the robot's end effector coordinate system relative to the base coordinate system. Find the homogeneous transformation matrix of the base coordinate system relative to the robot end effector coordinate system. ,and = ; S15. Calculate the rotation matrix of the calibration pattern coordinate system relative to the camera coordinate system. ; S16. Using the rotation matrix of the calibration pattern coordinate system relative to the camera coordinate system. The homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system is obtained. ; S17. By calibrating the homogeneous transformation matrix of the pattern coordinate system relative to the camera coordinate system. The homogeneous transformation matrix of the camera coordinates relative to the calibration pattern coordinate system is obtained. ,and = .
[0010] Furthermore, the rotation matrix of the robot end effector coordinate system relative to the base coordinate system in S12 The specific formula is as follows: .
[0011] Furthermore, the homogeneous transformation matrix of the robot end effector coordinate system relative to the base coordinate system in S13 The specific formula is as follows: .
[0012] Furthermore, step S2 specifically includes the following steps: S21. Keep the position of the calibration pattern unchanged, keep the position of the camera relative to the robot's end joint unchanged, capture a photo of the calibration pattern with the camera, and record the 6D pose of the base on the robot teach pendant relative to the robot's end joint. S22. Control the robot to move randomly multiple times in space, collect multiple calibration pattern photos taken in different poses, and record the 6D pose of the base on the robot teach pendant relative to the robot end effector for each movement. S23. Based on the images captured by the camera during each movement and the data recorded in S21 and S22, calculate the homogeneous transformation matrix of the base coordinate system relative to the robot end effector coordinate system and the homogeneous transformation matrix of the camera coordinate system relative to the calibration pattern coordinate system during each movement. Solve using the least squares method to obtain the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system. .
[0013] Furthermore, step S23 specifically includes the following steps: S231. Based on the data obtained in S16, the homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system obtained during each movement is defined as follows: , , ; S232, Based on the data obtained in S231 and = Solve for the homogeneous transformation matrices of the camera coordinate system relative to the calibration pattern coordinate system as follows: , , ; Based on the data obtained in S13 and the teach pendant data recorded during each movement, the homogeneous transformation matrices of the robot's end effector relative to the base coordinate system are calculated as follows: , , ; S234, Based on the data obtained in S233 and = The homogeneous transformation matrices of the base coordinate system relative to the robot end effector coordinate system are obtained as follows: , , ; S235. Based on the data obtained from S232 and S234, and using the homogeneous transformation matrix of the camera coordinate system relative to the robot end effector coordinate system... Unchanged, that is = * * Without changing the coordinates, we list the matrix equations and solve them using the least squares method to obtain the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system. .
[0014] Furthermore, the matrix equation in S235 is as follows: * * = * * = * * .
[0015] Furthermore, step S3 specifically includes the following steps: S31. Control the robot to move again, and the camera will capture a photo of the calibration pattern to obtain the 6D pose of the camera coordinate system relative to the calibration pattern coordinate system. The initial position of the camera can be changed during this process. S32. If the camera is mounted on the robot's end arm, i.e., the eye is on the hand, then proceed to S33. If the camera is mounted outside the robot's end arm, i.e., the eye is outside the hand, then proceed to S34. S33. According to S14 and S17, we know that... = , = And based on the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system obtained in S2... The homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system is calculated using data acquired by the camera and data from the robot's end effector. Homogeneous transformation matrix of robot end-effector coordinate system relative to base coordinate system And complete the real-time calibration of the relationship between the robot's end-effector coordinate system and the camera coordinate system. * * ; S34, According to S17, we know = And based on the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system obtained in S2 And using the 6D pose of the camera coordinate system relative to the calibration pattern coordinate system obtained in S31, the homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system is calculated. And complete the real-time calibration of the relationship between the robot base coordinate system and the camera coordinate system. = * .
[0016] Furthermore, step S4 specifically includes the following steps: S41. If the camera is mounted on the robot's end arm, i.e., the eye is on the hand, then proceed to S42. If the camera is mounted outside the robot's end arm, i.e., the eye is outside the hand, then proceed to S43. S42, based on S33 * * And based on the relationship between the robot's end effector and the base Then use a camera to obtain the three-dimensional coordinates of the target point. ,in = Find the coordinates of the target point in the base coordinate system. = This completes the entire calibration task. S43. Without changing the camera position, use the homogeneous transformation matrix between the robot base coordinate system and the camera coordinate system obtained in S34. Multiply the target point by the camera to obtain its 3D coordinates. ,in = The coordinates of the target point in the base coordinate system can be obtained. = * This completes the entire calibration task.
[0017] The beneficial effects of this invention are: 1. Efficiently and quickly acquire hand-eye relative relationships This invention proposes an efficient hand-eye calibration method, the core of which lies in rapidly and accurately determining the relative relationship between the hand and the vision system. In this invention, the relationship between the known calibration pattern and the base coordinate system is first determined. Then, by combining the known relative relationships between the robot base and the end effector, and between the camera and the calibration pattern, the hand-eye relationship can be quickly determined. This invention specifically addresses the matrix calculation problem encountered in traditional hand-eye calibration tasks, providing a simplified process and significantly improving the computational efficiency and accuracy in complex engineering tasks.
[0018] 2. The installation location of the vision system is flexible and can be changed. The hand-eye calibration method provided by this invention is particularly suitable for robot operating environments that require frequent adjustments to the installation position of the vision system. The core of this invention lies in accurately calculating the relationship between the robot base coordinate system and the calibration plate coordinate system. The position of the calibration plate remains fixed, while the position of the camera can be flexibly changed without affecting the final calibration result. Regardless of the specific installation position of the camera, this invention ensures accurate and efficient hand-eye calibration, greatly improving the flexibility and adaptability of the robot system. Therefore, this invention provides an ideal technical solution for complex robot application scenarios requiring multiple hand-eye calibrations.
[0019] 3. Smaller error and higher calibration accuracy The hand-eye calibration method proposed in this invention has significant advantages in improving calibration accuracy. It can simultaneously solve for the rotation vector rvec and translation vector tvec of the hand-eye matrix, effectively avoiding the transformation and accumulation of errors in traditional calibration methods. Compared to traditional methods, it does not calculate the translation and rotation matrices separately, but directly calculates the homogeneous transformation matrix containing both components. This invention also significantly improves calibration accuracy. By accurately calculating the rotation and translation parameters, this invention can achieve higher-precision hand-eye calibration.
[0020] 4. Applicable to a variety of scenarios This invention applies to two different scenarios for hand-eye calibration: one where the camera (“eye”) is mounted on the robot arm (“hand”), and the other where the camera is mounted outside the robot arm. For these two different cases, we propose a unified, standardized procedure. In this procedure, the positions of the calibration pattern and the robot base remain unchanged, thereby solving for their relative transformation relationship. The purpose of this is to enable fast and real-time hand-eye calibration in future multi-scenario tasks, depending on the specific task requirements (sometimes the camera is on the robot arm, sometimes it is outside the arm). Attached Figure Description
[0021] Figure 1 This is a flowchart of the present invention; Figure 2 This is a schematic diagram of the calibration pattern used in the calibration of this invention; Figure 3 A schematic diagram illustrating the calibration of the relationship between the robot's base coordinate system and the camera coordinate system (eye outside the hand). Figure 4 A schematic diagram illustrating the calibration of the relationship between the robot's end effector coordinate system and the camera coordinate system (eye on hand). Detailed Implementation
[0022] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.
[0023] Reference Figures 1 to 4 This application provides a real-time hand-eye calibration method for multi-task scenarios, including the following steps: S1. Based on the position and pose of the robot end-effector coordinate system after moving relative to the base coordinate system (read from the robot teach pendant), calculate the homogeneous transformation matrix of the base coordinate system relative to the robot end-effector coordinate system after each movement; then calculate the homogeneous transformation matrix of the camera coordinate system relative to the calibration pattern coordinate system after each movement. The calibration pattern is placed within the robot's movement range and the camera's field of view, ensuring that the calibration pattern is clearly imaged within the camera's field of view and that the position of the calibration pattern remains fixed throughout the process; the camera position is temporarily fixed on the robot's end joint, and the relative position between the camera and the robot's end joint remains fixed. S2. Based on the data obtained in S1, complete the calibration of the relationship between the robot base coordinate system and the calibration pattern coordinate system, and obtain the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system. S3. Use the data obtained in S2 to complete the real-time calibration of the relationship between the robot base coordinate system and the camera coordinate system, as well as the real-time calibration of the relationship between the robot end effector coordinate system and the camera coordinate system. S4. Use the data obtained in S3 to complete the robot calibration task.
[0024] This invention proposes an efficient hand-eye calibration method, the core of which lies in rapidly and accurately determining the relative relationship between the hand and the visual system. In this invention, the relationship between a known calibration pattern and a reference coordinate system is first determined. Then, by combining the known relative relationships between the base and end effector, and between the camera and the calibration pattern, the hand-eye relationship can be quickly determined. This invention specifically addresses the matrix calculation problem encountered in traditional hand-eye calibration tasks, providing a simplified process and significantly improving the computational efficiency and accuracy in complex engineering tasks.
[0025] The hand-eye calibration method provided by this invention is particularly suitable for robot operating environments that require frequent adjustments to the installation position of the vision system. The core of this invention lies in accurately calculating the relationship between the robot base coordinate system and the calibration plate coordinate system. The position of the calibration plate remains fixed, while the position of the camera can be flexibly changed without affecting the final calibration result. Regardless of the specific installation position of the camera, this invention ensures accurate and efficient hand-eye calibration, greatly improving the flexibility and adaptability of the robot system. Therefore, this invention provides an ideal technical solution for complex robot application scenarios requiring multiple hand-eye calibrations.
[0026] In some embodiments, S1 specifically includes the following steps: S11. First, define α, β, It is the Euler angle of the rotation of the robot's end-effector coordinate system relative to the base coordinate system; and it satisfies the following relationship: ; in, Rotate the initial coordinate system around the x-axis The rotation matrix after °; Let be the rotation matrix after rotating the initial coordinate system around the y-axis by β°; The rotation matrix is the result of rotating the initial coordinate system by α° around the z-axis. , , All are angles of rotation; S12. Then, use the relation in S11 to obtain the rotation matrix of the robot's end effector coordinate system relative to the base coordinate system. ; S13. Utilize the rotation matrix of the robot's end effector coordinate system relative to the base coordinate system. Given the relative position coordinates t, the homogeneous transformation matrix of the robot's end effector coordinate system relative to the base coordinate system is obtained. ; S14. Using the homogeneous transformation matrix of the robot's end effector coordinate system relative to the base coordinate system. Find the homogeneous transformation matrix of the base coordinate system relative to the robot end effector coordinate system. ,and = ; S15. Calculate the rotation matrix of the calibration pattern coordinate system relative to the camera coordinate system. ; S16. Using the rotation matrix of the calibration pattern coordinate system relative to the camera coordinate system. The homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system is obtained. ; S17. By calibrating the homogeneous transformation matrix of the pattern coordinate system relative to the camera coordinate system. The homogeneous transformation matrix of the camera coordinates relative to the calibration pattern coordinate system is obtained. ,and = .
[0027] In some embodiments, the rotation matrix of the robot end-effector coordinate system relative to the base coordinate system in S12 The specific formula is as follows: .
[0028] In some embodiments, the homogeneous transformation matrix of the robot end-effector coordinate system relative to the base coordinate system in S13 The specific formula is as follows: .
[0029] In some embodiments, S2 specifically includes the following steps: S21. Keep the position of the calibration pattern unchanged and the position of the camera relative to the robot end joint unchanged. The camera captures a photo of the calibration pattern. According to the corresponding calibration program, the 6D pose of the camera coordinate system relative to the calibration pattern coordinate system can be directly displayed on the photo captured by the camera, and the 6D pose of the base on the robot teach pendant relative to the robot end joint is recorded. S22. Control the robot to move randomly multiple times in space. The movement range should not be too small. During the movement, the calibration pattern should be within the camera's field of view and the image should be clear. Collect multiple photos of the calibration pattern taken in different poses. At the same time, record the 6D pose of the base on the robot teach pendant relative to the robot end effector for each movement. S23. Based on the images captured by the camera during each movement and the data recorded in S21 and S22, calculate the homogeneous transformation matrix of the base coordinate system relative to the robot end effector coordinate system and the homogeneous transformation matrix of the camera coordinate system relative to the calibration pattern coordinate system during each movement. Solve using the least squares method to obtain the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system. .
[0030] The calculation of the homogeneous transformation matrix of the camera coordinate system relative to the calibration pattern coordinate system specifically includes the following steps: First, based on the pose data of the calibration pattern coordinate system relative to the camera coordinate system in the captured image, including the rotation vector rvec and the translation vector tvec, the homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system is calculated. The rotation vector rvec is displayed in Euler angles in radians in the captured image. The homogeneous transformation matrix of the camera coordinate system relative to the calibration pattern coordinate system is then obtained using the translation vector tvec. .
[0031] The hand-eye calibration method proposed in this invention has significant advantages in improving calibration accuracy. It can simultaneously solve for the rotation vector rvec and translation vector tvec of the hand-eye matrix, effectively avoiding the transformation and accumulation of errors in traditional calibration methods. Compared to traditional methods, it does not calculate the translation and rotation matrices separately, but directly calculates the homogeneous transformation matrix containing both components. This invention also significantly improves calibration accuracy. By accurately calculating the rotation and translation parameters, this invention can achieve higher-precision hand-eye calibration.
[0032] In some embodiments, S23 specifically includes the following steps: S231. Based on the data obtained in S16, the homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system obtained during each movement is defined as follows: , , ; S232, Based on the data obtained in S231 and = Solve for the homogeneous transformation matrices of the camera coordinate system relative to the calibration pattern coordinate system as follows: , , ; Based on the data obtained in S13 and the teach pendant data recorded during each movement, the homogeneous transformation matrices of the robot's end effector relative to the base coordinate system are calculated as follows: , , ; S234, Based on the data obtained in S233 and = The homogeneous transformation matrices of the base coordinate system relative to the robot end effector coordinate system are obtained as follows: , , ; S235. Based on the data obtained from S232 and S234, and using the homogeneous transformation matrix of the camera coordinate system relative to the robot end effector coordinate system... Unchanged, that is = * * Without changing the coordinates, we list the matrix equations and solve them using the least squares method to obtain the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system. .
[0033] In some embodiments, the matrix equation in S235 is specifically as follows: * * = * * = * * .
[0034] In some embodiments, S3 specifically includes the following steps: S31. Move the robot again, and the camera captures a photo of the calibration pattern, obtaining the 6D pose of the camera coordinate system relative to the calibration pattern coordinate system. The initial camera position can be changed during this process; it is not necessarily fixed compared to the camera position in S2. The camera position in S31 can be flexibly changed, adapting to different task scenarios. This is because the remaining process calculates the homogeneous transformation matrix of the camera coordinate system relative to the robot's base coordinate system, which only depends on the calibration pattern position and is independent of the camera itself. S32. If the camera is mounted on the robot's end arm, i.e., the eye is on the hand, then proceed to S33. If the camera is mounted outside the robot's end arm, i.e., the eye is outside the hand, then proceed to S34. S33. According to S14 and S17, we know that... = , = And based on the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system obtained in S2... The homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system is calculated using data acquired by the camera and data from the robot's end effector. Homogeneous transformation matrix of robot end-effector coordinate system relative to base coordinate system And complete the real-time calibration of the relationship between the robot's end-effector coordinate system and the camera coordinate system. * * The value obtained in S2 It remains unchanged because the positions of the robot and the calibration pattern remain unchanged, therefore It can be used continuously, whether the eye is on the hand (i.e., the camera is mounted on the robotic arm including the robot's end effector) or the eye is outside the hand (i.e., the camera is not mounted on the robotic arm including the robot's end effector). It is known that the hand-eye matrix can be calculated quickly; S34, According to S17, we know = And based on the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system obtained in S2 And using the 6D pose of the camera coordinate system relative to the calibration pattern coordinate system obtained in S31, the homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system is calculated. And complete the real-time calibration of the relationship between the robot base coordinate system and the camera coordinate system. = * .
[0035] In some embodiments, S4 specifically includes the following steps: S41. If the camera is mounted on the robot's end arm, i.e., the eye is on the hand, then proceed to S42. If the camera is mounted outside the robot's end arm, i.e., the eye is outside the hand, then proceed to S43. S42, based on S33 * * And based on the relationship between the robot's end effector and the base Then use a camera to obtain the three-dimensional coordinates of the target point. ,in = Find the coordinates of the target point in the base coordinate system. = This completes the entire calibration task. S43. Without changing the camera position, use the homogeneous transformation matrix between the robot base coordinate system and the camera coordinate system obtained in S35. Multiply the target point by the camera to obtain its 3D coordinates. ,in = The coordinates of the target point in the base coordinate system can be obtained. = * This completes the entire calibration task.
[0036] This invention applies to two different scenarios for hand-eye calibration: one where the camera (“eye”) is mounted on the robot arm (“hand”), and the other where the camera is mounted outside the robot arm. For these two different cases, we propose a unified, standardized procedure. In this procedure, the positions of the calibration pattern and the robot base remain unchanged, thereby solving the relationship between them. The purpose of this is to enable fast and real-time hand-eye calibration in future multi-scenario tasks, depending on the specific task requirements (sometimes the camera is on the robot arm, sometimes it is outside the arm).
[0037] This invention addresses the technical problem that traditional hand-eye calibration methods require the pose information of a robot teach pendant and the calibration pattern information captured by a camera as input, which is a relatively complex process. It proposes a real-time and fast hand-eye calibration method for robot vision system position change calibration scenarios. This method is applicable to multiple repetitive calibration scenarios as well as scenarios where the position of the vision system changes. It is also applicable to calibration tasks where the eye is outside the hand and the eye is on the hand. Furthermore, it avoids the complex and tedious calculations required by traditional calibration methods in multiple calibration tasks.
[0038] Unlike traditional hand-eye calibration methods that directly calculate the relative relationship between the robot's end effector and the camera, this invention indirectly calculates the relative relationship between the calibration pattern and the robot base. Furthermore, since the camera calculates the relationship between the calibration pattern coordinate system and the camera coordinate system using Canny marker detection, Harris corner detection algorithms, and PNP pose estimation algorithms, the relative relationship between the calibration pattern and the robot base, as well as the relationship between the calibration pattern and the camera, is known. This allows for the rapid calculation of the relative relationship between the camera and the robot base, enabling quick calibration. In complex robot operations, the relative relationship between the camera and the robot base is crucial. The camera can directly acquire the coordinates of external spatial points, allowing for rapid calculation of the coordinates of those points on the robot base, thus guiding the robot closer to the target location for its operations. Moreover, after determining the relative relationship between the calibration pattern and the robot base, the camera's position can be freely changed without affecting the final calculation of the relative relationship. In repeated calibrations, as long as the pattern position remains unchanged, the relative relationship between the pattern and the camera can be directly acquired to quickly determine the relative relationship between the camera and the base coordinate system for the next calibration.
[0039] In traditional camera calibration processes (whether eye-on-hand or eye-on-hand), the camera position must remain fixed at all times. Furthermore, if multiple calibration tasks are performed, the tedious calculation process needs to be repeated multiple times, reducing the overall efficiency of the experiment.
[0040] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A real-time hand-eye calibration method for multi-task scenarios, characterized in that, Includes the following steps: S1. Based on the position and pose of the robot end-effector coordinate system after moving relative to the base coordinate system, calculate the homogeneous transformation matrix of the base coordinate system relative to the robot end-effector coordinate system after each movement; then calculate the homogeneous transformation matrix of the camera coordinate system relative to the calibration pattern coordinate system after each movement. S1 specifically includes the following steps: S11. First, define α, β, It is the Euler angle of the rotation of the robot's end-effector coordinate system relative to the base coordinate system; and it satisfies the following relationship: ； in, Rotate the initial coordinate system around the x-axis The rotation matrix after °; Let be the rotation matrix after rotating the initial coordinate system around the y-axis by β°; The rotation matrix is the result of rotating the initial coordinate system by α° around the z-axis. , , All are angles of rotation; S12. Then, use the relation in S11 to obtain the rotation matrix of the robot's end effector coordinate system relative to the base coordinate system. ; S13. Utilize the rotation matrix of the robot's end effector coordinate system relative to the base coordinate system. Given the relative position coordinates t, the homogeneous transformation matrix of the robot's end effector coordinate system relative to the base coordinate system is obtained. ; S14. Using the homogeneous transformation matrix of the robot's end effector coordinate system relative to the base coordinate system. Find the homogeneous transformation matrix of the base coordinate system relative to the robot end effector coordinate system. ,and = ; S15. Using the depth camera, the Canny marker detection and Harris corner detection algorithms, along with the PNP pose estimation algorithm, are called to obtain the Euler angles α and β of the calibration pattern coordinate system relative to the camera coordinate system. And the relative positions x, y, z of the calibration pattern coordinate system and the camera coordinate system; S16. Based on the relative poses α and β obtained in S15, The rotation matrix of the calibration pattern coordinate system relative to the camera coordinate system is calculated using the relative positions x, y, z and the relationship in S11. The relative position coordinates t are used to obtain the homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system. ; S17. By calibrating the homogeneous transformation matrix of the pattern coordinate system relative to the camera coordinate system. The homogeneous transformation matrix of the camera coordinates relative to the calibration pattern coordinate system is obtained. ,and = ; S2. Based on the data obtained in S1, complete the calibration of the relationship between the robot base coordinate system and the calibration pattern coordinate system, and obtain the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system. S3. Use the data obtained in S2 to complete the real-time calibration of the relationship between the robot base coordinate system and the camera coordinate system, as well as the real-time calibration of the relationship between the robot end effector coordinate system and the camera coordinate system. S4. Use the data obtained in S3 to complete the robot calibration task.
2. The real-time hand-eye calibration method according to claim 1, characterized in that, The rotation matrix of the robot end effector coordinate system relative to the base coordinate system in S12 The specific formula is as follows: 。 3. The real-time hand-eye calibration method according to claim 2, characterized in that, The homogeneous transformation matrix of the robot end effector coordinate system relative to the base coordinate system in S13 The specific formula is as follows: = 。 4. The real-time hand-eye calibration method according to claim 3, characterized in that, S2 specifically includes the following steps: S21. Keep the position of the calibration pattern unchanged, keep the position of the camera relative to the robot's end joint unchanged, capture a photo of the calibration pattern with the camera, and record the 6D pose of the base on the robot teach pendant relative to the robot's end joint. S22. Control the robot to move randomly multiple times in space, collect multiple calibration pattern photos taken in different poses, and record the 6D pose of the base on the robot teach pendant relative to the robot end effector for each movement. S23. Based on the images captured by the camera during each movement and the data recorded in S21 and S22, calculate the homogeneous transformation matrix of the base coordinate system relative to the robot end effector coordinate system and the homogeneous transformation matrix of the camera coordinate system relative to the calibration pattern coordinate system during each movement. Solve using the least squares method to obtain the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system. .
5. The real-time hand-eye calibration method according to claim 4, characterized in that, S23 specifically includes the following steps: S231. Based on the data obtained in S16, the homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system obtained during each movement is defined as follows: , , ; S232, Based on the data obtained in S231 and = Solve for the homogeneous transformation matrices of the camera coordinate system relative to the calibration pattern coordinate system as follows: , , ; Based on the data obtained in S13 and the teach pendant data recorded during each movement, the homogeneous transformation matrices of the robot's end effector relative to the base coordinate system are calculated as follows: , , ; S234, Based on the data obtained in S233 and = The homogeneous transformation matrices of the base coordinate system relative to the robot end effector coordinate system are obtained as follows: , , ; S235. Based on the data obtained from S232 and S234, and using the homogeneous transformation matrix of the camera coordinate system relative to the robot end effector coordinate system... Unchanged, that is = * * Without changing the coordinates, we list the matrix equations and solve them using the least squares method to obtain the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system. .
6. The real-time hand-eye calibration method according to claim 5, characterized in that, The matrix equation in S235 is as follows: * * = * * = * * 。 7. The real-time hand-eye calibration method according to claim 6, characterized in that, S3 specifically includes the following steps: S31. Control the robot to move again, and the camera will capture a photo of the calibration pattern to obtain the 6D pose of the camera coordinate system relative to the calibration pattern coordinate system. The initial position of the camera can be changed during this process. S32. If the camera is mounted on the robot's end arm, i.e., the eye is on the hand, then proceed to S33. If the camera is mounted outside the robot's end arm, i.e., the eye is outside the hand, then proceed to S34. S33. According to S14 and S17, we know that... = , = And based on the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system obtained in S2... The homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system is calculated using data acquired by the camera and data from the robot's end effector. Homogeneous transformation matrix of robot end-effector coordinate system relative to base coordinate system And complete the real-time calibration of the relationship between the robot's end-effector coordinate system and the camera coordinate system. * * ; S34, According to S17, we know = And based on the homogeneous transformation matrix of the calibration pattern coordinate system relative to the base coordinate system obtained in S2 And using the 6D pose of the camera coordinate system relative to the calibration pattern coordinate system obtained in S31, the homogeneous transformation matrix of the calibration pattern coordinate system relative to the camera coordinate system is calculated. And complete the real-time calibration of the relationship between the robot base coordinate system and the camera coordinate system. = * .
8. The real-time hand-eye calibration method according to claim 7, characterized in that, S4 specifically includes the following steps: S41. If the camera is mounted on the robot's end arm, i.e., the eye is on the hand, then proceed to S42. If the camera is mounted outside the robot's end arm, i.e., the eye is outside the hand, then proceed to S43. S42, based on S33 * * And based on the relationship between the robot's end effector and the base Then use a camera to obtain the three-dimensional coordinates of the target point. Find the coordinates of the target point in the base coordinate system. = This completes the entire calibration task. S43. Without changing the camera position, use the homogeneous transformation matrix between the robot base coordinate system and the camera coordinate system obtained in S35. Multiply the target point by the camera to obtain its 3D coordinates. The coordinates of the target point in the base coordinate system can be obtained. = * This completes the entire calibration task.