Automatic hand-eye calibration method
By aligning the optical axis of the camera system with the center of the calibration plate, and combining inverse kinematics solutions and automatic image filtering, the problem of time-consuming hand-eye calibration is solved, achieving an efficient and accurate calibration process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGDONG UNIV OF TECH
- Filing Date
- 2026-04-07
- Publication Date
- 2026-05-12
Smart Images
Figure CN122008247A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robotics and industrial automation, and more specifically, to an automated hand-eye calibration method. Background Technology
[0002] Most existing hand-eye calibration methods rely on a calibration board, performing calibration by solving high-dimensional nonlinear matrix equations such as AX = XB. Here, A represents the transformation relationship of the robot's end effector relative to the base coordinate system, B represents the transformation relationship of the camera relative to the calibration board coordinate system, and X represents the desired pose of the camera relative to the robot's end effector or the robot's base. To solve for the hand-eye relationship, the robot needs to acquire multiple (typically 15-20) high-quality calibration images in different poses, ensuring that the feature points of the calibration images cover the entire image area. Therefore, the calibration process is time-consuming. Summary of the Invention
[0003] The purpose of this invention is to overcome the shortcomings of the existing technology in hand-eye calibration, which is time-consuming, and to provide an automated hand-eye calibration method that can improve the efficiency of hand-eye calibration and shorten the calibration time.
[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: An automated hand-eye calibration method is provided, comprising the following steps: S1: Install the camera system onto the robot's end effector and adjust its position to ensure that there is no interference between the camera system and the robot's end effector; S2: Place the calibration plate in the robot's workspace and ensure that the calibration plate is in a position that the robot can observe from all sides. Move the robot's end effector to directly above the calibration plate so that the normal vector of the robot's end flange is perpendicular to the plane of the calibration plate. Adjust the position of the robot's end effector up and down so that the calibration plate fills the field of view of the camera system. S3: Generate a series of pose points so that the optical axis of the camera system points to the center area of the calibration board, perform path planning on the pose points and drive the robot end effector to move to each pose point, control the camera system to capture images of the calibration board, and record the robot pose at this time. S4: Using multiple sets of calibration board images captured, calibrate the intrinsic and extrinsic parameters of the camera system and complete the image registration of the camera system; S5: Based on the recorded robot pose data and the corresponding calibration board image, establish and solve the hand-eye calibration equation to obtain the pose transformation relationship between the camera system and the robot end effector, and complete the calibration of the robot hand-eye system.
[0005] The automated hand-eye calibration method of the present invention improves the success rate of the calibration board appearing completely within the field of view of the camera system by ensuring that the optical axis of the camera system always points to the center area of the calibration board, reduces the number of invalid images, improves calibration efficiency, and shortens calibration time.
[0006] Furthermore, in step S1, when installing the camera system onto the robot end effector, the position of the camera system is adjusted so that the optical axis of the camera system is aligned with the positive Z-axis of the robot end effector coordinate system, i.e., the TCP coordinate system. In step S2, adjust the position of the robot's end effector so that the normal vector of the robot's end flange points to the center of the calibration plate. Adjust the position of the robot's end effector vertically until the calibration plate fills the field of view of the camera system. Record the coordinates of the robot's end effector in the base coordinate system at this time as P. t =( x t , y t , z t The coordinates of the center point of the calibration board in the robot's base coordinate system are P. o =( x t , y t , z t - r 0), Measure the vertical distance from the robot's end effector to the center point of the calibration board. Establish a spherical coordinate system with the center point of the calibration plate as the origin. ,by Create a virtual hemisphere around the calibration plate with radius [radius]. In step S3, the method for generating pose points that make the optical axis of the camera system point to the center region of the calibration board is as follows: randomly sample points on the virtual hemisphere to generate a series of spherical pose points pointing to the center of the calibration board, perform path planning on the spherical pose points and drive the robot end effector to move to each pose point, control the camera system to capture images of the calibration board, and record the robot pose at this time.
[0007] By aligning the optical axis of the camera system with the positive Z-axis of the TCP coordinate system and driving the robot end effector to move to a spherical pose point pointing towards the center of the calibration board, the optical axis of the camera system can approximately point towards the center area of the calibration board as the robot end effector moves along the virtual hemisphere. This improves the success rate of the calibration board appearing completely within the field of view of the camera system, reduces the number of invalid images, and improves calibration efficiency.
[0008] Furthermore, in step S3, the process of generating a series of pose points that make the optical axis of the camera system point to the center region of the calibration board includes the following steps: S31: Given a point in a spherical coordinate system Points in the Cartesian coordinate system The conversion relationship is as follows:
[0009] Based on the translation relationship between the coordinate systems, any point on the virtual hemisphere surface... The expression in the robot's base coordinate system is:
[0010] At this point, the position of any point P on the sphere in the robot's base coordinate system is obtained. ; S32: Calculate the vector pointing from the spherical point to the center point of the calibration plate in the base coordinate system. :
[0011] vector n The unit vector is used as the Z-axis direction of the end effector's orientation during robot movement:
[0012] vector n Y-axis component for: (3)
[0013] in, Let Z be the unit direction vector of the robot's base coordinate system; finally, the direction vector of the X-axis can be obtained by the right-hand rule. :
[0014]
[0015] Finally, the robot's end-effector posture, which aligns the camera's optical axis with the center region of the calibration plate, is obtained. ; S33: For points on the virtual sphere By randomly sampling and substituting the coordinates into the robot's end effector posture, a series of pose points can be obtained that make the optical axis of the camera system point to the center area of the calibration board.
[0016] Furthermore, in step S33, for the radius in A series of points on a virtual sphere within the range Random sampling increases image richness and improves calibration accuracy.
[0017] Furthermore, in step S3, the path planning process for the spherical points is as follows: Solve for the robot's inverse kinematics solution for the spherical points in the robot's base coordinate system, and plan a continuous robot motion trajectory based on the inverse kinematics solution. Continuous motion trajectory planning is only performed for pose points with valid inverse kinematics solutions. This increases the validity judgment of the robot's inverse kinematics solution, eliminating invalid pose points without valid solutions, and preventing the robot from attempting to move its end effector to unreachable poses, thus avoiding motion failures and jamming. Driving the robot's end effector smoothly to each valid pose point according to the planned continuous trajectory reduces the number of robot starts and stops, improves the robot's end effector movement efficiency, and shortens the calibration image acquisition time.
[0018] Furthermore, in step S3, after capturing the calibration board image, it is first checked whether the calibration board is successfully recognized. If the recognition is successful, the robot pose and the calibration board pose are recorded. If the recognition fails, the image is captured again until a preset number of valid calibration board images are collected. After the robot's end effector moves to the target pose point, it controls the camera system to capture an image of the calibration board. After capturing the image, the image is first processed for calibration board recognition and detection to determine whether the calibration board is completely visible in the field of view and whether the corner points are clear and extractable. If the recognition is successful, it is determined to be a valid image, and the robot's end effector pose and the calibration board pose at the corresponding position are recorded simultaneously. If the recognition fails, it is determined to be an invalid image, and no data is recorded. This process is repeated until a preset number of valid calibration board images are collected. This ensures that all images used for subsequent calibration are valid images, avoiding invalid images from participating in parameter calibration and equation solving, thus improving calibration accuracy. It eliminates the need for manual screening and judgment of image validity, automating the calibration process and avoiding subjective errors from manual screening. The method of re-aligning and capturing images can supplement the number of valid images, ensuring the required sample size for calibration and avoiding problems such as low accuracy and poor stability of calibration results due to insufficient valid images.
[0019] Furthermore, the randomly sampled spherical points are first filtered by height, and a height coefficient is set. c Only the z-axis coordinate in the robot's base coordinate system is selected. z s >c· r The height coefficient is then used to solve the inverse kinematics problem for the spherical point with a height of 0. c =0.6~0.9. Before performing inverse kinematics calculations on the randomly sampled spherical points, the Z-axis coordinates of each spherical point in the robot's base coordinate system are extracted. z s ;judge z s Does it meet the requirements? z s >c· rThe condition of 0 is used to select only spherical points that meet the height condition for subsequent inverse kinematics solution and trajectory planning, and eliminate pose points that do not meet the height condition. This improves the success rate of robot inverse kinematics solution, reduces invalid calculations, reduces computer load, and improves calibration efficiency. c A value of 0.6 to 0.9 can simultaneously ensure the effectiveness and variation of the pose, while balancing solution efficiency and calibration accuracy.
[0020] Furthermore, the camera system includes an RGB industrial camera and a 3D camera. Using multiple sets of calibration board images, the intrinsic and extrinsic parameters of the RGB industrial camera and the 3D camera are calibrated, and image registration is completed. The rigid body transformation matrix between the RGB industrial camera and the 3D camera is then calculated, completing the calibration between the two cameras. Using multiple sets of valid calibration board images, the intrinsic and extrinsic parameters of the RGB industrial camera and the 3D camera are independently calibrated to reduce the imaging errors of each camera. Image registration is performed on the color image output by the RGB industrial camera and the depth image or point cloud data output by the 3D camera, unifying them in the same coordinate system. Finally, the rigid body transformation matrix between the RGB industrial camera and the 3D camera is calculated to complete the pose relationship calibration between the two cameras, achieving collaborative calibration. The RGB camera captures visual information such as the color and texture of objects, while the 3D camera captures information such as the depth, surface curvature, and spatial pose of objects. The two cameras work together, enabling the hand-eye system to meet the operational needs of complex industrial scenarios.
[0021] Furthermore, when calibrating the intrinsic and extrinsic parameters of the RGB industrial camera and the 3D camera, image distortion correction for the RGB industrial camera and point cloud correction for the 3D camera are performed simultaneously. While calibrating the intrinsic and extrinsic parameters of the RGB industrial camera using the calibration board image, image distortion correction is simultaneously performed on the color images acquired by the RGB industrial camera to eliminate barrel and pincushion distortions caused by the optical characteristics of the camera lens. Simultaneously, while calibrating the intrinsic and extrinsic parameters of the 3D camera, point cloud correction is simultaneously performed on the depth images or point cloud data acquired by the 3D camera to eliminate errors such as point cloud offset and noise caused by the 3D camera's ranging principle and environmental interference. This improves the accuracy of the image data acquired by both cameras, reduces calibration errors, and improves the accuracy of subsequent dual-camera registration and hand-eye calibration. The corrected images and point cloud data are more closely aligned with the actual scene, enabling more accurate subsequent image registration and feature extraction, and avoiding feature matching failures caused by errors in the original data.
[0022] Further, in step S4, the image registration is completed by: extracting the corner points of the calibration plate, and registering the color image output by the RGB industrial camera with the depth image or point cloud data output by the 3D camera using the corner points; the process of solving the rigid body transformation matrix between the two cameras is as follows: obtaining the extrinsic parameters of the RGB industrial camera. , extrinsic parameters of any monocular camera of a 3D camera , The rigid body transformation matrices of the RGB camera and the 3D camera are calculated using the following formula to complete the calibration of the RGB camera and the 3D camera: .
[0023] The corner points of the calibration board are extracted using a vision processing library. Using the corner points of the calibration board as the matching reference, the color image output by the RGB industrial camera is accurately registered with the depth image or point cloud data output by the 3D camera. The extrinsic parameters of the RGB industrial camera and the extrinsic parameters of any single eye of the 3D camera are extracted. The rigid body transformation matrix of the RGB camera and the 3D camera is calculated by a preset formula to complete the calibration of the two cameras. The image registration method based on the corner points of the calibration board has high feature recognition and high matching accuracy, achieving high-precision registration of images from two cameras.
[0024] Compared with the prior art, the beneficial effects of the present invention are: The automated hand-eye calibration method of this invention: 1. By ensuring that the optical axis of the camera system always points to the center region of the calibration board, the success rate of the calibration board appearing completely within the field of view of the camera system is improved, the number of invalid images is reduced, calibration efficiency is improved, and calibration time is shortened; 2. Before performing inverse kinematics solution, only those satisfying z are selected. s >c For spherical points with height condition r0, subsequent inverse kinematics solutions and trajectory planning are performed, eliminating pose points that do not meet the height condition, improving the success rate of robot inverse kinematics solutions, reducing invalid calculations, reducing computer load, and improving calibration efficiency. Attached Figure Description
[0025] Figure 1 A schematic diagram of the hand-eye calibration system; Figure 2 This is a schematic diagram of the camera system. Figure 3 Flowchart for automated calibration of camera systems; Figure 4 A flowchart for automatic image acquisition for calibration; Figure 5 A schematic diagram illustrating the motion position of the camera system during the automated image acquisition process; Figure 6 A schematic diagram of a hand-eye calibration system when the robot's end effector moves to multiple different pose points; Figure 7 A model diagram for hand-eye calibration. Detailed Implementation
[0026] The present invention will be further described below with reference to specific embodiments. The accompanying drawings are for illustrative purposes only, representing schematic diagrams rather than actual physical objects, and should not be construed as limiting the scope of this patent. To better illustrate the embodiments of the present invention, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0027] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present patent. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.
[0028] Example 1 like Figures 1 to 7 The first embodiment of the automated hand-eye calibration method of the present invention is shown, which includes the following steps: S1: Install the camera system onto the robot's end effector and adjust its position to ensure that there is no interference between the camera system and the robot's end effector; S2: Place the calibration plate in the robot's workspace and ensure that the calibration plate is in a position that the robot can observe from all sides. Move the robot's end effector to directly above the calibration plate so that the normal vector of the robot's end flange is perpendicular to the plane of the calibration plate. Adjust the position of the robot's end effector up and down so that the calibration plate fills the field of view of the camera system. S3: Generate a series of pose points so that the optical axis of the camera system points to the center area of the calibration board, perform path planning on the pose points and drive the robot end effector to move to each pose point, control the camera system to capture images of the calibration board, and record the robot pose at this time. S4: Using multiple sets of calibration board images captured, calibrate the intrinsic and extrinsic parameters of the camera system and complete the image registration of the camera system; S5: Based on the recorded robot pose data and the corresponding calibration board image, establish and solve the hand-eye calibration equation to obtain the pose transformation relationship between the camera system and the robot end effector, and complete the calibration of the robot hand-eye system.
[0029] The automated hand-eye calibration method of the present invention improves the success rate of the calibration board appearing completely within the field of view of the camera system by ensuring that the optical axis of the camera system always points to the center area of the calibration board, thereby reducing the number of invalid images and improving calibration efficiency.
[0030] In step S1, when installing the camera system onto the robot end effector, adjust the position of the camera system so that the optical axis of the camera system is aligned with the positive Z-axis of the robot end effector coordinate system, i.e., the TCP coordinate system. In step S2, adjust the position of the robot's end effector so that the normal vector of the robot's end flange points to the center of the calibration plate. Adjust the position of the robot's end effector vertically until the calibration plate fills the field of view of the camera system. Record the coordinates of the robot's end effector in the base coordinate system at this time as P. t =( x t , y t , z t The coordinates of the center point of the calibration board in the robot's base coordinate system are P. o =( x t , y t , z t - r 0), Measure the vertical distance from the robot's end effector to the center point of the calibration board. Establish a spherical coordinate system with the center point of the calibration plate as the origin. ,by Create a virtual hemisphere around the calibration plate with radius , such as Figure 1 As shown; In step S3, the method for generating pose points that make the optical axis of the camera system point to the center region of the calibration board is as follows: randomly sample points on the virtual hemisphere to generate a series of spherical pose points pointing to the center of the calibration board, perform path planning on the spherical pose points and drive the robot end effector to move to each pose point, control the camera system to capture images of the calibration board, and record the robot pose at this time.
[0031] By aligning the optical axis of the camera system with the positive Z-axis of the TCP coordinate system and driving the robot end effector to move to a spherical pose point pointing towards the center of the calibration board, the optical axis of the camera system can approximately point towards the center area of the calibration board as the robot end effector moves along the virtual hemisphere. This improves the success rate of the calibration board appearing completely within the field of view of the camera system, reduces the number of invalid images, and improves calibration efficiency.
[0032] Step S3, the process of generating a series of pose points that make the optical axis of the camera system point to the center region of the calibration board, includes the following steps: S31: Given a point in a spherical coordinate system Points in the Cartesian coordinate system The conversion relationship is as follows:
[0033] Based on the translation relationship between the coordinate systems, any point on the virtual hemisphere surface... The expression in the robot's base coordinate system is:
[0034] At this point, the position of any point P on the sphere in the robot's base coordinate system is obtained. ; S32: Calculate the vector pointing from the spherical point to the center point of the calibration plate in the base coordinate system. :
[0035] The unit vector of the vector is used as the Z-axis direction of the end effector's orientation during robot movement:
[0036] vector n Y-axis component for: (3)
[0037] in, Let Z be the unit direction vector of the robot's base coordinate system. Finally, the direction vector of the X-axis can be obtained using the right-hand rule. :
[0038]
[0039] Finally, the robot's end-effector posture, which aligns the camera's optical axis with the center region of the calibration plate, is obtained. ; S33: For points on the virtual sphere By randomly sampling and substituting the coordinates into the robot's end effector posture, a series of pose points can be obtained that make the optical axis of the camera system point to the center area of the calibration board.
[0040] In step S33, for the radius in A series of points on a virtual sphere within the range Random sampling increases image richness and improves calibration accuracy.
[0041] In step S3, the path planning process for the spherical points is as follows: Solve for the robot's inverse kinematics (IK) for the spherical points in the robot's base coordinate system, and plan a continuous robot motion trajectory based on the IK. Continuous trajectory planning is only performed for pose points with valid IK solutions. This increases the validity of the IK solutions and eliminates invalid pose points without valid solutions, preventing the robot from attempting to move its end effector to unreachable poses, which could lead to motion failures or jamming. Driving the robot's end effector smoothly to each valid pose point according to the planned continuous trajectory reduces the number of robot starts and stops, improves the robot's end effector movement efficiency, and shortens the time for acquiring calibration images.
[0042] In step S3, such as Figure 3 As shown, after capturing the calibration board image, the system first checks whether the calibration board has been successfully recognized. If recognition is successful, the robot pose and calibration board pose are recorded; if recognition fails, the image is recaptured until a preset number of valid calibration board images are collected. After the robot's end effector moves to the target pose point, the camera system captures the calibration board image. After capture, the image is first processed to detect the calibration board, determining whether it appears completely in the field of view and whether the corner points are clear and extractable. If recognition is successful, it is considered a valid image, and the robot end effector pose and calibration board pose at the corresponding position are recorded simultaneously. If recognition fails, it is considered an invalid image, and no data is recorded. This process is repeated until a preset number of valid calibration board images are collected. This ensures that all images used for subsequent calibration are valid, avoiding invalid images from participating in parameter calibration and equation solving, thus improving calibration accuracy. It eliminates the need for manual image screening and validity assessment, automating the calibration process and avoiding subjective errors from manual screening. The recapture method supplements the number of valid images, ensuring the required sample size for calibration and avoiding problems such as low accuracy and poor stability in calibration results due to insufficient valid images.
[0043] In step S5, the process of establishing and solving the hand-eye calibration equation is as follows: S51: For any two poses during robot movement, establish the hand-eye calibration equation: ,in, A 1. A 2 represents the pose of the robot's end effector in the base coordinate system. C 1. C 2 represents the pose of the camera system in the calibration board coordinate system. B The desired pose of the camera system in the TCP coordinate system; S52: Decompose the hand-eye calibration equation into rotation equation and translation equation. Using the multiple sets of calibration images acquired in step S3, solve them using the Tsai two-step method to obtain the hand-eye calibration matrix and complete the hand-eye calibration.
[0044] The calibration equation is solved using the classic Tsai two-step method. The algorithm is mature and efficient, and it is suitable for the needs of rapid calibration in industrial settings. The use of multiple calibration images in the solution can effectively offset random errors in the data acquisition process, improve the accuracy and reliability of hand-eye calibration results, and meet the requirements of high-precision operation of industrial robots.
[0045] Example 2 This embodiment is the second embodiment of the automated hand-eye calibration method of the present invention. This embodiment is similar to the first embodiment, except that, as Figure 4 and Figure 5 As shown, the randomly sampled spherical points are first filtered by height, and a height coefficient is set. c Only the Z-axis coordinate in the robot's base coordinate system is selected. z s >c· r The height coefficient is then used to solve the inverse kinematics problem for the spherical point with a height of 0. c =0.6~0.9. Before performing inverse kinematics calculations on the randomly sampled spherical points, the Z-axis coordinates of each spherical point in the robot's base coordinate system are extracted. z s ;judge z s Does it meet the requirements? z s >c· r The condition of 0 is used to select only spherical points that meet the height condition for subsequent inverse kinematics solution and trajectory planning, and eliminate pose points that do not meet the height condition. This improves the success rate of robot inverse kinematics solution, reduces invalid calculations, reduces computer load, and improves calibration efficiency. c A value of 0.6 to 0.9 can simultaneously ensure the validity and variation of the pose, balancing solution efficiency and calibration accuracy. In this embodiment, c =0.8, which best balances solution efficiency and calibration accuracy.
[0046] In this embodiment, the process of automatic acquisition of calibration images is as follows: Position the mobile robot's end effector directly above the calibration plate; based on the position of the robot's end effector and its distance from the calibration plate. r 0. Establish a spherical coordinate system with the center of the calibration plate as the center and a radius of 0. r A virtual hemisphere of 0; randomly sampled zenith angle and azimuth Generate the robot's end-effector pose on the virtual sphere; determine if it satisfies... z s >c· rIf the condition of 0 is met, solve for the inverse kinematics solution; if not, extract the next pose for judgment. If an inverse kinematics solution exists, use it for motion planning of the robot's end effector; if not, extract the next pose for judgment.
[0047] Example 3 This embodiment is the third embodiment of the automated hand-eye calibration method of the present invention. This embodiment is similar to embodiment two, except that, as Figure 2 As shown, the camera system includes an RGB industrial camera and a 3D camera. Using multiple sets of calibration board images, the intrinsic and extrinsic parameters of the RGB industrial camera and the 3D camera are calibrated, and image registration is completed. The rigid body transformation matrix between the RGB industrial camera and the 3D camera is then calculated, completing the calibration between the two cameras. Using multiple sets of valid calibration board images, the intrinsic and extrinsic parameters of the RGB industrial camera and the 3D camera are independently calibrated to reduce the imaging errors of each camera. Image registration is performed on the color image output by the RGB industrial camera and the depth image or point cloud data output by the 3D camera, unifying them in the same coordinate system. Finally, the rigid body transformation matrix between the RGB industrial camera and the 3D camera is calculated, completing the pose relationship calibration between the two cameras and achieving collaborative calibration. The RGB camera captures visual information such as the color and texture of objects, while the 3D camera captures information such as the depth, surface curvature, and spatial pose of objects. The two cameras work together, enabling the hand-eye system to meet the operational needs of complex industrial scenarios.
[0048] When calibrating the intrinsic and extrinsic parameters of the RGB industrial camera and the 3D camera, image distortion correction of the RGB industrial camera and point cloud correction of the 3D camera are performed simultaneously. In this embodiment, the 3D camera can be an RGB-D camera, a binocular 3D camera, a structured light 3D camera, etc. While calibrating the intrinsic and extrinsic parameters of the RGB industrial camera using the calibration board image, image distortion correction is performed simultaneously on the color images acquired by the RGB industrial camera to eliminate barrel and pincushion distortions caused by the optical characteristics of the camera lens. While calibrating the intrinsic and extrinsic parameters of the 3D camera, point cloud correction is performed simultaneously on the depth images or point cloud data acquired by the 3D camera to eliminate errors such as point cloud offset and noise caused by the 3D camera's ranging principle and environmental interference. This improves the accuracy of the image data acquired by the two cameras, reduces calibration errors, and improves the accuracy of subsequent dual-camera registration and hand-eye calibration. The corrected images and point cloud data are more consistent with the actual scene, enabling more accurate subsequent image registration and feature extraction, and avoiding feature matching failures caused by errors in the original data.
[0049] In step S4, the image registration is completed by extracting the corner points of the calibration plate and registering the color image output by the RGB industrial camera with the depth image or point cloud data output by the 3D camera using the corner points. The process of solving the rigid body transformation matrix between the two cameras is as follows: obtaining the extrinsic parameters of the RGB industrial camera. , extrinsic parameters of any monocular camera of a 3D camera , The rigid body transformation matrices of the RGB camera and the 3D camera are calculated using the following formula to complete the calibration of the RGB camera and the 3D camera: .
[0050] The corner points of the calibration board are extracted using a vision processing library. Using these corner points as a matching reference, the color image output by the RGB industrial camera is accurately registered with the depth image or point cloud data output by the 3D camera. The extrinsic parameters of the RGB industrial camera and any single-eye extrinsic parameter of the 3D camera are extracted. The rigid body transformation matrix of the RGB and 3D cameras is calculated using a preset formula, completing the dual-camera calibration. This image registration method based on the calibration board corner points offers high feature recognition and matching accuracy, achieving high-precision registration of dual-camera images. In this embodiment, the OpenCV vision processing library is used.
[0051] The automated hand-eye calibration method in this embodiment includes: Phase 1: Assembling the robot's hand-eye system; S1: Install the end effector on the robot end, install the camera bracket and robot end flange on the end effector, and install the RGB industrial camera and 3D camera parallel to each other on the camera bracket, so that the optical axes of the RGB industrial camera and 3D camera are parallel to the normal of the end flange, and ensure that there is no interference between the camera system and the robot end effector. Phase Two: Automatic acquisition of calibration images; S2: Place the calibration plate in the robot's workspace and ensure that the calibration plate is in a position that the robot can observe from all sides. Move the robot's end effector to directly above the calibration plate so that the normal vector of the robot's end flange is perpendicular to the plane of the calibration plate. Adjust the position of the robot's end effector up and down so that the calibration plate fills the field of view of the camera system. S3: Generate a series of pose points so that the optical axis of the camera system points to the center area of the calibration board, perform path planning on the pose points and drive the robot end effector to move to each pose point, control the camera system to capture images of the calibration board, and record the robot pose at this time. S31: Given a point in a spherical coordinate system Points in the Cartesian coordinate system The conversion relationship is as follows: (1) Based on the translation relationship between the coordinate systems, any point on the virtual hemisphere surface... The expression in the robot's base coordinate system is: (2) At this point, the position of any point P on the sphere in the robot's base coordinate system is obtained. ; S32: For the radius in A series of points on a virtual sphere within the range Random sampling, where , , Substituting the spherical coordinates into equation (2), we obtain a series of position points P on the virtual hemisphere under the base coordinates.
[0052] S33: Filter the location points generated in S32 and select those that meet the requirements. z s >c· r The trajectory of the robot's end effector is planned using the position point 0; S34: Calculate the vector pointing from the spherical point to the center point of the calibration plate in the base coordinate system. : (3) The unit vector of the vector is used as the Z-axis direction of the end effector's orientation during robot movement: (4) vector n Y-axis component for: (5) (6) in, Let Z be the unit direction vector of the robot's base coordinate system; finally, the direction vector of the X-axis can be obtained by the right-hand rule. : (7) (8) Finally, the robot's end-effector posture, which aligns the camera's optical axis with the center region of the calibration plate, is obtained. ; S35: Through steps S31 to S34, a series of pose points can be obtained to ensure that the camera optical axis always points to the center of the calibration plate. ,in Let P be the coordinates of the point on the sphere. ; at the specified pose point Solve the inverse kinematics of the robot and perform path planning for the series of pose points to drive the robot to move; S35: When the end effector of the robot moves to each pose point generated in step S35, control the RGB camera and 3D camera to capture images of the calibration board and save them for solving the calibration relationship; Phase 3: Complete hand-eye calibration; S4: Calibrate RGB and 3D cameras; S41: Using the calibration board image obtained in step S3, use OpenCV to calibrate the intrinsic and extrinsic parameters of the RGB camera and the 3D camera, and perform image distortion correction and point cloud correction. S42: Using OpenCV to extract corner points of the calibration board, register the color image generated by the RGB camera with the depth image or point cloud data generated by the 3D camera, and obtain the RGB camera extrinsic parameters from step S41. , extrinsic parameters of any one of the 3D cameras , The RGB and 3D camera calibrations are completed by calculating the transformation matrices of the RGB and 3D cameras using the following formula: (9) S5: Calibration of camera system and robot end effector; S51: Establish the hand-eye calibration equation; for any two poses during robot movement, the following formula holds: (10) Among them, such as Figure 7 As shown, A 1. A 2 represents the pose of the robot's end effector in the base coordinate system. C 1. C 2 represents the pose of the camera system in the calibration board coordinate system. B Let the pose of the camera system in the TCP coordinate system be the desired pose; according to equation (10), we can obtain: (11) The above formula is a typical example. Problem. Equation (10) above can be written as: (12) (13) Where the rotation matrix Translation vector The hand and eye are used to identify the object to be solved.
[0053] S52: The hand-eye calibration equations, rotation equations, and translation equations are used. For solving equation sets (12) and (13), this invention employs the Tsai two-step method; its main idea is to first calculate the rotation part, then calculate the translation part; when the solution is obtained... back, The following can be obtained from the system of linear equations: (14) Solving for rotational components using axis-angle representation: (15) (16) in: (17) , and The axis of rotation in the axis-angle representation is... This is the modified Rodriguez transform. Rotation angle. for: (18) S53: Using the multiple sets of calibration images acquired in step S3, solve the rotation equation and translation equation to obtain the solution of the hand-eye calibration matrix; The Tsai method requires the robot to generate at least two sets of rigid body transformations and two images taken in the corresponding poses to obtain a numerical solution, and the rotation axes of the two transformations cannot be parallel. Within a certain range, increasing the number of poses acquired can improve the accuracy of the algorithm's calibration results. The robot's end effector pose is output in real time by the controller. To improve calibration accuracy, this invention acquires multiple sets of poses for calculation. Therefore, in actual calculation, equation (15) is written as: (19) The above equation is an overdetermined system of equations. Let the above equation be... Then, by using the least squares method, we can obtain the solution. The rotation axis and rotation angle can be obtained from equations (16) and (18), and thus the rotation matrix can be obtained. Finally, based on equation (13), the least squares method is used to obtain the result. At this point, and This constitutes the pose of the camera relative to the robot's end effector. B The hand-eye relationship was thus determined.
[0054] The physical examples are as follows: The robot used is a UR5 six-DOF articulated robot. The RGB industrial camera selected is the IDS UI-1220LE-CHQ model, and the 3D camera is the IDS N10 binocular 3D camera. Both cameras transmit images via USB, and the robot communicates with the host computer via TCP / IP. Experiments using the automatic hand-eye calibration method have shown excellent calibration accuracy and stability. After acquiring 30 calibration images, the calibration results and accuracy between the 3D camera and the robotic arm are as follows: Table 1 Calibration Results
[0055] Table 2 Calibration Reflection Error
[0056] In the specific implementation of the above embodiments, the technical features can be combined in any non-contradictory way. For the sake of brevity, not all possible combinations of the above technical features are described. However, as long as the combination of these technical features is not contradictory, it should be considered to be within the scope of this specification.
[0057] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. An automated hand-eye calibration method, characterized in that, Includes the following steps: S1: Install the camera system onto the robot's end effector and adjust its position to ensure that there is no interference between the camera system and the robot's end effector; S2: Place the calibration plate in the robot's workspace and ensure that the calibration plate is in a position that the robot can observe from all sides. Move the robot's end effector to directly above the calibration plate so that the normal vector of the robot's end flange is perpendicular to the plane of the calibration plate. Adjust the position of the robot's end effector up and down so that the calibration plate fills the field of view of the camera system. S3: Generate a series of pose points so that the optical axis of the camera system points to the center area of the calibration board, perform path planning on the pose points and drive the robot end effector to move to each pose point, control the camera system to capture images of the calibration board, and record the robot pose at this time. S4: Using multiple sets of calibration board images captured, calibrate the intrinsic and extrinsic parameters of the camera system and complete the image registration of the camera system; S5: Based on the recorded robot pose data and the corresponding calibration board image, establish and solve the hand-eye calibration equation to obtain the pose transformation relationship between the camera system and the robot end effector, and complete the calibration of the robot hand-eye system.
2. The automated hand-eye calibration method according to claim 1, characterized in that, In step S1, when installing the camera system onto the robot end effector, adjust the position of the camera system so that the optical axis of the camera system is aligned with the positive Z-axis of the robot end effector coordinate system, i.e., the TCP coordinate system. In step S2, adjust the position of the robot's end effector so that the normal vector of the robot's end flange points to the center of the calibration plate. Adjust the position of the robot's end effector vertically until the calibration plate fills the field of view of the camera system. Record the coordinates of the robot's end effector in the base coordinate system at this time as P. t =(x t , y t , z t The coordinates of the center point of the calibration board in the robot's base coordinate system are P. o =(x t , y t , z t -r0), measure the vertical distance from the robot's end effector to the center point of the calibration plate. Establish a spherical coordinate system with the center point of the calibration plate as the origin. ,by Create a virtual hemisphere around the calibration plate with radius [radius]. In step S3, the method for generating pose points that make the optical axis of the camera system point to the center region of the calibration board is as follows: randomly sample points on the virtual hemisphere to generate a series of spherical pose points pointing to the center of the calibration board, perform path planning on the spherical pose points and drive the robot end effector to move to each pose point, control the camera system to capture images of the calibration board, and record the robot pose at this time.
3. The automated hand-eye calibration method according to claim 2, characterized in that, Step S3, the process of generating a series of pose points that make the optical axis of the camera system point to the center region of the calibration board, includes the following steps: S31: Given a point in a spherical coordinate system Points in the Cartesian coordinate system The conversion relationship is as follows: Based on the translation relationship between the coordinate systems, any point on the virtual hemisphere surface... The expression in the robot's base coordinate system is: At this point, the position of any point P on the sphere in the robot's base coordinate system is obtained. ; S32: Calculate the vector pointing from the spherical point to the center point of the calibration plate in the base coordinate system. n : vector The unit vector is used as the Z-axis direction of the end effector's orientation during robot movement: vector n Y-axis component for: （3） in, Let Z be the unit direction vector of the robot's base coordinate system; finally, the direction vector of the X-axis can be obtained by the right-hand rule. : Finally, the robot's end-effector posture, which aligns the camera's optical axis with the center region of the calibration plate, is obtained. ; S33: For points on the virtual sphere By randomly sampling and substituting the coordinates into the robot's end effector posture, a series of pose points can be obtained that make the optical axis of the camera system point to the center area of the calibration board.
4. The automated hand-eye calibration method according to claim 3, characterized in that, In step S33, for the radius in A series of points on a virtual sphere within the range Random sampling.
5. The automated hand-eye calibration method according to claim 3, characterized in that, In step S3, the process of path planning for the pose point is as follows: solve the robot inverse kinematics solution for the spherical pose point in the robot base coordinate system, and plan the continuous robot motion trajectory based on the inverse kinematics solution.
6. The automated hand-eye calibration method according to claim 2, characterized in that, In step S3, after capturing the calibration board image, the system first checks whether the calibration board has been successfully recognized. If the recognition is successful, the robot pose and the calibration board pose are recorded. If the recognition fails, the image is captured again until a preset number of valid calibration board images are collected.
7. The automated hand-eye calibration method according to claim 2, characterized in that, In step S3, the height of the randomly sampled spherical points is first filtered, and a height coefficient is set. c Only the Z-axis coordinate in the robot's base coordinate system is selected. z s > c · r The height coefficient is then used to solve the inverse kinematics problem for the spherical point with a height of 0. c =0.6~0.
9.
8. The automated hand-eye calibration method according to claim 1, characterized in that, The camera system includes an RGB industrial camera and a 3D camera. In step S4, the intrinsic and extrinsic parameters of the RGB industrial camera and the 3D camera are calibrated using multiple sets of calibration board images captured, and image registration is completed. The rigid body transformation matrix between the RGB industrial camera and the 3D camera is solved to complete the calibration between the two cameras.
9. The automated hand-eye calibration method according to claim 8, characterized in that, When calibrating the intrinsic and extrinsic parameters of RGB industrial cameras and 3D cameras, image distortion correction of RGB industrial cameras and point cloud correction of 3D cameras are performed simultaneously.
10. The automated hand-eye calibration method according to claim 9, characterized in that, In step S4, the image registration is completed by extracting the corner points of the calibration plate and registering the color image output by the RGB industrial camera with the depth image or point cloud data output by the 3D camera using the corner points. The process of solving the rigid body transformation matrix between the two cameras is as follows: obtaining the extrinsic parameters of the RGB industrial camera. , extrinsic parameters of any monocular camera of a 3D camera , The rigid body transformation matrices of the RGB camera and the 3D camera are calculated using the following formula to complete the calibration of the RGB camera and the 3D camera: .