Robot hand-eye calibration method based on radar camera joint calibration

Abstract

The invention discloses a robot hand-eye calibration method based on radar camera joint calibration. The robot hand-eye calibration method comprises the following steps that a mechanical arm and a depth camera are installed on a robot; chessboard information in a color two-dimensional image collected by a camera is extracted, coordinate points of a chessboard in the image are detected according to a semantic segmentation method, and then the camera is calibrated; according to the calculated camera parameters, radar data and image data, 3D-2D point pairs of the point cloud data of the circle center and the image data are obtained; optimizing camera parameters according to the joint calibration of the radar and the camera; and carrying out camera and mechanical arm hand-eye calibration according to the camera installation mode. The single checkerboard calibration plate is adopted to calibrate the camera, and the checkerboard in the calibration plate is provided with four round holes, so that the edge of the point cloud, angular points in the image and other depth discontinuous characteristics can be fully utilized during detection, and the obtained solution is more accurate and has higher robustness.

Description

Technical Field

[0001] The present invention belongs to the field of robots, and particularly relates to a robot hand-eye calibration method based on radar-camera joint calibration. Background Art

[0002] In the overall system design of autonomous driving or robots, it is necessary to process data from multiple sensors simultaneously. Among numerous sensor combinations, the combination of lidar and camera is one of the combinations in environmental perception devices. However, when fusing data from different sensors, the difficulty lies in how to accurately determine the internal parameters of the camera and the rigid body transformation between the two sensor coordinate systems through external calibration.

[0003] When calibrating the internal parameters of the existing camera, it is usually based on the pinhole model. However, in the camera projection process, it does not exactly correspond to the pinhole model, and there is no absolute optical center point in the pinhole model between the two. Therefore, the camera internal parameter calibration process is an approximate measurement based on the pinhole model. Moreover, due to the defects of the camera structure and the uncertainty in the optimization of the non-linear function used in the approximate measurement process, the obtained solution is usually not the optimal solution. Summary of the Invention

[0004] The technical problem of the present invention is to provide a robot hand-eye calibration method based on a radar and camera joint calibration method, so that the robot can obtain accurate coordinate mapping when it is not on a horizontal plane, and at the same time improve the accuracy and robustness of lidar and camera data fusion in the intelligent robot system.

[0005] The technical solution of the present invention is a robot hand-eye calibration method based on radar-camera joint calibration, including the following steps: S1: Install a robotic arm and a depth camera on the robot; S2: Extract the checkerboard information in the collected color two-dimensional image by the camera, and detect the coordinate points of the checkerboard in the image according to the semantic segmentation method, and then calibrate the camera; S3: According to the calculated camera parameters, radar data and image data, obtain the 3D-2D point pairs of the point cloud data and the image data of the circle center; S4: Optimize the camera parameters according to the joint calibration of the radar and the camera; S5: Perform a hand-eye calibration method for the camera and the robotic arm according to the camera installation method.

[0006] Preferably, in step S1, the depth camera is integrated with the radar and the camera, and is installed on a stable base; the calibration board is placed within the camera's field of view to calibrate the collected data and align the point cloud data and pixels in all image areas.

[0007] Preferably, in step S2, the camera is calibrated, including calibrating the camera based on the checkerboard corner points to obtain the camera internal parameters. The expression of the camera internal parameters is: ; In the formula, and represent the camera focal length, and represent the offsets of the camera optical axis on the image coordinate system.

[0008] Further, step S3 includes the following sub-steps: S31: Extract the radar data of the circular hole by returning the three-dimensional pose of the target through the radar sensor; S32: Calculate the mapped circular center point according to the camera internal parameters, distortion factors, and camera external parameters obtained by calibrating based on the checkerboard corner points; S33: Calculate the three-dimensional and two-dimensional points of the circular center based on the synchronized lidar data and image data, and select the coordinates of a preset number of point pairs to calculate and calibrate the external parameters of the lidar and the camera.

[0009] Further, step S31 includes the following sub-steps: S311: Based on the radar point cloud data scanned by the radar , use RANSAC plane fitting with direction constraints to segment the calibration target board from the point cloud data ; ; S312: Generate a point cloud mask with the same geometric structure as the calibration target , and obtain the most matching point cloud mask and the target board in the radar coordinate system according to the grid search method.

[0010] Preferably, step S4 includes multiple groups of 3D-2D point pairs of the center of the circle on the calibration board, and optimizes the internal and external parameters of the camera through constraint conditions to align the lidar point cloud data and the camera image data of the calibration board. The calculation formula of the constraint conditions is: ; In the formula, , respectively represent the external parameters from the radar coordinate system to the camera coordinate system, represents the pixel coordinates of the circular center point calculated through the checkerboard before joint optimization, represents the three-dimensional point cloud coordinates of the circular center point in the radar coordinate system, and i represents the counting unit.

[0011] Preferably, step S4 also includes constraining the internal parameters of the camera using the obtained checkerboard data. The calculation formula of the internal parameter constraint is: ; In the formula, represents the three-dimensional point the pixel coordinate point after projection, represents the pixel point where the checkerboard corner point is detected.

[0012] Preferably, step S4 further includes the constraint on the center point of the circle. The two-dimensional point of the center of the circle calculated according to the size of the calibration board, and the calculation formula of the constraint is: ; In the formula, represents the external parameters from the calibration board to the camera.

[0013] Preferably, in step S43, it further includes using the angular axis rotation vector to represent the rotation matrix and solving the optimization equation in the calibration process through the Ceres solver.

[0014] Furthermore, step S5 includes the following sub-steps: S51: According to the pose information of the end of the robotic arm in the base coordinate system collected by the depth camera and the pose information of the calibration board relative to the camera, marked as: ; ; In the formula, R represents the rotation transformation matrix of the collected robotic arm and calibration board, t represents its translation transformation matrix, c is the matrix data flag of the calibration board relative to the camera, eb represents the matrix data of the end of the robotic arm in the base coordinate system for distinction, and m represents the number of acquisitions; S52: Use the Tasi two-step method for hand-eye calibration to obtain the hand-eye calibration matrix. First, solve the external parameters through S51, and then solve the internal parameters. On the basis of obtaining the external parameters, further solve the transformation matrix, that is, find the solution of the equation The calculation formulas of A and B are: ; ; In the formula, gc represents a series of pose information of the calibration board relative to the camera, gb represents a series of pose information of the end of the robotic arm, A represents the transformation matrix between camera coordinate systems for each two acquisitions, B represents the transformation matrix between robotic arm coordinate systems for each two acquisitions, and a unique solution X can be obtained through multiple groups of A and B data.

[0015] Compared with the prior art, the beneficial effects of the present invention include: 1) The present invention calibrates the camera using a single checkerboard calibration board. There are also 4 round holes in the checkerboard of the calibration board, enabling the full utilization of the depth discontinuity features such as the edges of the point cloud and the corner points in the image during detection, and the obtained solution is more accurate and robust.

[0016] 2) The present invention forms a point pair consisting of the center radar three-dimensional points extracted and the calculated two-dimensional circle center points. After alignment, the relative positions between the round holes can be fully utilized, improving the accuracy and robustness of the data fusion between the lidar and the camera in the autonomous driving system or the intelligent robot system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be further described below in conjunction with the drawings and embodiments.

[0018] Figure 1 It is a schematic diagram of the robot hand-eye calibration process based on the combined calibration of radar and camera according to the embodiment of the present invention; Figure 2 It is a calibration board diagram of the robot hand-eye calibration based on the combined calibration of radar and camera according to the embodiment of the present invention; Figure 3 It is a schematic diagram of the radar data of the calibration board for the robot hand-eye calibration based on the combined calibration of radar and camera according to the embodiment of the present invention; Figure 4 It is a schematic diagram of the relationship between the devices for the robot hand-eye calibration based on the combined calibration of radar and camera according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] As Figure 1 shown, a robot hand-eye calibration method based on the combined calibration of radar and camera includes the following steps: S1: Install a robotic arm and a depth camera on the robot.

[0020] In step S1, the depth camera is integrated with the radar and the camera and is installed on a stable base; the calibration board is placed within the camera's field of view to calibrate the collected data and align the point cloud data and pixels in all image areas.

[0021] S2: Extract the checkerboard information in the collected color two-dimensional image through the camera, and detect the coordinate points of the checkerboard in the image according to the semantic segmentation method, and then calibrate the camera.

[0022] In step S2, calibrating the camera includes calibrating the camera based on the checkerboard corner points to obtain the camera internal parameters. The expression of the camera internal parameters is: ; In the formula, , represent the camera focal lengths, , Represents the offset of the camera optical axis in the image coordinate system.

[0023] S3: Based on the calculated camera parameters, radar data, and image data, obtain the point cloud data of the circle center and the 3D-2D point pairs of the image data.

[0024] Step S3 includes the following sub-steps: S31: Extract the radar data of the circular hole by returning the three-dimensional pose of the target through the radar sensor.

[0025] Step S31 includes the following sub-steps: S311: Based on the radar point cloud data scanned by the radar , use RANSAC plane fitting with direction constraints to segment the calibration target board from the point cloud data . .

[0026] S312: Generate a point cloud mask with the same geometric structure as the calibration target , and obtain the most matching point cloud mask and target board in the radar coordinate system according to the grid search method and target board .

[0027] S32: Calculate the mapped circle center point according to the camera internal parameters, distortion factors, and camera external parameters obtained by calibrating the checkerboard corner points.

[0028] S33: Based on the synchronized lidar data and image data, calculate the three-dimensional and two-dimensional points of the circle center, and select the coordinates of a preset number of point pairs to calculate the external parameters of the lidar and the camera for external parameter calibration.

[0029] S4: Optimize the camera parameters according to the joint calibration of the radar and the camera.

[0030] Step S4 includes multiple groups of 3D-2D point pairs of the circle center on the calibration board, and optimizes the internal and external parameters of the camera through constraint conditions to align the lidar point cloud data and camera image data of the calibration board. The calculation formula of the constraint conditions is: ; In the formula, , respectively represent the external parameters from the radar coordinate system to the camera coordinate system, represents the pixel coordinates of the circle center point calculated by the checkerboard before joint optimization, represents the three-dimensional point cloud coordinates of the circle center point in the radar coordinate system, and i represents the counting unit.

[0031] Step S4 also includes using the obtained checkerboard data to constrain the internal parameters of the camera. The calculation formula of the internal parameter constraint is: ; In the formula, represents a three-dimensional point which is the pixel coordinate point after projection, and

[0032] Step S4 also includes the constraint on the center point of the circle. The two-dimensional point of the center of the circle calculated according to the calibration plate size, the calculation formula of the constraint is: ; In the formula, represents the external parameters from the calibration plate to the camera.

[0033] The constraint on the center point of the circle also includes using the angular axis rotation vector to represent the rotation matrix and solving the optimization equation in the calibration process through the Ceres solver.

[0034] S5: A hand-eye calibration method for the camera and the robotic arm according to the camera installation method.

[0035] Furthermore, step S5 includes the following sub-steps: S51: According to the pose information of the end of the robotic arm in the base coordinate system collected by the depth camera and the pose information of the calibration plate relative to the camera, which is marked as: ; ; In the formula, R represents the rotation transformation matrix of the collected robotic arm and the calibration plate, t represents its translation transformation matrix, c represents the matrix data flag of the calibration plate relative to the camera, eb represents the matrix data of the end of the robotic arm in the base coordinate system, m and represents the acquisition quantity. S52: Use the Tasi two-step method to perform hand-eye calibration to obtain the hand-eye calibration matrix. First, solve the external parameters through S51, and then solve the internal parameters. On the basis of obtaining the external parameters, further solve the transformation matrix, that is, find the solution of the equation ; ; In the formula, gc represents a series of pose information of the calibration plate relative to the camera, gb represents a series of pose information of the end of the robotic arm, A represents the transformation matrix between the camera coordinate systems for each two acquisitions, B represents the transformation matrix between the robotic arm coordinate systems for each two acquisitions, and a unique solution X can be obtained through multiple groups of A and B data.

[0036] The present invention was tested on a vision system constructed using the Windows system and a Kinect2 depth camera. The Kinect2 itself has integrated an RGB camera and a radar. A calibration board was printed on A4 paper. The specifications of the calibration board are that the side length of each small grid and the diameter of the round hole are 27 mm, and the number of internal points of the checkerboard is 4×6, as shown in the attached drawing Figure 2 . By fixing the depth camera and moving the calibration board to collect calibration data, after collecting multiple groups of data, 7 groups of data were selected for testing, including: 7 RGB images and 7 depth data at corresponding positions.

[0037] First, the collected data was calibrated using the checkerboard calibration method to obtain the internal parameter matrix and distortion coefficients: ; Distortion coefficients: [[-5.043e-01, 1.634e+00, 7.317e-02, -2.287e-04, -4.512e+00]]; The external parameters are (i = 1, 2... 7), where R represents the rotation transformation matrix of the collected calibration board relative to the Kinect2 depth camera, and t represents the translation transformation matrix of the calibration board relative to the Kinect2 depth camera; (i = 1, 2... 7), where R here represents the rotation transformation matrix of the robotic arm relative to the Kinect2 depth camera, and t represents the translation transformation matrix of the robotic arm relative to the Kinect2 depth camera; It can be seen from this that and represent 1×7 matrices.

[0038] The obtained radar data is not convenient to be represented by data, Figure 3 is an example diagram of the calibration board radar data during testing.

[0039] After calculation and comparison, the average reprojection error decreased from 0.416 to 0.253. What is proposed in the present invention can be used on different depth cameras and robotic arms, and at the same time utilizes depth data and image data, reducing the error influence caused by inaccurate camera internal parameters. The above also proves the applicability of this solution.

Claims

1. A robot hand-eye calibration method based on radar camera joint calibration, characterized in that: The following steps are involved: S1: Install the robotic arm and depth camera on the robot; S2: Extract chessboard information from the color 2D image collected by the camera, detect the coordinate points of the chessboard in the image based on the semantic segmentation method, and then calibrate the camera; S3: Obtain the point cloud data of the circle center and the 3D-2D point pair of the image data according to the calculated camera parameters, radar data and image data; S4: Optimize camera parameters based on joint calibration of radar and camera; S5: Hand-eye calibration method for camera and robotic arm based on camera installation method.

2. According to claim 1, a robot hand-eye calibration method based on radar camera joint calibration is characterized in that: In step S1, the depth camera is an integration of radar and camera and is mounted on a stable base; a calibration plate is placed within the camera's field of view to calibrate the collected data and align point cloud data and pixels in all image areas.

3. The robot hand-eye calibration method based on radar camera joint calibration according to claim 1, characterized in that: In the step S2, the camera is calibrated, including calibrating the camera according to the checkerboard corner points to obtain the camera intrinsic parameters, and the expression of the camera intrinsic parameters is: ； In the formula, , represents the focal length of the camera, , Indicates the offset of the camera optical axis in the image coordinate system.

4. The robot hand-eye calibration method based on radar camera joint calibration according to claim 1, characterized in that: The step S3 includes the following sub-steps: S31: extracting the radar data of the circular hole through the three-dimensional pose of the target returned by the radar sensor; S32: Calculate the center point of the mapped circle according to the camera intrinsic parameters, distortion factors and camera extrinsic parameters obtained by the chessboard corner point calibration; S33: Based on the synchronized lidar data and image data, calculate the three-dimensional point and the two-dimensional point of the circle center, and select a preset number of point pairs to calculate the external parameters of the lidar and camera for external parameter calibration.

5. The robot hand-eye calibration method based on radar camera joint calibration according to claim 1, characterized in that: The step S31 includes the following sub-steps: S311: Based on the radar point cloud data scanned by the radar , using RANSAC plane fitting with orientation constraints from point cloud data Separate the calibration target plate ; S312: Generate a point cloud mask with the same geometry as the calibration target , and obtain the best matching point cloud mask in the radar coordinate system according to the grid search method and target board .

6. The robot hand-eye calibration method based on radar camera joint calibration according to claim 1, characterized in that: The step S4 includes optimizing the intrinsic and extrinsic parameters of the camera through constraints for multiple groups of 3D-2D point pairs at the center of the circle on the calibration plate to align the radar point cloud data of the calibration plate and the camera image data. The constraint condition is calculated as follows: ； In the formula, , They represent the external parameters from the radar coordinate system to the camera coordinate system, represents the pixel coordinates of the center point of the circle calculated by the chessboard before joint optimization, Represents the three-dimensional point cloud coordinates of the center point of the circle in the radar coordinate system, and i represents the counting unit.

7. The robot hand-eye calibration method based on radar camera joint calibration according to claim 1, characterized in that: The step S4 also includes using the acquired chessboard data to constrain the camera's internal parameters, and the calculation formula of the internal parameter constraint is: ； In the formula, Representing 3D points The pixel coordinates after projection, Indicates the pixel points where the chessboard corners are detected.

8. The robot hand-eye calibration method based on radar camera joint calibration according to claim 1, characterized in that: The step S4 also includes a constraint on the center point of the circle, and the two-dimensional point of the center of the circle is calculated based on the size of the calibration plate. The constraint calculation formula is: ； In the formula, Represents the external parameters from the calibration plate to the camera.

9. The robot hand-eye calibration method based on radar camera joint calibration according to claim 8, characterized in that: The constraint on the center point of the circle also includes using an angular axis rotation vector to represent a rotation matrix, and solving an optimization equation in the calibration process using a Ceres solver.

10. The robot hand-eye calibration method based on radar camera joint calibration according to claim 1, characterized in that: Step S5 includes the following sub-steps: S51: The pose information of the end of the robotic arm in the base coordinate system and the pose information of the calibration plate relative to the camera are collected by the depth camera and marked as: ; ; In the formula, R represents the collected rotation transformation matrix of the manipulator and calibration plate, t represents its translation transformation matrix, c is the matrix data mark of the calibration plate relative to the camera, eb represents the matrix data of the end of the manipulator in the base coordinate system for distinction, and m represents the number of acquisitions; S52: Use Tasi two-step method to calibrate the hand-eye to obtain the hand-eye calibration matrix. First solve the external parameters through S51, then solve the internal parameters, and then further solve the transformation matrix based on the external parameters, that is, solve the equation The solution of A and B is: ； ； Where gc represents a series of pose information of the calibration plate relative to the camera, gb represents a series of pose information of the end of the robotic arm, A represents the transformation matrix between the camera coordinate system every two acquisitions, and B represents the transformation matrix between the robotic arm coordinate system every two acquisitions.

Images