Laser camera-based robot hand-eye calibration method and device

Abstract

The laser camera-based mechanical arm hand-eye calibration method and device are characterized by the following: a calibration tool is connected to the end arm of the mechanical arm, a laser camera is arranged on the base of the mechanical arm, the calibration tool is provided with a calibration plate with a changeable pose, the laser camera is used to collect intensity maps and point cloud data of the calibration plate, the mechanical arm is kept stationary, the pose of the calibration plate is adjusted, the pose conversion relationship between the laser camera and the end of the mechanical arm is solved based on the point cloud position information in the camera coordinate system and the point cloud position information in the world coordinate system under different poses of the calibration plate, and the hand-eye calibration is completed. The present application reduces a plurality of intermediate processes in the existing calibration process, does not require RGB images and mechanical arm pose information, reduces the introduction error of the mechanical arm and the point cloud data, improves the calibration accuracy, and is especially suitable for live-line working robots, and can quickly complete the hand-eye calibration on the working site, and is safe, fast and reliable.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of robot technology and relates to a hand-eye calibration method and device for a robot. BACKGROUND

[0002] Robots have multi-joint multi-degree-of-freedom mechanical arms, and relevant work is performed by controlling the position of the mechanical arm end. In the prior art, the spatial position of a robot is generally determined by machine vision. In order to establish a relationship between the camera (i.e., the eye of the robot) and the mechanical arm end (i.e., the hand of the robot) coordinate system, the robot mechanical arm and the camera coordinate system need to be calibrated, and this calibration process is also called hand-eye calibration.

[0003] Generally, the hand-eye relationship of a robot is divided into eye-in-hand and eye-to-hand, wherein eye-in-hand is eye on hand, and the vision system of the robot moves with the mechanical arm end; and eye-to-hand is eye on hand, and the vision system of the robot is relatively fixed with the robot base position and does not move in the world coordinate system. In the eye-in-hand relationship, the camera is arranged at the mechanical arm end, and the calibration plate is arranged at a fixed position beside the robot, the position relationship between the robot base and the calibration plate is always unchanged, and the quantity to be solved is the pose relationship between the camera and the mechanical arm end coordinate system. In the eye-to-hand relationship, the calibration plate is arranged at the mechanical arm end, and the camera is arranged at a fixed position beside the robot, the pose relationship between the mechanical arm end and the calibration plate is always unchanged, and the quantity to be solved is the pose relationship between the camera and the robot base coordinate system. Regardless of which calibration method is used, when the pose relationship is solved, the mechanical arm needs to be moved to change the position of the camera or the calibration plate, and then a solving matrix of different coordinate system poses is constructed, which requires a large space to facilitate the movement of the mechanical arm. At the same time, since the above two calibration methods both rely on the movement of the mechanical arm to change the position relationship between the camera and the calibration plate to construct the pose solving matrix, and the pose data of the mechanical arm end effector is acquired as the parameter of the pose solving matrix during the movement of the mechanical arm, the errors of the mechanical arm itself and the errors of the calibration plate itself will be introduced into the calibration solution, which will affect the accuracy of the calibration.

[0004] The common mechanical arm hand-eye calibration in the prior art is completed by means of camera detection of a checkerboard, which needs to utilize an RGB image, which will involve camera intrinsic parameters and introduce camera intrinsic error. There is also a scheme for utilizing a point cloud camera to perform hand-eye calibration, which utilizes an ICP algorithm to complete hand-eye calibration of the 3D point cloud camera without RGB image. However, the ICP method needs to match as many regular objects as possible in a large space, and at the same time, the mechanical arm is also necessarily moved, which will also introduce the self error of the mechanical arm in calibration.

[0005] The hot-line work robot is a special robot for line work, which mainly replaces manual work to complete a series of high-risk operations such as high-voltage cable lapping and dismounting in the air. The working environment is outdoor and often changes. The existing mechanical arm hand-eye calibration method mostly requires the mechanical arm to complete calibration by changing the pose in a relatively stable environment, while the hot-line work robot needs to move frequently. On the one hand, the mechanical arm motion error is prone to occur after long use, and on the other hand, the different working environments may need to update the hand-eye calibration. The working environment of the hot-line work robot is not suitable for the existing calibration method to perform on-site calibration. If on-site calibration is performed, the pose transformation of the mechanical arm may affect the surrounding cables. SUMMARY

[0006] The problem to be solved by the present application is that the prior art needs a large site to complete the calibration work when calibrating the mechanical arm hand-eye, and due to the basic principle of the calibration method, the camera intrinsic parameters or the self error of the mechanical arm is introduced into the calibration solution, which affects the calibration accuracy. Especially for the hot-line work robot, the existing calibration method requires many conditions, which is not conducive to the rapid hand-eye calibration of the hot-line work robot.

[0007] The technical scheme of the present application is: a mechanical arm hand-eye calibration method based on a laser camera, a calibration tool is connected to the end arm of the mechanical arm, a laser camera is arranged on the base of the mechanical arm, a calibration board with variable pose is arranged on the calibration tool, the laser camera is used to collect intensity maps and point cloud data of the calibration board, and the method comprises the following steps:

[0008] 1) The laser camera faces the calibration board to collect point cloud data and intensity maps, and the point cloud position information P in the camera coordinate system and the real point cloud position information Q in the world coordinate system are obtained from the collected point cloud data;

[0009] 2) The feature points of the calibration board are extracted from the obtained intensity maps, three points not in a line are selected as target points, and the point cloud position information P of the corresponding target points is obtained;

[0010] 3) keeping the mechanical arm still, adjusting the pose of the calibration board, the different poses including displacement or rotation in horizontal, vertical and depth directions, the adjustment including at least one displacement in the depth direction, repeating step 1) 2) collecting intensity maps and point cloud data of the calibration board in different poses, and simultaneously solving the obtained point cloud position information P of the target point and the real point cloud position information Q in the corresponding pose to obtain the pose conversion relationship of the laser camera and the end of the mechanical arm, completing the hand-eye calibration.

[0011] Further, solving the pose conversion relationship of the laser camera and the end of the mechanical arm is specifically:

[0012]

[0013] wherein, subscripts m1, m2 represent different poses of the calibration board, M represents the pose conversion relationship of the laser camera and the end of the mechanical arm, represents the point cloud position information in the real world collected at position m1, is the point cloud depth data in the real world, represents the point cloud intensity data in the real world at position m1.

[0014] The pose conversion relationship M is obtained by simultaneously solving the point cloud data of the calibration board in two poses with different depths.

[0015] Further, when step 1) 2) is performed, the point cloud data of the calibration board in the same pose m is collected N times, and the average point cloud information is calculated:

[0016]

[0017] wherein, P m,n represents the point cloud data collected at the nth time in the pose m, P m represents the average point cloud information in the pose m, and the average point cloud information is used as the camera coordinate system point cloud position information of the target point to participate in the solving of the pose conversion relationship.

[0018] Further, the calibration work includes a support and a calibration board, the support includes a connecting piece, an xyz three-way displacement platform, a rotating platform and a calibration board clamping piece, the connecting piece is used to connect the calibration tool on the mechanical arm, the calibration board clamping piece is used to fix the calibration board, and the xyz three-way displacement platform and the rotating platform are used to displace or rotate the calibration board in horizontal, vertical and depth directions.

[0019] Further, the mechanical arm is the mechanical arm of a live-line working robot.

[0020] The application also provides a laser camera-based mechanical arm hand-eye calibration device, comprising a laser camera, a calibration tool and a calibration module, the calibration tool is arranged on the end arm of the mechanical arm, the laser camera is arranged on the base of the mechanical arm, the calibration tool is provided with a calibration plate with a changeable pose, the calibration module is in data connection with the laser camera, the calibration module comprises a memory and a processor, the memory stores a computer program, and the processor realizes the calibration calculation step of the above method when executing the computer program; the calibration module receives the data collected by the laser camera and outputs the pose conversion relationship information for hand-eye calibration.

[0021] The method of the application does not need to move the mechanical arm throughout the whole process, and the calibration relationship can be solved based on the laser point cloud data by collecting a small amount of target position information of the calibration plate at different positions through the hand-eye calibration tool with a changeable pose. The application directly uses the point cloud data without the aid of a color camera for calibration, thereby reducing the need and dependence on the calibration environment, the style of the calibration plate, the mechanical arm pose information, the number of fixed calibration positions in the real physical world, and the mechanical arm and laser camera error information. The application places all the needed position and rotation information on the design of the calibration tool, thereby avoiding the introduction of mechanical arm error information.

[0022] Meanwhile, the application also provides a corresponding solution for the problem that the point clouds at different positions in the same frame of data collected by the laser camera will have errors. The average value is calculated in each dimension through the three-dimensional information of the same point in multiple frames.

[0023] The application reduces multiple intermediate processes in the existing calibration process, does not need RGB images and mechanical arm pose information, reduces the introduction errors of the mechanical arm and the point cloud data, and improves the calibration accuracy. The application is particularly suitable for live-line working robots, does not need any action of the mechanical arm, and does not need to consider the mechanical arm itself error, and can quickly complete the hand-eye calibration in the working site, and is safe, fast and reliable. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 It is a flowchart of the method of the application.

[0025] Figure 2 It is a configuration diagram of the laser camera and the calibration tool on the mechanical arm in the method of the application.

[0026] Figure 3 It is a side view of Figure 2 .

[0027] Figure 4 It is a structure diagram of the calibration tool in the method of the application.

[0028] Figure 5 It is a schematic diagram of two embodiments of the calibration plate in the method of the application. DETAILED DESCRIPTION

[0029] The present application proposes a hand-eye calibration method based on a mechanical arm and a laser camera, aiming to introduce as little error of the mechanical arm as possible, and calibrate the environment without limitation; at the same time, accurate information of the real physical world can also be introduced into the point cloud data.

[0030] As shown in Figure 1 and Figure 2 、 3 The present application connects a calibration tool at the end of the mechanical arm, and sets a laser camera on the base of the mechanical arm, and the calibration tool is provided with a calibration plate with changeable pose, and the laser camera is used to collect intensity map and point cloud data of the calibration plate, including the following steps:

[0031] 1) Collect point cloud data and intensity map by facing the calibration plate with the laser camera. The laser sensor of the laser camera used in the present application can output laser intensity map, which can be converted into a simple gray scale map, without the help of a color camera, reducing the error introduced in the calibration process, i.e. the influence of camera distortion on the calibration process. The point cloud position information P in the camera coordinate system and the real point cloud position information Q in the world coordinate system are obtained from the collected point cloud data. The coordinate systems of P and the real point cloud position information Q are different, and the coordinate origin of the point cloud position information P is at the sensor optical center position; the coordinate point of the real point cloud position information Q is at the zero point of the tool scale and the upper left corner of the calibration plate plane. In addition, the accuracy of the point cloud position information P and the real point cloud position information Q is different, and the point cloud data reflected back by the laser on the plane is uneven; the real world point cloud data itself is a plane.

[0032] 2) Extract the feature points of the calibration plate from the obtained intensity map, select three non-collinear points as target points, and obtain the point cloud position information P of the corresponding target points. The selection of feature points is generally the center of the calibration plate or the corner points. These points are easy to be used as target points for subsequent processing due to their obvious color characteristics, size information, etc.

[0033] 3) keep the robot arm still, adjust the pose of the calibration board, the different poses include displacement or rotation in horizontal, vertical and depth direction, here the horizontal, vertical and depth refer to the plane that the camera faces as the horizontal and vertical plane, and the plane perpendicular to this plane is the depth, the adjustment at least includes displacement in depth direction, repeat steps 1) and 2) to collect the intensity map and point cloud data of the calibration board in different poses, and the obtained point cloud position information P of the target point and the real point cloud position information Q in the corresponding pose are associated, the point cloud information collected in different poses is subtracted, and the pose conversion relationship of the laser camera and the end of the robot arm is solved, and the hand-eye calibration is completed.

[0034] In order to further reduce the influence of the error existing in the point cloud data, the present application subtracts the data between different two frames, and simultaneously takes the data in depth and the data on the point cloud intensity map as accurate information, and finally obtains the method for solving the conversion matrix. The pose conversion relationship of the laser camera and the end of the robot arm is solved by using the point cloud information, and the specific method is as follows:

[0035]

[0036] Wherein, the subscripts m1 and m2 represent different poses of the calibration board, M represents the pose conversion relationship between the laser camera and the end of the robot arm, represents the point cloud position information in the real world collected at position m1, is the point cloud depth data in the real world, represents the point cloud intensity data in the real world at position m1, the pose conversion relationship M is solved by associating the point cloud data of the calibration board in two different poses with different depths.

[0037] According to the above formula, M is solved by the point cloud intensity map and the point cloud depth data.

[0038]

[0039] R refers to a rotation matrix, which is obtained by multiplying the rotation matrices of three coordinate axes. t refers to a translation vector, which refers to the translation distance in three directions.

[0040] Compared with the prior art, the present application does not need to obtain the pose data of the end effector of the mechanical arm, and all the position and rotation information needed is placed on the design of the calibration tool, so that the error information of the mechanical arm is not introduced. The prior art needs to adjust the six-degree-of-freedom pose of the mechanical arm, and the present application only needs to translate at least once in the "depth" direction. The existing eye-to-hand calibration needs to collect data of the same point at different poses as calibration data, and the present application selects three points that are not collinear in each frame, i.e. the target point in the calibration process.

[0041] Further, when performing step 1) 2), the point cloud data of the calibration plate at the same position m is collected N times, and the average point cloud information is calculated:

[0042]

[0043] Where P m,n represents the point cloud data obtained at the nth time of collecting at the pose m, P m represents the average point cloud information at the pose m, and the point cloud position information of the target point is used as the average point cloud information to participate in the solution of the pose conversion relationship. The present application uses laser point cloud data to perform hand-eye calibration, compared with the existing calibration method of collecting RGB images by means of a color camera, without obtaining the camera intrinsic parameter, the calibration process is simplified. However, this may also introduce errors in the point cloud data, so the present application uses the method of calculating the average value to reduce the error influence of the point cloud data. The information of each point in the point cloud is three-dimensional, so the three-dimensional information of the corresponding same point in multiple frames collected by the laser camera is calculated, and the average value is calculated in each dimension.

[0044] As shown in one embodiment of Figure 4 , the calibration work of the present application includes a support and a calibration plate, the support includes a connecting piece, an xyz three-direction displacement platform, a rotating platform and a calibration plate clamping piece, the connecting piece is used to connect the calibration tool on the mechanical arm, the calibration plate clamping piece is used to fix the calibration plate, and the xyz three-direction displacement platform and the rotating platform are used to displace or rotate the calibration plate in the horizontal, vertical and depth directions. Under this structure, the process requirement of the calibration tool is that the number of clamping pieces of the calibration plate is as few as possible, the overall stability is high, and the displacement and rotation adjustment error is small.

[0045] As shown in Figure 5 , the present application does not have special requirements for the form of the calibration plate, and commonly used chessboard or round hole calibration plates can be used.

[0046] The application also provides a laser camera-based mechanical arm hand-eye calibration device, comprising a laser camera, a calibration tool and a calibration module, the calibration tool is arranged on the end arm of the mechanical arm, the laser camera is arranged on the base of the mechanical arm, the calibration tool is provided with a calibration plate with a changeable pose, the calibration module is in data connection with the laser camera, the calibration module comprises a memory and a processor, the memory stores a computer program, and the processor realizes the calibration calculation steps of the above method when executing the computer program; the calibration module receives the data collected by the laser camera and outputs the pose conversion relationship information for hand-eye calibration.

[0047] The whole calibration process of the application does not require mechanical arm error information, sensor error information, and does not require mechanical arm pose information or camera internal parameters, which is the key difference between the application and the prior art. The application controls the error in the designed calibration tool, and only uses the tool with high accuracy in the depth direction to complete the calibration work. There is no special requirement for the calibration environment site. The displacement or rotation accuracy in the depth direction is determined by the accuracy of the hand-eye calibration and the accuracy of the tool that can be designed. For example, in current robot applications, if the accuracy of the tool in the depth direction is 0.1mm, the accuracy of the hand-eye calibration can reach millimeter level. The application is inclined to eye-to-hand calibration mode, and the camera is fixed. However, the existing eye-to-hand calibration technology reads the pose information of the mechanical arm itself to bring out the calibration plate information, which naturally introduces the arm error. The prior art mostly ignores this error. The method of the application isolates the influence of the mechanical arm assembly error and the tool assembly error. The tool assembly error refers to the error caused by the existing tool or calibration plate for hand-eye calibration, sensor distortion and the like on the market. For example, the calibration plate has a 0.1mm accuracy, and the final calibration result still has a 3mm error, which is caused by the accuracy of the point cloud data itself. At the same time, the application does not need to adjust the mechanical arm movement and transform different poses, which avoids the influence of the mechanical arm control error on the hand-eye calibration, especially for live-line robot, a special robot, the calibration mode without mechanical arm movement is especially suitable for calibration in outdoor high-voltage work site. At the same time, since the mechanical arm is fixed, the application will introduce another problem, that is, where does the accurate position information in the real world come from. To this end, the application uses point cloud data to realize the simultaneous solution of point cloud data and real-world position information by subtracting two frames of point cloud data through the design of three-direction + rotation calibration tool. At the same time, the tool designed by the application cannot pay attention to three dimensions, and the requirement for the calibration tool is too high, which will lose industrial value. To this end, the application selects the depth dimension information of the point cloud to cooperate with the laser intensity diagram for calibration, and reduces the introduction of data in the other two dimensions.

[0048] The following two tables show the calibration embodiments of the application, and the operation process is as follows:

[0049] 1. Fix the calibration tool on the end of the robot arm.

[0050] 2. Set the position of the calibration board arbitrarily, and adjust the height so that the camera can take pictures of the calibration board; select any three target points and record their corresponding point cloud position information.

[0051] 3. Adjust the pose of the calibration board on the tool, including at least the change in depth; select the corresponding three target points in step 2, and record the corresponding point cloud position information.

[0052] 4. Calculate the transformation matrix according to the pose transformation relationship formula.

[0053] 5. Disassemble the calibration tool.

[0054] 6. Install the end effector, such as a needle-shaped tool, on the end of the robot arm, and input the shape parameters of the tool into the transformation matrix.

[0055] 7. Take pictures of any target with the camera, and manually select the position points.

[0056] 8. Control the robot arm to touch the position points selected in step 7 with the end of the needle-shaped tool.

[0057] 9. Measure the distance between the touch points and the position points, and output the verification deviation.

[0058] Table 1 and Table 2 are the orientation results output by the above embodiment, Table 1 is the accuracy measurement result of the robot arm touching the fixed object under different pitch angles of the camera to the touch points, and Table 2 is the accuracy measurement result of the robot arm touching the fixed object under different distances between the camera and the touch points. It can be seen that the calibration method of the present application can achieve good calibration accuracy under the condition that the robot arm does not move.

[0059] Table 1 (unit: mm)

[0060]

[0061]

[0062] Table 2 (unit: mm)

[0063] Target pitch angle (unit: degree) Shooting distance X-direction deviation Y-direction deviation Z-direction deviation 0 700 <1 1.3 -0.3 0 700 1.3 1.4 -0.4 0 700 <1 1.8 0.2 0 700 1 1 0.2 0 800 4.2 1.2 -1.5 0 800 4.6 1.8 -3 0 800 4.2 1.3 -2.1 0 800 3.1 1.5 -3.2 0 900 2.2 -1.7 1 0 900 1.6 -3.1 -1.8 0 900 2.7 -3.3 -1.6 0 900 1.3 -1.8 -2.8 0 1000 7.1 -0.2 1.2 0 1000 9 -1.3 0.8 0 1000 5.3 -2.3 1.6 0 1000 8.2 -1.7 1.2 0 1100 4.3 -3.8 0 0 1100 6 -3.1 1 0 1100 5.6 -2.2 1.9 0 1100 5.9 -4.6 0.4 0 1200 7.5 -2.7 0 0 1200 5.3 -2.9 0.7 0 1200 6.2 -3.5 0.7 0 1200 5.8 -3.3 -0.3

Claims

1. A method for laser camera based hand-eye calibration of a robot arm, characterized in that The calibration tool is fixed on the end of the mechanical arm, a laser camera is arranged on the base of the mechanical arm, the calibration tool is provided with a calibration plate with changeable poses, the laser camera is used to collect intensity maps and point cloud data of the calibration plate, and the method comprises the following steps: 1) collecting point cloud data and intensity maps by facing the calibration plate with the laser camera, obtaining point cloud position information P in the camera coordinate system and real point cloud position information Q in the world coordinate system from the collected point cloud data; 2) extracting feature points of the calibration plate from the obtained intensity maps, selecting three non-collinear points as target points, and obtaining point cloud position information P of the corresponding target points; 3) keeping the mechanical arm still, adjusting the pose of the calibration board, different poses including horizontal, vertical and depth direction displacement or rotation, taking the plane facing the camera as the horizontal and vertical plane, and the plane perpendicular to this plane as the depth, the adjustment at least including displacement in the depth direction, repeating steps 1) 2) collecting intensity maps and point cloud data of the calibration board under different poses, and solving the pose conversion relationship between the laser camera and the end of the mechanical arm by the obtained point cloud position information P of the target point and the real point cloud position information Q under the corresponding pose, completing the hand-eye calibration; when performing steps 1) 2), collect N the point cloud data of the secondary calibration board under the same pose m , and calculate the average point cloud information: wherein, P m,n representing the point cloud data obtained by the first acquisition, m n representing the average point cloud information at the pose P m representing the average point cloud information at the pose m , the camera coordinate system point cloud position information of the target point being used in the solution of the pose conversion relationship.​ The calibration tool comprises a support and a calibration plate, the support comprises a connecting piece, an xyz three-direction displacement platform, a rotating platform and a calibration plate clamping piece, the connecting piece is used to connect the calibration tool on the mechanical arm, the calibration plate clamping piece is used to fix the calibration plate, and the xyz three-direction displacement platform and the rotating platform are used to adjust the displacement or rotation of the calibration plate in the horizontal, vertical and depth directions.

2. The laser-camera-based robot hand-eye calibration method of claim 1, wherein The mechanical arm is the mechanical arm of the live-line working robot.

3. A laser-camera based robot hand-eye calibration apparatus, characterized by The calibration tool is arranged on the end arm of the mechanical arm, the laser camera is arranged on the base of the mechanical arm, the calibration tool is provided with a calibration plate with changeable poses, the calibration module is in data connection with the laser camera, the calibration module comprises a memory and a processor, the memory stores a computer program, the processor realizes the calibration calculation steps of the method in claim 1 or 2 when executing the computer program, the calibration module receives the data collected by the laser camera, and outputs pose conversion relationship information used for hand-eye calibration.

Images