Hand-eye calibration method and system based on orthogonal ChArUco calibration board

Abstract

The application discloses a kind of hand-eye calibration method and system based on orthogonal ChArUco calibration board, the field is computer vision and camera calibration field, including: by industrial camera and line laser scanner respectively to two mutually perpendicular orthogonal ChArUco calibration board are photographed and scanned, by algorithm, the key points of two-dimensional image and three-dimensional point cloud image key points are registered, then by the conversion relationship between two-dimensional point and three-dimensional point cloud, the transformation matrix between industrial camera coordinate system and line laser scanner coordinate system is obtained, finally, the hand-eye calibration of system is completed, realizes the accurate positioning and grasping of manipulator.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and camera calibration, and particularly relates to a hand-eye calibration method and system based on an orthogonal ChArUco calibration board. Background Technology

[0002] The core of using an industrial camera to capture images of a checkerboard calibration plate is to calculate the transformation matrix between the camera coordinate system and the robot's end effector coordinate system through calibration. This allows the image information captured by the camera to be accurately used by the robot to perform related tasks. This transformation matrix is ​​the key to achieving information exchange between the camera and the robot, such as grasping, assembly, and welding.

[0003] Currently, traditional robot "hand-eye calibration" technology mainly relies on industrial cameras and a checkerboard calibration board. By capturing images of the calibration board from different angles, the transformation relationship between the camera coordinate system and the calibration board coordinate system is determined. Then, based on robot kinematics principles, the transformation relationship between the robot base and the end effector's coordinate system is determined. Through calibration at different positions and the derivation of mathematical relationships, the final transformation relationship between the camera coordinate system and the end effector's coordinate system is obtained, enabling the calculation of poses at different points. Traditional hand-eye calibration methods are cumbersome and have relatively low calibration accuracy. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention provides a hand-eye calibration method based on an orthogonal ChArUco calibration plate, comprising:

[0005] Two-dimensional images of ChArUco calibration boards No. 1 and No. 2 were captured using an industrial camera to obtain corner points in the two-dimensional images.

[0006] The corner points in the three-dimensional point cloud image were obtained by scanning the No. 1 and No. 2 ChArUco calibration boards with a line laser scanner.

[0007] Establish the correspondence between the camera coordinate system and the point cloud coordinate system based on the corner points in the two-dimensional image and the corner points in the three-dimensional point cloud image;

[0008] A transformation matrix is ​​constructed based on the correspondence between the camera coordinate system and the point cloud coordinate system;

[0009] The transformation matrix is ​​processed based on the hand-eye calibration algorithm to complete the overall system calibration.

[0010] Preferably, the expression for the corner point in the two-dimensional image is:

[0011] P 2D ={p=(u i ,v i )};

[0012] Among them, P 2D Let p be a corner point extracted from a 2D image; u i v is the x-coordinate of a corner point in a two-dimensional image; i y is the ordinate of a corner point in the 2D image; i is a random index of a corner point in the 2D image.

[0013] Preferably, the expression for the corner points in the three-dimensional point cloud image is:

[0014] P 3D ={q=(x i ,y i ,z i )};

[0015] Among them, P 3D x represents a corner point in the extracted 3D point cloud image, where q is a specific extracted corner point in the 3D image; i The x-coordinate of a corner point in a 3D image; the y-coordinate of the corner point. i The z-coordinate is the y-coordinate of a corner point in a 3D image; i The z-coordinate of a corner point in a 3D image.

[0016] Preferably, the process of constructing the transformation matrix based on the correspondence between the camera coordinate system and the point cloud coordinate system includes:

[0017] Calculate the centroids of the corner points in the two-dimensional image and the corresponding corner points in the three-dimensional point cloud image, and center them to obtain the centered points;

[0018] The covariance matrix is ​​calculated using the centered points, and the rotation matrix is ​​calculated using singular value decomposition.

[0019] The transformation matrix is ​​constructed based on the covariance matrix and the rotation matrix.

[0020] Preferably, the expression for the centroid of the corner point is:

[0021]

[0022] Among them, C 2D Let C be the centroid of a corner point in a two-dimensional image. 3D Let be the centroid of the corner point in the 3D image.

[0023] Preferably, the expression for the rotation matrix is:

[0024] R = VU T ;

[0025] Where U is an orthogonal matrix representing the left singular vector of H; V is an orthogonal matrix representing the right singular vector of H; and T is a translation vector.

[0026] Preferably, the expression for the translation vector is:

[0027] T = C 3D -R·C 2D .

[0028] Preferably, the expression for the transformation matrix is:

[0029]

[0030] Where M is the transformation matrix.

[0031] To address the aforementioned technical problems, this invention also provides a hand-eye calibration system based on an orthogonal ChArUco calibration board, comprising:

[0032] Industrial camera, line laser scanner, two mutually perpendicular or orthogonal ChArUco calibration plates, robotic gripper, robotic forearm, robotic upper arm, robotic forearm drive motor, robotic upper arm drive motor, mechanical fixed support base and core algorithm processing unit.

[0033] The industrial camera is fixed above the robotic gripper by a fastener. The line laser scanner is mounted at the center of the robotic gripper. The robotic forearm is mounted on the robotic forearm drive motor and connected to the robotic gripper at the end of the robotic forearm. The core algorithm processing unit is mounted at the end of the robotic forearm. The head of the robotic arm is connected to the robotic arm drive motor, and the tail is connected to the robotic forearm drive motor. The mechanical fixed support base remains in a fixed position, and its relative position to the two mutually perpendicular ChArUco calibration plates remains unchanged.

[0034] Compared with the prior art, the present invention has the following advantages and technical effects:

[0035] (1) It is less affected by the detection environment and has lower requirements for the external detection environment. It will not have a significant impact on the accuracy of the pose calibration due to the detection environment.

[0036] (2) By registering the key points of the two-dimensional image and the key points of the same part in the three-dimensional point cloud image, the transformation relationship between the two coordinate systems is obtained with high accuracy and good effect.

[0037] (3) The transformation relationship between the camera coordinate system and the mechanical claw coordinate system required for "hand-eye calibration" can be obtained directly through the algorithm, avoiding a large number of coordinate transformations, simplifying the calibration process and mathematical calculations, and avoiding errors caused by calculations.

[0038] (4) A set of mutually orthogonal and perpendicular ChArUco calibration plates are used, which avoids the problem that traditional hand-eye calibration is limited to a single plane and can realize data acquisition and calibration from multiple angles. Attached Figure Description

[0039] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is a schematic diagram of the system structure of the hand-eye calibration system and method based on the orthogonal ChArUco calibration plate according to an embodiment of the present invention;

[0041] Figure 2 This is a flowchart illustrating the hand-eye calibration process of the hand-eye calibration system and method based on the orthogonal ChArUco calibration plate according to an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the coordinate system for the hand-eye calibration system and method based on the orthogonal ChArUco calibration plate according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the ChArUco calibration plate for the hand-eye calibration system and method based on the orthogonal ChArUco calibration plate according to an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the ChArUco calibration plate corner point selection for the hand-eye calibration system and method based on the orthogonal ChArUco calibration plate according to an embodiment of the present invention.

[0045] The components include: 1. ChArUco calibration board 1; 2. ChArUco calibration board 2; 3. Line laser scanner; 4. Industrial camera; 5. Robotic forearm; 6. Robotic upper arm; 7. Robotic upper arm drive motor; 8. Mechanical fixed support base; 9. Robotic forearm drive motor; 10. Core algorithm processing unit; and 11. Robotic gripper. Detailed Implementation

[0046] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0048] Example 1

[0049] like Figure 1 As shown, this embodiment provides a hand-eye calibration method based on an orthogonal ChArUco calibration board, including:

[0050] Two-dimensional images of ChArUco calibration boards No. 1 and No. 2 were captured using an industrial camera 4 to obtain corner points in the two-dimensional images.

[0051] The corner points in the three-dimensional point cloud image were obtained by scanning the No. 1 and No. 2 ChArUco calibration boards with a line laser scanner 3.

[0052] Establish the correspondence between the camera coordinate system and the point cloud coordinate system based on the corner points in the two-dimensional image and the corner points in the three-dimensional point cloud image;

[0053] A transformation matrix is ​​constructed based on the correspondence between the camera coordinate system and the point cloud coordinate system;

[0054] The transformation matrix is ​​processed based on the hand-eye calibration algorithm to complete the overall system calibration.

[0055] Assume that the corner point extracted from the 2D image acquired by industrial camera 4 is P. 2D Then the corner points in the extracted two-dimensional image can be represented by the following formula (1).

[0056] P 2D ={p=(u i ,v i )} (1)

[0057] Assume that the corner point at the same location extracted from the 3D point cloud image acquired by the line laser scanner 3 is P. 3D Then, the corner points in the extracted 3D point cloud image can be represented by the following formula (2).

[0058] P 3D ={q=(x i ,y i ,z i (2)

[0059] In the two-dimensional point set P 2D and the three-dimensional point set P 3D Choose the number of 3 point pairs to uniquely determine a transformation matrix from the three-dimensional space. Let n2D be the number of two-dimensional corner points and n3D be the number of three-dimensional corner points. Use the following formula (3) to generate three random indices i1, i2, i3.

[0060] i j =rand(0,n2D-1) j=1,2,3,...,N (3)

[0061] From the two-dimensional point set P2D and the three-dimensional point set P 3D The corner point is obtained as shown in equation (4) below.

[0062]

[0063] Ensure that the distance between the three selected points is not zero, and that the three points are on the same plane and not collinear. Store the selected point pairs as a set. Extract the coordinates of the three point pairs from the stored set, as shown in equations (5) and (6) below.

[0064] Two-dimensional image point P 2D :

[0065] P 2D ={p1=(u1,v1),p2=(u2,v2),p3=(u3,v3)} (5)

[0066] 3D image point P 3D :

[0067] P 3D ={q1=(x1,y1,z1),q2=(x2,y2,z2),q3=(x3,y3,z3)} (6)

[0068] After extracting the coordinates of the three point pairs, the centroid of each pair is calculated as follows.

[0069]

[0070] The point pairs are centered as shown in equation (9).

[0071]

[0072] The covariance matrix is ​​calculated using the centered points, as shown in equations (10) and (11).

[0073]

[0074] The rotation matrix is ​​calculated using singular value decomposition, as shown in equation (12).

[0075] H=UλV T (12)

[0076] Where: U is an orthogonal matrix, representing the left singular vector of H;

[0077] λ is a diagonal matrix with singular values ​​on the diagonal and 0 for the rest;

[0078] V is an orthogonal matrix representing the right singular vector of H.

[0079] Then, the rotation matrix R can be expressed as shown in equation (13).

[0080] R = VU T (13)

[0081] Then, the translation vector T can be represented by the following equation (14).

[0082] T = C 3D -R·C 2D (14)

[0083] Then, the final transformation matrix M can be expressed as shown in equation (15).

[0084]

[0085] After obtaining the transformation matrix M between the camera coordinate system and the line laser scanner coordinate system, the hand-eye calibration of the system can be completed, the coordinate transformation relationship between different points can be calculated, and the robot arm can achieve precise positioning and grasping.

[0086] Example 2

[0087] like Figure 2-5 As shown, this embodiment provides a hand-eye calibration system based on an orthogonal ChArUco calibration board, comprising:

[0088] It includes an industrial camera 4 and a line laser scanner 3, a ChArUco calibration board 1, a ChArUco calibration board 2, a robotic gripper 11, a robotic forearm 5, a robotic upper arm 6, a robotic forearm drive motor 9, a robotic upper arm drive motor 7, a mechanical fixed support base 8, and a core algorithm processing unit 10.

[0089] as follows Figure 1 As shown, the mechanical fixed support base 8 keeps its position fixed, and the relative positions between the two mutually orthogonal and perpendicular calibration plates, No. 1 ChArUco calibration plate 1 and No. 2 ChArUco calibration plate 2, remain unchanged. It is connected to the mechanical arm drive motor 7 through the support component and provides support to keep the system relatively stable.

[0090] The upper end of the robotic boom 6 is connected to the robotic boom drive motor 7, and the lower end of the robotic boom 6 is connected to the robotic forearm drive motor 9. The robotic boom drive motor 7 provides power to the robotic boom 6. The robotic boom 6 provides support for the robotic forearm drive motor 9.

[0091] The robotic arm 5 is mounted on the robotic arm drive motor 9 and is driven by the robotic arm drive motor 9. The robotic arm 5 is connected to the robotic claw 11 at the end of the robotic arm via a connector and provides support for the robotic claw 11.

[0092] The line laser scanner 3 is mounted at the center of the end effector 11 of the robotic arm, without interfering with the grasping behavior of the end effector 11. The end effector 11 does not interfere with the field of view of the line laser scanner 3, ensuring that it can scan a clear 3D point cloud image of the ChArUco calibration board.

[0093] The industrial camera 4 is fixed above the end-effector 11 of the robotic arm by a fastener. Similar to the line laser scanner 3, the industrial camera 4 and the end-effector 11 do not interfere with each other, ensuring that the industrial camera 4 can clearly capture two-dimensional images of the ChArUco calibration plate.

[0094] The core algorithm processing unit 10 is installed at the end of the robotic arm 5, and mainly runs relevant detection algorithms on it.

[0095] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A hand-eye calibration method based on a quadrature ChArUco calibration board, characterized in that, The method comprises the following steps: Based on the industrial camera, the two-dimensional images of the No. 1 and No. 2 ChArUco calibration boards are captured, and the corner points in the two-dimensional images are obtained; The point cloud scanning of the No. 1 and No. 2 ChArUco calibration boards is carried out by using a line laser scanner, and the corner points in the three-dimensional point cloud images are obtained; The correspondence between the camera coordinate system and the point cloud coordinate system is established based on the corner points in the two-dimensional images and the corner points in the three-dimensional point cloud images; The transformation matrix is constructed based on the correspondence between the camera coordinate system and the point cloud coordinate system; The transformation matrix is processed based on the hand-eye calibration algorithm, and the overall system calibration is completed; The process of constructing the transformation matrix based on the correspondence between the camera coordinate system and the point cloud coordinate system comprises: The center of mass of the corner points based on the two-dimensional images and the corresponding corner points in the three-dimensional point cloud images is calculated and centralized, and the centralized points are obtained; The covariance matrix is calculated by using the centralized points, and the rotation matrix is calculated by using singular value decomposition; The transformation matrix is constructed based on the covariance matrix and the rotation matrix; The expression of the center of mass of the corner points is: ； ； wherein is the centroid of the corner point in the two-dimensional image, is the centroid of the corner point in the three-dimensional image; The expression of the rotation matrix is: ； wherein is an orthogonal matrix representing the left singular vectors of H; is an orthogonal matrix representing the right singular vectors of H, and T is a translation vector; The expression of the translation vector is: 。 2. The method of claim 1, wherein, The expression of the corner points in the two-dimensional images is: ； wherein, is an extracted corner point in a two-dimensional image, is a certain extracted corner point in a two-dimensional image; is an abscissa of a corner point in a two-dimensional image; is an ordinate of a corner point in a two-dimensional image; is a random index of a certain corner point in a two-dimensional image.

3. The method of claim 1, wherein, The expression of the corner points in the three-dimensional point cloud images is: ； wherein, is an angle point in the extracted three-dimensional point cloud image, is a certain extracted angle point in the three-dimensional image; is an x-coordinate of the angle point in the three-dimensional image; is a y-coordinate of the angle point in the three-dimensional image; is a z-coordinate of the angle point in the three-dimensional image.

4. The method of claim 1, wherein, The expression of the transformation matrix is: Wherein, M is the transformation matrix.

5. The system of hand-eye calibration method based on orthogonal ChArUco board according to any one of claims 1-4, characterized in that, The method comprises the following steps: An industrial camera, a line laser scanner, two mutually perpendicular and orthogonal ChArUco calibration boards, a mechanical claw, a mechanical small arm, a mechanical large arm, a mechanical small arm driving motor, a mechanical large arm driving motor, a mechanical fixed support base and a core algorithm processing unit are included; The industrial camera is fixed above the mechanical claw by a fixing part, the line laser scanner is installed at the center of the mechanical claw, the mechanical small arm is installed on the mechanical small arm driving motor and connected with the mechanical claw at the end of the mechanical small arm, the core algorithm processing unit is installed at the end of the mechanical small arm, the mechanical large arm is connected with the mechanical large arm driving motor at the first end and connected with the mechanical small arm driving motor at the end, and the mechanical fixed support base keeps the position fixed and the relative position between the two mutually perpendicular and orthogonal ChArUco calibration boards keeps unchanged.

Images