Hand-eye calibration method and system based on orthogonal ChArUco calibration board

Abstract

The invention discloses a hand-eye calibration method and system based on orthogonal ChArUco calibration boards, and belongs to the field of computer vision and camera calibration, and the method comprises the steps: carrying out the shooting and scanning of two mutually perpendicular and orthogonal ChArUco calibration boards through an industrial camera and a line laser scanner, the obtained two-dimensional image key points and three-dimensional point cloud image key points are registered through an algorithm, then a transformation matrix between an industrial camera coordinate system and a line laser scanner coordinate system is obtained through the transformation relation between two-dimensional points and three-dimensional point clouds, finally, hand-eye calibration of the system is completed, and precise positioning and grabbing of the manipulator are achieved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and camera calibration, and in particular relates to a hand-eye calibration method and system based on an orthogonal ChArUco calibration plate. Background Art

[0002] The core of using an industrial camera to capture images of a checkerboard calibration plate is to calculate the transformation matrix between the camera coordinate system and the robot end-effector coordinate system through calibration, so that the image information captured by the camera can be accurately used by the robot to perform related tasks. This transformation matrix is ​​the key to achieving information docking between the camera and the robot, such as grasping, assembly, and welding.

[0003] At present, the traditional robot "hand-eye calibration" technology is mainly carried out through industrial cameras and checkerboard calibration plates. By taking images of the checkerboard calibration plate at different angles, the conversion relationship between the camera coordinate system and the calibration plate coordinate system is determined; through the principle of robot operation, the conversion relationship between the robot base and the mechanical claw coordinate system at the end of the robot arm is determined; through calibration at different positions and the deduction of mathematical relationships, the conversion relationship between the camera coordinate system and the mechanical claw coordinate system at the end of the robot arm is finally obtained to achieve the solution of different point poses. The traditional hand-eye calibration method is relatively cumbersome and has low calibration accuracy. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a hand-eye calibration method based on an orthogonal ChArUco calibration plate, comprising:

[0005] Based on the two-dimensional images of ChArUco calibration plates No. 1 and No. 2 taken by industrial cameras, the corner points in the two-dimensional images are obtained;

[0006] Performing point cloud scanning on the No. 1 and No. 2 ChArUco calibration plates by a line laser scanner to obtain corner points in a three-dimensional point cloud image;

[0007] Establishing a correspondence between a camera coordinate system and a point cloud coordinate system based on the corner points in the two-dimensional image and the corner points in the three-dimensional point cloud image;

[0008] Constructing a transformation matrix based on the correspondence between the camera coordinate system and the point cloud coordinate system;

[0009] The transformation matrix is ​​processed based on the hand-eye calibration algorithm to complete the overall system calibration.

[0010] Preferably, the expression of the corner point in the two-dimensional image is:

[0011] P 2D ={p=(u i ,v i )};

[0012] Among them, P 2D is the corner point extracted from the two-dimensional image, p is a corner point extracted from the two-dimensional image; u i is the horizontal coordinate of the corner point in the two-dimensional image; v i is the ordinate of the corner point in the two-dimensional image; i is the random index of a corner point in the two-dimensional image.

[0013] Preferably, the expression of the corner point in the three-dimensional point cloud image is:

[0014] P 3D = {q = (x i ,y i ,z i )};

[0015] Among them, P 3D is the corner point in the extracted 3D point cloud image, q is a corner point extracted in the 3D image; x i is the x coordinate of the corner point in the three-dimensional image; y i is the y coordinate of the corner point in the three-dimensional image; z i is the z coordinate of the corner point in the 3D image.

[0016] Preferably, the process of constructing a transformation matrix based on the correspondence between the camera coordinate system and the point cloud coordinate system includes:

[0017] Calculating the centroid of the corner points in the two-dimensional image and the corresponding corner points in the three-dimensional point cloud image and centralizing them to obtain the centralized points;

[0018] Calculate the covariance matrix through the centered points, and calculate the rotation matrix through singular value decomposition;

[0019] The transformation matrix is ​​constructed based on the covariance matrix and the rotation matrix.

[0020] Preferably, the centroid of the corner point is expressed as:

[0021]

[0022] Among them, C 2D is the centroid of the corner points in the two-dimensional image, C 3D is the centroid of the corner points in the 3D image.

[0023] Preferably, the expression of the rotation matrix is:

[0024] R=VU T ;

[0025] Among them, U is an orthogonal matrix, representing the left singular vectors of H; V is an orthogonal matrix, representing the right singular vectors of H, and T is the translation vector.

[0026] Preferably, the expression of the translation vector is:

[0027] T=C 3D -R.C 2D .

[0028] Preferably, the expression of the transformation matrix is:

[0029]

[0030] Where M is the transformation matrix.

[0031] In order to solve the above technical problems, the present invention also provides a hand-eye calibration system based on an orthogonal ChArUco calibration board, comprising:

[0032] Industrial camera, line laser scanner, two mutually perpendicular ChArUco calibration plates, mechanical claw, mechanical small arm, mechanical large arm, mechanical small arm drive motor, mechanical large arm drive motor, mechanical fixed support base and core algorithm processing unit;

[0033] The industrial camera is fixed above the mechanical claw through a fixing part, the line laser scanner is installed at the center of the mechanical claw, the mechanical fork arm is installed on the mechanical fork arm drive motor and is connected to the mechanical claw at the end of the mechanical fork arm, the core algorithm processing unit is installed at the end of the mechanical fork arm, the head end of the mechanical arm is connected to the mechanical arm drive motor, and the end is connected to the mechanical fork arm drive motor, the mechanical fixed support base keeps a fixed position, and the relative position with the two mutually perpendicular and orthogonal ChArUco calibration plates remains unchanged.

[0034] Compared with the prior art, the present invention has the following advantages and technical effects:

[0035] (1) It is less affected by the detection environment and has lower requirements on the external detection environment, so the accuracy of posture calibration will not be greatly affected by the detection environment.

[0036] (2) By aligning the key points of the two-dimensional image with the key points of the same part in the three-dimensional point cloud image, the transformation relationship between the two coordinate systems is obtained, which has high accuracy and good effect.

[0037] (3) The algorithm can directly obtain the conversion relationship between the camera coordinate system and the mechanical gripper coordinate system required for “hand-eye calibration”, avoiding a large number of coordinate conversions, simplifying the calibration process and mathematical calculations, and avoiding errors caused by calculations.

[0038] (4) A set of mutually orthogonal ChArUco calibration plates is used to avoid the problem that traditional hand-eye calibration is limited to a single plane, and acquisition and calibration can be achieved at multiple angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 A schematic diagram of the system structure of a hand-eye calibration system and method based on an orthogonal ChArUco calibration board according to an embodiment of the present invention;

[0041] Figure 2 It is a hand-eye calibration flow chart of a hand-eye calibration system and method based on an orthogonal ChArUco calibration board according to an embodiment of the present invention;

[0042] Figure 3 Schematic diagram of a coordinate system of a hand-eye calibration system and method based on an orthogonal ChArUco calibration plate according to an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of a ChArUco calibration plate of a hand-eye calibration system and method based on an orthogonal ChArUco calibration plate according to an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of corner point selection of a ChArUco calibration plate in a hand-eye calibration system and method based on an orthogonal ChArUco calibration plate according to an embodiment of the present invention;

[0045] Among them, 1. ChArUco calibration plate No. 1; 2. ChArUco calibration plate No. 2; 3. Line laser scanner; 4. Industrial camera; 5. Robotic forearm; 6. Robotic arm; 7. Robotic arm drive motor; 8. Mechanical fixed support base; 9. Robotic forearm drive motor; 10. Core algorithm processing unit; 11. Robotic claw. DETAILED DESCRIPTION

[0046] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] Embodiment 1

[0049] like Figure 1 As shown, this embodiment provides a hand-eye calibration method based on an orthogonal ChArUco calibration plate, including:

[0050] Based on the two-dimensional images of ChArUco calibration plates No. 1 and No. 2 captured by industrial camera 4, corner points in the two-dimensional images are obtained;

[0051] Performing point cloud scanning on the No. 1 and No. 2 ChArUco calibration plates by using a line laser scanner 3 to obtain corner points in a three-dimensional point cloud image;

[0052] Establishing a correspondence between a camera coordinate system and a point cloud coordinate system based on the corner points in the two-dimensional image and the corner points in the three-dimensional point cloud image;

[0053] Constructing a transformation matrix based on the correspondence between the camera coordinate system and the point cloud coordinate system;

[0054] The transformation matrix is ​​processed based on the hand-eye calibration algorithm to complete the overall system calibration.

[0055] Assume that the corner point extracted from the two-dimensional image captured by industrial camera 4 is P 2D , then the corner points extracted from the two-dimensional image can be expressed by the following formula (1).

[0056] P 2D ={p=(u i ,v i )} (1)

[0057] Assume that the corner points at the same position extracted from the three-dimensional point cloud image collected by the line laser scanner 3 are P 3D , then the corner points in the extracted three-dimensional point cloud image can be expressed by the following formula (2).

[0058] P 3D = {q = (x i ,y i ,z i )} (2)

[0059] In the two-dimensional point set P 2D and the three-dimensional point set P 3D The number of 3 pairs of points is selected so as to uniquely determine a transformation matrix from the three-dimensional space. Let n2D be the number of two-dimensional corner points and n3D be the number of three-dimensional corner points. Use the following formula (3) to generate three random indexes i1, i2, i3.

[0060] i j =rand(0,n2D-1) j=1,2,3,...,N (3)

[0061] From the two-dimensional point set P2D and the three-dimensional point set P 3D The corner points are obtained as shown in the following formula (4).

[0062]

[0063] Ensure that the mutual distance between the three selected points is not 0, and ensure that the three points are on the same plane and not collinear, and store the selected point pairs as a set. Extract the coordinates of the three point pairs from the stored set, as shown in the following equations (5) and (6).

[0064] 2D image point P 2D :

[0065] P 2D ={p1=(u1,v1),p2=(u2,v2),p3=(u3,v3)} (5)

[0066] 3D image point P 3D :

[0067] P 3D ={q1=(x1,y1,z1),q2=(x2,y2,z2),q3=(x3,y3,z3)} (6)

[0068] After extracting the coordinates of the three point pairs, calculate the centroid of each group separately as shown below.

[0069]

[0070] The point pairs are centered as shown in equation (9).

[0071]

[0072] The covariance matrix is ​​calculated through the centered points, as shown in the following equations (10) and (11).

[0073]

[0074] The rotation matrix is ​​calculated using singular value decomposition as shown in equation (12).

[0075] H=UλV T (12)

[0076] Where: U is an orthogonal matrix, representing the left singular vectors of H;

[0077] λ is a diagonal matrix with singular values ​​on the diagonal and the rest of the elements are 0;

[0078] V is an orthogonal matrix representing the right singular vectors of H.

[0079] Then, the rotation matrix R can be expressed as shown in the following formula (13).

[0080] R=VU T (13)

[0081] Then, the translation vector T can be expressed by the following equation (14).

[0082] T=C 3D -R.C 2D (14)

[0083] Then, the final transformation matrix M can be expressed as shown in the following formula (15).

[0084]

[0085] After obtaining the transformation matrix M between the camera coordinate system and the line laser scanner coordinate system, the hand-eye calibration of the system can be completed, the coordinate transformation relationship between different points can be solved, and the precise positioning and grasping of the robot can be achieved.

[0086] Embodiment 2

[0087] like Figure 2-5 As shown, this embodiment provides a hand-eye calibration system based on an orthogonal ChArUco calibration board, including:

[0088] It includes an industrial camera 4 and a line laser scanner 3, a ChArUco calibration plate No. 1 1, a ChArUco calibration plate No. 2 2, a mechanical claw 11, a mechanical small arm 5, a mechanical large arm 6, a mechanical small arm drive motor 9, a mechanical large arm drive motor 7, a mechanical fixed support base 8 and a core algorithm processing unit 10.

[0089] as follows Figure 1 As shown, the mechanical fixed support base 8 maintains a fixed position, and the relative position between the two mutually orthogonal and perpendicular calibration plates, No. 1 ChArUco calibration plate 1 and No. 2 ChArUco calibration plate 2 remains unchanged. It is connected to the mechanical arm drive motor 7 through a support member and provides support to keep the system relatively stable.

[0090] The head end of the mechanical arm 6 is connected to the mechanical arm drive motor 7, and the end of the mechanical arm 6 is connected to the mechanical arm drive motor 9. The mechanical arm drive motor 7 provides electric drive for the mechanical arm 6. The mechanical arm 6 provides support for the mechanical arm drive motor 9.

[0091] The mechanical arm 5 is mounted on the mechanical arm driving motor 9 and driven by the mechanical arm driving motor 9. The mechanical arm 5 is connected to the mechanical claw 11 at the end of the mechanical arm through a connecting piece and provides support for the end mechanical claw 11.

[0092] The line laser scanner 3 is installed at the center of the mechanical claw 11 at the end of the mechanical arm, and does not interfere with the grasping behavior of the end mechanical claw 11. The end mechanical claw 11 does not interfere with the field of view of the line laser scanner 3, ensuring that it can scan a clear three-dimensional point cloud image of the ChArUco calibration plate.

[0093] The industrial camera 4 is fixed above the mechanical claw 11 at the end of the mechanical arm through a fixing part. Like the line laser scanner 3, the industrial camera 4 and the end mechanical claw 11 do not interfere with each other, ensuring that the industrial camera 4 can clearly capture the two-dimensional image of the ChArUco calibration plate.

[0094] The core algorithm processing unit 10 is installed at the end of the mechanical arm 5, and mainly runs relevant detection algorithms thereon.

[0095] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A hand-eye calibration method based on an orthogonal ChArUco calibration plate, characterized in that: include: Based on the two-dimensional images of ChArUco calibration plates No. 1 and No. 2 taken by industrial cameras, the corner points in the two-dimensional images are obtained; Performing point cloud scanning on the No. 1 and No. 2 ChArUco calibration plates by a line laser scanner to obtain corner points in a three-dimensional point cloud image; Establishing a correspondence between a camera coordinate system and a point cloud coordinate system based on the corner points in the two-dimensional image and the corner points in the three-dimensional point cloud image; Constructing a transformation matrix based on the correspondence between the camera coordinate system and the point cloud coordinate system; The transformation matrix is ​​processed based on the hand-eye calibration algorithm to complete the overall system calibration.

2. A hand-eye calibration method based on an orthogonal ChArUco calibration plate according to claim 1, characterized in that: The expression of the corner point in the two-dimensional image is: P 2D ={p=(u i ,v i )}; Among them, P 2D is the corner point extracted from the two-dimensional image, p is a corner point extracted from the two-dimensional image; u i is the horizontal coordinate of the corner point in the two-dimensional image; v i is the ordinate of the corner point in the two-dimensional image; i is the random index of a corner point in the two-dimensional image.

3. The hand-eye calibration method based on the orthogonal ChArUco calibration plate according to claim 1, characterized in that: The expression of the corner points in the three-dimensional point cloud image is: P 3D ={q=(x i ,y i ,z i )}; Among them, P 3D is the corner point in the extracted 3D point cloud image, q is a corner point extracted in the 3D image; x i is the x coordinate of the corner point in the three-dimensional image; y i is the y coordinate of the corner point in the three-dimensional image; z i is the z coordinate of the corner point in the 3D image.

4. The hand-eye calibration method based on the orthogonal ChArUco calibration plate according to claim 1, characterized in that: The process of constructing a transformation matrix based on the correspondence between the camera coordinate system and the point cloud coordinate system includes: Calculating the centroid of the corner points in the two-dimensional image and the corresponding corner points in the three-dimensional point cloud image and centralizing them to obtain the centralized points; Calculate the covariance matrix through the centered points, and calculate the rotation matrix through singular value decomposition; The transformation matrix is ​​constructed based on the covariance matrix and the rotation matrix.

5. The hand-eye calibration method based on the orthogonal ChArUco calibration plate according to claim 4, characterized in that: The expression of the centroid of the corner point is: Among them, C 2D is the centroid of the corner points in the two-dimensional image, C 3D is the centroid of the corner points in the 3D image.

6. The hand-eye calibration method based on the orthogonal ChArUco calibration plate according to claim 4, characterized in that: The expression of the rotation matrix is: R=VU T ; Among them, U is an orthogonal matrix, representing the left singular vectors of H; V is an orthogonal matrix, representing the right singular vectors of H, and T is the translation vector.

7. The hand-eye calibration method based on the orthogonal ChArUco calibration plate according to claim 6, characterized in that: The expression of the translation vector is: T=C 3D -R·C 2D 。 8. The hand-eye calibration method based on the orthogonal ChArUco calibration plate according to claim 1, characterized in that: The expression of the transformation matrix is: Where M is the transformation matrix.

9. A hand-eye calibration system based on an orthogonal ChArUco calibration plate, characterized in that: include: Industrial camera, line laser scanner, two mutually perpendicular ChArUco calibration plates, mechanical claw, mechanical small arm, mechanical large arm, mechanical small arm drive motor, mechanical large arm drive motor, mechanical fixed support base and core algorithm processing unit; The industrial camera is fixed above the mechanical claw through a fixing part, the line laser scanner is installed at the center of the mechanical claw, the mechanical fork arm is installed on the mechanical fork arm drive motor and is connected to the mechanical claw at the end of the mechanical fork arm, the core algorithm processing unit is installed at the end of the mechanical fork arm, the head end of the mechanical arm is connected to the mechanical arm drive motor, and the end is connected to the mechanical fork arm drive motor, the mechanical fixed support base keeps a fixed position, and the relative position with the two mutually perpendicular and orthogonal ChArUco calibration plates remains unchanged.

Images