Automatic calibration method and system based on binocular camera and robotic arm, and storage medium

Abstract

The present invention discloses a binocular camera and robotic arm automatic calibration method, system, and storage medium, belonging to the technical field of automatic calibration. This invention proposes a fast, efficient, and highly accurate binocular camera and robotic arm automatic calibration system. This system requires only a few simple operations to complete the calibration task, obtaining the binocular camera's intrinsic parameters, the transformation matrix between the camera coordinate system and the robotic arm coordinate system, and the position offset of the end mechanism relative to the end of the robotic arm. After the calibration is completed, the calibration results can be verified to ensure their accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic calibration, and in particular relates to an automatic calibration method and system based on a binocular camera and a robotic arm, and a storage medium. Background Art

[0002] A binocular camera is an important type of stereo vision camera. It can match the feature points of two aligned images through a binocular matching algorithm, and then restore the depth information of the pixels based on the parallax, baseline, and focal length. Due to its easy installation and use and low price, it has been widely used in fields such as industrial robots and autonomous driving.

[0003] Binocular matching algorithms require that the two images be aligned row by row, meaning that the epipolar lines of the two images are parallel. However, in the real world, two cameras are not perfectly parallel, nor are they perfectly horizontal relative to each other. Therefore, the image pairs obtained by the binocular cameras must be calibrated to ensure proper alignment. This image correction depends on the relative pose of the cameras, so calibration of the binocular cameras is necessary to ensure proper operation.

[0004] Robotic arms are common equipment in the automation industry. When grasping with a robotic arm, it's often necessary to know the positional relationship between the target and the robotic arm. Using a camera to obtain the target's position is an effective method. However, simply using a camera to obtain the target's position is in the camera's coordinate system, while grasping tasks require the position between the target and the robotic arm. Therefore, some method is needed to determine the positional relationship between the camera and the robotic arm, thereby transforming the target from the camera coordinate system to the robotic arm's coordinate system. After binocular camera calibration, hand-eye calibration is performed based on the positional relationship between the camera and the robotic arm to determine the coordinate transformation relationship between the camera and the robotic arm. Finally, to accurately control the robotic arm's motion, end-point calibration is required.

[0005] Binocular cameras in industrial scenarios operate in diverse environments, requiring frequent calibration to maintain high precision. Existing calibration methods typically rely on specialized calibration equipment and require relatively high levels of operator input. The process is cumbersome and inefficient, hindering efficient calibration. Current calibration methods require manual image acquisition and separate camera, hand-eye, and end-point calibration, significantly impacting calibration efficiency. Summary of the Invention

[0006] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides an automatic calibration method and system based on a binocular camera and a robotic arm, and a storage medium.

[0007] The present invention adopts the following technical solution: an online automatic calibration method based on a binocular camera and a robotic arm, comprising at least the following steps:

[0008] Set the robot, binocular camera and calibration block according to the predetermined positions and create the basic coordinate system respectively , Robotic arm coordinate system , camera coordinate system and calibration plate coordinate system ;

[0009] The end of the robotic arm moves along a predetermined path, recording the path points during the movement; the binocular camera image of each path point is collected, and the binocular camera image is effectively screened to obtain a valid image set;

[0010] Based on the valid image set, dual-target calibration is performed to obtain the camera calibration parameters. The valid image set and camera calibration parameters are combined to perform hand-eye calibration to obtain the hand-eye transformation matrix between the camera coordinate system and the robotic arm coordinate system, i.e., the hand-eye calibration information.

[0011] Obtain the binocular camera image of the end of the robotic arm at the end position, calculate the pixel position of the end of the robotic arm in the camera coordinate system, and combine the hand-eye calibration information to obtain the transformation relationship of the robotic arm end at the end position relative to the mechanical coordinate system, that is, the end calibration information;

[0012] Output camera calibration parameters, hand-eye calibration information, and end-point calibration information.

[0013] In a further embodiment, the valid image set is obtained by the following steps:

[0014] Step 201: Move along the X-axis of the robot coordinate system according to a predetermined point to obtain multiple binocular camera images about the X-axis;

[0015] Step 202: Determine whether there is a calibration plate in the binocular camera image about the X axis. If so, define it as a valid image about the X axis until the calibration plate is no longer within the field of view of the binocular camera.

[0016] Step 203: Count the number of valid images obtained ,like , then continue to obtain along the X axis until ;

[0017] Step 204: Move along the Y-axis of the robot coordinate system according to a predetermined point to obtain multiple binocular camera images about the Y-axis;

[0018] Step 205: Determine whether there is a calibration plate in the binocular camera image about the Y axis. If so, it is defined as a valid image about the Y axis until the calibration plate is no longer within the field of view of the binocular camera.

[0019] Step 206: Count the number of valid images obtained ,like , then repeat steps 205 to 206 until ;

[0020] Step 207: Move along the Z axis of the robot coordinate system according to a predetermined point to obtain multiple binocular camera images about the Z axis;

[0021] Step 208: Determine whether there is a calibration plate in the binocular camera image along the Z axis. If so, define it as a valid image along the Z axis until the calibration plate is no longer within the field of view of the binocular camera.

[0022] Step 209: Count the number of valid images obtained ,like , then repeat steps 205 to 206 until ;

[0023] The valid image about the X axis, the valid image about the Y axis, and the valid image about the Z axis are stored as a valid image set, wherein, 、 and These are pre-set values.

[0024] In a further embodiment, the calibration process for performing dual-target calibration based on a valid image set is as follows:

[0025] Get the coordinates of a point on the calibration plate in the world system as , and the coordinates of the point in the camera coordinate system are 、 ; Based on coordinates 、 and The rotation and translation matrices of the point relative to the left camera are calculated as follows: 、 , and the rotation and translation matrices of this point relative to the right camera are 、 ;

[0026] Then, the following formulas are satisfied for this point:

[0027] , calculated ;

[0028] At the same time, the following formulas are satisfied: ;

[0029] Then, using the formula: ;

[0030] Calculation yields: 、 Where, is the rotation matrix of the right camera relative to the left camera, is the translation matrix of the right camera relative to the left camera, then the rotation matrix and translation matrices This is the camera calibration parameter.

[0031] In a further embodiment, the calculation process of the hand-eye transformation matrix between the camera coordinate system and the robotic arm coordinate system is as follows:

[0032] Define the robot arm in the base coordinate system If the position in is (X, Y), it is calculated by the following formula:

[0033] ;

[0034] ;

[0035] because and The coordinate system is fixed, so Does not change, so:

[0036]

[0037] ;

[0038] remember for , for , for , then:

[0039] ; and B are known quantities, so we solve Get the hand-eye transformation matrix;

[0040] Where, is the position of the point on the calibration plate relative to the robot coordinate system; Indicates the transformation relationship from the manipulator coordinate system to the base coordinate system; Indicates the transformation relationship from the camera coordinate system to the manipulator coordinate system; Indicates the transformation relationship from the camera coordinate system to the calibration plate coordinate system; Indicates the transformation relationship from the calibration plate coordinate system to the base coordinate system; is the position of the point on the calibration plate relative to the calibration plate coordinate system; Indicates the conversion relationship of the X-axis from the robot coordinate system to the basic coordinate system. Indicates the conversion relationship of the X-axis from the camera coordinate system to the manipulator coordinate system. Indicates the transformation relationship of the Y axis from the robot coordinate system to the basic coordinate system, Indicates the transformation relationship of the Y axis from the camera coordinate system to the manipulator coordinate system. In a further embodiment, the method for obtaining the transformation relationship of the end of the manipulator relative to the mechanical coordinate system when the end of the manipulator is at the end position is as follows:

[0041] Define the tool center point at the end of the robot arm as P , find the point in the corresponding binocular camera image P , calculate the point based on the camera calibration parameters P Pixel position in camera coordinate system ;

[0042] Combined with the hand-eye calibration information, the following formula is used to calculate the point P The offset vector :

[0043] Where, Indicates the conversion relationship from the camera coordinate system to the manipulator coordinate system, the offset vector This is the terminal calibration information.

[0044] In a further embodiment, the terminal calibration information is further verified, and the verification method is as follows:

[0045] The offset vector calculated based on hand-eye calibration Obtain the TCP position, project it onto the image according to the camera calibration parameters, and obtain the theoretical pixel position of the end position on the image. If the calculated theoretical pixel position in the image coincides with the TCP position in the field of view, the calibration result is reliable. If not, it indicates that the binocular calibration parameter error is large and the binocular camera parameters need to be recalibrated.

[0046] In a further embodiment, the calibration plate is a checkerboard.

[0047] A computer system comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0048] A computer-readable storage medium stores a computer program, which implements the steps of the method described above when executed by a processor.

[0049] Beneficial effects of the present invention: The present invention integrates binocular calibration, robotic arm hand-eye calibration and terminal calibration into one system, automatically collects path points, quickly and efficiently calculates the calibration information of the binocular vision sensor, and then optimizes and solves the hand-eye calibration transformation matrix multiple times to determine the relative position of the camera and the robotic arm hand. Combining the theory of spatial coordinate transformation with robot kinematics, the TCP calibration and verification process is integrated to overcome the shortcomings of slow calibration speed and insufficient calibration accuracy in the traditional calibration process. After the automatic calibration is completed, the system can verify the calibration results based on the displayed terminal pixel position to ensure the correctness of the calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of the automatic calibration method based on a binocular camera and a robotic arm of Example 1.

[0051] Figure 2 This is a flow chart of the acquisition of the valid image set in Example 1.

[0052] Figure 3 This is a flowchart for obtaining terminal calibration information in Example 1. DETAILED DESCRIPTION

[0053] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0054] The present invention proposes a fast, efficient and high-precision binocular camera and robotic arm automatic calibration system. The calibration task can be completed with only convenient operations to obtain the binocular camera internal parameters, the transformation matrix between the camera coordinate system and the robotic arm coordinate system, and the position offset of the end mechanism relative to the end of the robotic arm. After the calibration is completed, the calibration results can also be verified to ensure the correctness of the calibration results. The overall process is as follows: Figure 1 As shown, the following describes:

[0055] Set the robot, binocular camera and calibration block according to the predetermined positions and create the basic coordinate system respectively , Robotic arm coordinate system , camera coordinate system and calibration plate coordinate system ;

[0056] The end of the robotic arm moves along a predetermined path, recording the path points during the movement; the binocular camera image of each path point is collected, and the binocular camera image is effectively screened to obtain a valid image set;

[0057] Based on the valid image set, dual-target calibration is performed to obtain the camera calibration parameters. The valid image set and camera calibration parameters are combined to perform hand-eye calibration to obtain the hand-eye transformation matrix between the camera coordinate system and the robotic arm coordinate system, i.e., the hand-eye calibration information.

[0058] Obtain the binocular camera image of the end of the robotic arm at the end position, calculate the pixel position of the end of the robotic arm in the camera coordinate system, and combine the hand-eye calibration information to obtain the transformation relationship of the robotic arm end at the end position relative to the mechanical coordinate system, that is, the end calibration information;

[0059] Output camera calibration parameters, hand-eye calibration information, and end-point calibration information.

[0060] It should be noted that the above solution can be applied to a variety of robotic arms. The binocular camera can be installed on the robotic arm or fixed in a certain position. In the automatic acquisition path, the movement order of the robotic arm can be adjusted, for example, in the order of Y axis, X axis, and Z axis.

[0061] In a further embodiment, since the feature points of a checkerboard need to be used in the hand-eye calibration process, a certain amount of checkerboard images need to be collected. That is, in this embodiment, the calibration plate uses a checkerboard.

[0062] In the following, we take a chessboard as an example. During the acquisition process, the chessboard is required to be completely within the camera's field of view and as clear as possible, otherwise it will affect the accuracy of the subsequent hand-eye transformation matrix calculation. Therefore, this embodiment proposes a method for acquiring binocular camera images and valid image sets, combined with Figure 2 The following steps are involved:

[0063] To determine the path collection points, perform the following steps:

[0064] Step 201: Move along the X-axis of the robot coordinate system according to a predetermined point to obtain multiple binocular camera images about the X-axis;

[0065] Step 202: Determine whether there is a calibration plate in the binocular camera image about the X axis. If so, define it as a valid image about the X axis until the calibration plate is no longer within the field of view of the binocular camera.

[0066] Step 203: Count the number of valid images obtained ,like , then continue to obtain along the X axis until ;

[0067] Step 204: Move along the Y-axis of the robot coordinate system according to a predetermined point to obtain multiple binocular camera images about the Y-axis;

[0068] Step 205: Determine whether there is a calibration plate in the binocular camera image about the Y axis. If so, it is defined as a valid image about the Y axis until the calibration plate is no longer within the field of view of the binocular camera.

[0069] Step 206: Count the number of valid images obtained ,like , then repeat steps 205 to 206 until ;

[0070] Step 207: Move along the Z axis of the robot coordinate system according to a predetermined point to obtain multiple binocular camera images about the Z axis;

[0071] Step 208: Determine whether there is a calibration plate in the binocular camera image along the Z axis. If so, define it as a valid image along the Z axis until the calibration plate is no longer within the field of view of the binocular camera.

[0072] Step 209: Count the number of valid images obtained ,like , then repeat steps 205 to 206 until ;

[0073] The valid image about the X axis, the valid image about the Y axis, and the valid image about the Z axis are stored as a valid image set, wherein, 、 and All are pre-set values ​​and the collection is completed.

[0074] Binocular calibration is the foundation for hand-eye calibration and end-point calibration of the robotic arm. Accurate binocular camera parameters are crucial for precise control of the robotic arm. The binocular camera parameters are calculated based on automatically acquired image information. Binocular calibration not only requires calibrating the distortion parameters of the two cameras, but also crucially, determining their relative position.

[0075] Therefore, this embodiment discloses a calibration process for performing dual-target calibration based on a valid image set as follows:

[0076] Get the coordinates of a point on the calibration plate in the world system as , and the coordinates of the point in the camera coordinate system are 、 ; Based on coordinates 、 and The rotation and translation matrices of the point relative to the left camera are calculated as follows: 、 , and the rotation and translation matrices of this point relative to the right camera are 、 ;

[0077] Then, the following formulas are satisfied for this point:

[0078] , calculated ;

[0079] At the same time, the following formulas are satisfied: ;

[0080] Then, using the formula: ;

[0081] Calculation yields: 、 Where, is the rotation matrix of the right camera relative to the left camera, is the translation matrix of the right camera relative to the left camera, then the rotation matrix and translation matrices These are the camera calibration parameters. Afterwards, multiple optimizations are performed to obtain more accurate parameters.

[0082] After the binocular camera calibration is completed, the hand-eye calibration is performed. The calculation process of the hand-eye transformation matrix between the camera coordinate system and the robotic arm coordinate system is as follows:

[0083] Define the robot arm in the base coordinate system If the position in is (X, Y), it is calculated by the following formula:

[0084] ;

[0085] ;

[0086] because and The coordinate system is fixed, so Does not change, so:

[0087]

[0088] ;

[0089] remember for , for , for , then:

[0090] ; and B are known quantities, so we solve Get the hand-eye transformation matrix;

[0091] Where, is the position of the point on the calibration plate relative to the robot coordinate system; Indicates the transformation relationship from the manipulator coordinate system to the base coordinate system; Indicates the transformation relationship from the camera coordinate system to the manipulator coordinate system; Indicates the transformation relationship from the camera coordinate system to the calibration plate coordinate system; Indicates the transformation relationship from the calibration plate coordinate system to the base coordinate system; is the position of the point on the calibration plate relative to the calibration plate coordinate system; Indicates the conversion relationship of the X-axis from the robot coordinate system to the basic coordinate system. Indicates the conversion relationship of the X-axis from the camera coordinate system to the manipulator coordinate system. Indicates the transformation relationship of the Y axis from the robot coordinate system to the basic coordinate system, Indicates the transformation relationship of the Y-axis from the camera coordinate system to the robot coordinate system.

[0092] The tool center point (TCP) of an industrial robot is the actual motion trajectory of the robot. After completing the binocular and hand-eye calibration, the end mechanism needs to be calibrated, that is, the offset vector from the TCP to the robot arm is obtained. .

[0093] Combine Figure 3 , the method for obtaining the transformation relationship of the end of the robot arm relative to the mechanical coordinate system when it is at the end position is as follows:

[0094] Define the tool center point at the end of the robot arm as P , find the point in the corresponding binocular camera image P , calculate the point based on the camera calibration parameters P Pixel position in camera coordinate system ;

[0095] Combined with the hand-eye calibration information, the following formula is used to calculate the point P The offset vector :

[0096] Where, Indicates the conversion relationship from the camera coordinate system to the manipulator coordinate system, the offset vector This is the terminal calibration information.

[0097] In order to improve precision and accuracy, this embodiment also discloses verifying the terminal calibration information. The verification method is as follows:

[0098] The offset vector calculated based on hand-eye calibration Obtain the TCP position, project it onto the image according to the camera calibration parameters, and obtain the theoretical pixel position of the end position on the image. If the calculated theoretical pixel position in the image coincides with the TCP position in the field of view, the calibration result is reliable. If not, it indicates that the binocular calibration parameter error is large and the binocular camera parameters need to be recalibrated.

[0099] In summary, this embodiment discloses a method for automatically capturing image information of path points, saving all information, and calibrating the valid information. By integrating binocular calibration, hand-eye calibration, and end-point TCP calibration, the efficiency and accuracy of robotic arm calibration are improved. Furthermore, end-point calibration, based on selecting TCP pixel positions, allows the theoretical pixel coordinates of the end points in the image to be obtained from the calibration results and compared with the actual positions to verify the accuracy of the calibration results. Example

[0100] This embodiment discloses a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described in Example 1 are implemented.

[0101] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in Example 1.

Claims

1. An online automatic calibration method based on a binocular camera and a robotic arm, characterized in that: At least the following steps are included: Set the robot, binocular camera and calibration block according to the predetermined positions and create the basic coordinate system respectively , Robotic arm coordinate system , camera coordinate system and calibration plate coordinate system ; The end of the robotic arm moves along a predetermined path, recording the path points during the movement; Collect binocular camera images at each path point and effectively screen the binocular camera images to obtain a valid image set; Based on the valid image set, dual-target positioning is performed to obtain the camera calibration parameters. The hand-eye calibration is performed by combining the valid image set and the camera calibration parameters to obtain the hand-eye transformation matrix between the camera coordinate system and the robot arm coordinate system, i.e., the hand-eye calibration information. The calibration process for dual-target positioning based on the valid image set is as follows: Get the coordinates of a point on the calibration plate in the world system as , and the coordinates of the point in the camera coordinate system are 、 ; Based on coordinates 、 and The rotation and translation matrices of the point relative to the left camera are calculated as follows: 、 , and the rotation and translation matrices of this point relative to the right camera are 、 ; Then, the following formulas are satisfied for this point: , calculated ; At the same time, the following formulas are satisfied: ; Then, using the formula: ; Calculation yields: 、 Where, is the rotation matrix of the right camera relative to the left camera, is the translation matrix of the right camera relative to the left camera, then the rotation matrix and translation matrices That is the camera calibration parameter; Acquire the binocular camera image of the end of the robotic arm at the endpoint, calculate the pixel position of the end of the robotic arm in the camera coordinate system, and combine it with the hand-eye calibration information to obtain the transformation relationship of the robotic arm end at the endpoint relative to the mechanical coordinate system, that is, the end calibration information. The method for obtaining the transformation relationship of the robotic arm end at the endpoint relative to the mechanical coordinate system is as follows: Define the tool center point at the end of the robot arm as P , find the point in the corresponding binocular camera image P , calculate the point based on the camera calibration parameters P Pixel position in camera coordinate system ; Combined with the hand-eye calibration information, the following formula is used to calculate the point P The offset vector : Where, Indicates the conversion relationship from the camera coordinate system to the manipulator coordinate system, the offset vector That is the terminal calibration information; Output camera calibration parameters, hand-eye calibration information, and end-point calibration information.

2. The online automatic calibration method based on a binocular camera and a robotic arm according to claim 1, characterized in that: The valid image set is obtained by the following steps: Step 201: Move along the X-axis of the robot coordinate system according to a predetermined point to obtain multiple binocular camera images about the X-axis; Step 202: Determine whether there is a calibration plate in the binocular camera image about the X axis. If so, define it as a valid image about the X axis until the calibration plate is no longer within the field of view of the binocular camera. Step 203: Count the number of valid images obtained ,like , then continue to obtain along the X axis until ; Step 204: Move along the Y-axis of the robot coordinate system according to a predetermined point to obtain multiple binocular camera images about the Y-axis; Step 205: Determine whether there is a calibration plate in the binocular camera image about the Y axis. If so, it is defined as a valid image about the Y axis until the calibration plate is no longer within the field of view of the binocular camera. Step 206: Count the number of valid images obtained ,like , then repeat steps 205 to 206 until ; Step 207: Move along the Z axis of the robot coordinate system according to a predetermined point to obtain multiple binocular camera images about the Z axis; Step 208: Determine whether there is a calibration plate in the binocular camera image along the Z axis. If so, define it as a valid image along the Z axis until the calibration plate is no longer within the field of view of the binocular camera. Step 209: Count the number of valid images obtained ,like , then repeat steps 205 to 206 until ; The valid image about the X axis, the valid image about the Y axis, and the valid image about the Z axis are stored as a valid image set, wherein, 、 and These are pre-set values.

3. The online automatic calibration method based on a binocular camera and a robotic arm according to claim 1, characterized in that: The calculation process of the hand-eye transformation matrix between the camera coordinate system and the robotic arm coordinate system is as follows: Define the robot arm in the base coordinate system If the position in is (X, Y), it is calculated by the following formula: ； ； because and The coordinate system is fixed, so Does not change, so: ； remember for , for , for , then: ; and B are known quantities, so we solve Get the hand-eye transformation matrix; Where, is the position of the point on the calibration plate relative to the robot coordinate system; Indicates the transformation relationship from the manipulator coordinate system to the base coordinate system; Indicates the transformation relationship from the camera coordinate system to the manipulator coordinate system; Indicates the transformation relationship from the camera coordinate system to the calibration plate coordinate system; Indicates the transformation relationship from the calibration plate coordinate system to the base coordinate system; is the position of the point on the calibration plate relative to the calibration plate coordinate system; Indicates the conversion relationship of the X-axis from the robot coordinate system to the basic coordinate system. Indicates the conversion relationship of the X-axis from the camera coordinate system to the manipulator coordinate system. Indicates the transformation relationship of the Y axis from the robot coordinate system to the basic coordinate system, Indicates the transformation relationship of the Y-axis from the camera coordinate system to the robot coordinate system.

4. The online automatic calibration method based on a binocular camera and a robotic arm according to claim 1, characterized in that: The terminal calibration information is also verified, and the verification method is as follows: The offset vector calculated based on hand-eye calibration Obtain the TCP position, project it onto the image according to the camera calibration parameters, and obtain the theoretical pixel position of the end position on the image. If the calculated theoretical pixel position in the image coincides with the TCP position in the field of view, the calibration result is reliable. If not, it indicates that the binocular calibration parameter error is large and the binocular camera parameters need to be recalibrated.

5. The online automatic calibration method based on a binocular camera and a robotic arm according to claim 1, characterized in that: The calibration plate is a checkerboard.

6. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images