A vision-based composite robot hand-eye calibration and workpiece positioning device and method

Abstract

The application discloses a kind of composite robot hand-eye calibration and workpiece positioning device and method based on vision, method includes: step S1: according to preset operation height, camera calibration and end tool TCP calibration are carried out to two cameras respectively;Step S2: control robot arm to reach calibration position, collect image data of hand-eye calibration;Control robot arm tool end reaches the feature point position of checkerboard calibration plate, and feature point data of hand-eye calibration is collected;Based on the image data and the feature point data, hand-eye calibration parameter solution is carried out;Step S3: teach robot arm to workpiece grasping pose, and box camera respectively collects the image of checkerboard and Aruco two-dimensional code before and after teaching, and carries out work station calibration parameter solution.The application completes hand-eye calibration in a more simple and fast way, and solves the problem of the decline of vision-guided workpiece grasping accuracy caused by chassis movement, while post-processing is carried out on subsequent workpiece grasping pose.

Description

Technical Field

[0001] This invention belongs to the field of visual positioning and industrial robots, especially hand-eye calibration and visual workpiece positioning technology, specifically involving a vision-based composite robot hand-eye calibration and workpiece positioning device and method. Background Technology

[0002] Currently, considering flexibility, the hand-eye calibration technology for composite robots generally adopts a vision solution where the eye is on the hand. This means the camera sensor moves with the robot's operation. Considering the typical hand-eye calibration and workstation positioning process for fixed-base robots: it usually requires moving the robotic arm to different poses and simultaneously acquiring image data from a high-precision calibration board and the robotic arm's posture data to calculate the calibration parameters for the hand-eye pose. Subsequent workpiece positioning requires setting up a working area, with the working area and calibration board fixed in position. Accurately obtaining the actual three-dimensional offset between the working area and the calibration board is necessary—this is the workstation calibration. In practice, simply re-capturing the calibration board image and, based on the transformations of the hand-eye and robotic arm matrices, adding the known workstation calibration offset, allows the robot to accurately grasp the workpiece.

[0003] For composite robots, if hand-eye calibration is performed at the work point using the same sampling pose method, the robotic arm end effector needs to be adjusted to multiple different poses at the work point. This requires a large workspace. However, in real production conditions, the calibration board needs to be placed in a small, enclosed environment. Multiple sampling poses may cause collisions with the protective walls of the workpiece storage box. Furthermore, the calibration method using multiple sampling poses is often time-consuming and complex. Additionally, composite robots require an extra step before actual visual workstation localization: the mobile chassis must first be navigated to the work point before the grasping task can be performed. Due to radar or sensor errors, coupled with uneven ground conditions at the work point, the pose of the mobile chassis at the work point will differ from the position and attitude measured during calibration. Since traditional workpiece localization methods read the TCP offset before and after teaching, only considering translation in three-dimensional space and not rotation, adding offsets based on the calibration pose ignores the pose error caused by inaccurate navigation. This can lead to a discrepancy between the calculated and actual workpiece positions, significantly reducing the robotic arm's grasping accuracy. Furthermore, considering environmental factors such as the ground at the actual work point, the chassis's position after docking is unstable. After the robotic arm reaches the calculated workpiece positioning point, the chassis may occasionally shift, introducing further errors. Current positioning technologies cannot handle these occasional errors. Additionally, the accuracy of the robotic arm's grasping posture at the workstation needs to be evaluated. Current technologies assume precise arrival and do not consider state detection of the robotic arm's grasping posture upon arrival at the workstation.

[0004] In the prior art, publication number CN 110842928 A, publication date: 2020.02.28, relates to a composite robot vision-guided positioning device and method. The parameter used for station calibration is the TCP offset before and after teaching. It only considers positional quantities and cannot autonomously handle the error introduced by navigation offset. Therefore, it requires adding multiple end-effector posture detection sensors, increasing hardware consumption and algorithm processing complexity. Publication number CN 115609591... A, Publication (Announcement) Date: January 17, 2023, relates to a visual positioning method and system based on 2D Markers. The workstation calibration method uses a teaching approach to reach the workstation grasping pose. It obtains the rigid transformation matrix of the workstation coordinate system relative to the Marker coordinate system by reading the TCP terminal pose. This matrix is ​​then used for workpoint repositioning and grasping. Theoretically, this scheme can accurately obtain the workstation coordinate system pose after repositioning. However, during the actual control of the robotic arm's movement, due to environmental factors at the workpoint in actual production, the robot body may occasionally experience slight swaying. The workstation coordinate system calculated using the pre-calibrated rigid transformation matrix will have errors compared to the actual coordinate system, and this method cannot handle such errors. Furthermore, this method cannot determine whether the moved robotic arm has truly reached the workpiece grasping pose at the calibration time, and it does not consider the pose accuracy issue between the actual reached workpiece grasping pose and the calibrated workpiece grasping pose. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a vision-based composite robot hand-eye calibration and workpiece positioning device and method, which completes hand-eye calibration in a simpler and faster manner, solves the problem of decreased accuracy in vision-guided workpiece grasping caused by chassis movement, and performs post-processing on the subsequent workpiece grasping posture.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A vision-based composite robot hand-eye calibration and workpiece positioning device includes: a composite robot and a vision gripper;

[0008] The composite robot includes a mobile chassis and a robotic arm. The base of the robotic arm is fixed to the mobile chassis and moves with the mobile chassis.

[0009] The vision gripper includes a vision sensor, a TCP tool end effector and an end gripper, an Aruco code, an end connector, and an end camera. The end camera, TCP tool end effector, and end connector are all bolted together, while the Aruco code is attached to the back plate of the end connector using adhesive. The end gripper and end connector are bolted together to the end flange of the robotic arm.

[0010] It also includes: protective case, camera inside the case, workpiece holder, and checkerboard calibration plate;

[0011] The protective case is the outer shell of the workbench at the work point and is fixed to the workbench; the camera inside the case is located at the top inside the protective case and is fixed by bolts; the checkerboard calibration plate is fixedly placed at the edge of the workbench, and the workpiece holders are arranged in sequence at the center of the workbench.

[0012] Preferably, the vision gripper is connected to the end flange of the robotic arm by an "L"-shaped mounting piece; the end camera is bolted to one end of the "L"-shaped block, and the other end is bolted to the end flange of the robotic arm; the end gripper is installed along the vertical line of the flange face, and the TCP tool end is set between the gripper tool and the camera.

[0013] This invention also provides a vision-based method for composite robot hand-eye calibration and workpiece positioning, implemented using the aforementioned vision-based composite robot hand-eye calibration and workpiece positioning device, comprising:

[0014] Step S1: Perform camera calibration and end-effector TCP calibration for the two cameras according to the preset working height;

[0015] Step S2: Control the robotic arm to reach the calibration position and collect image data for hand-eye calibration; control the end effector of the robotic arm to reach the feature point position of the checkerboard calibration board and collect feature point data for hand-eye calibration; calculate the hand-eye calibration parameters based on the image data and the feature point data.

[0016] Step S3: Teach the robotic arm to the workpiece grasping pose. The camera inside the box captures the chessboard and Aruco QR code images before and after teaching, and calculates the station calibration parameters.

[0017] Preferably, in step S2, controlling the robotic arm to reach the calibration position and collecting image data for hand-eye calibration; controlling the end effector of the robotic arm to reach the feature point position of the checkerboard calibration board and collecting feature point data for hand-eye calibration; the method for calculating hand-eye calibration parameters based on the image data and the feature point data includes:

[0018] Step S21: Move the composite robot to the working point, control the end of the robotic arm to reach the position of the camera calibration height, and adjust the end posture to vertically shoot the calibration board. Record the current posture of the robotic arm as the hand-eye calibration posture. Under the hand-eye calibration posture, acquire the image of the chessboard calibration board.

[0019] Step S22: Control the end effector of the robotic arm to reach the corner feature points of the checkerboard calibration board, and use the tip of the needle to sequentially poke the preset m feature points of the checkerboard calibration board in a clockwise order. During the poke process, record the m spatial coordinates in the base coordinate system of the robotic arm.

[0020] Step S23: Calculate the hand-eye calibration parameters based on the collected image of the chessboard calibration board and the feature point data of the chessboard calibration board.

[0021] Preferably, in step S23, the method for calculating hand-eye calibration parameters based on the acquired image and the feature point data of the chessboard calibration board includes:

[0022] Corner detection is performed on the image data of the chessboard calibration board to obtain the pixel coordinates of n corner points, i.e., P. i Cam1 (u i v i ), i = 1, 2, ..., n;

[0023] The top left corner of the calibration board is set as the origin of the calibration board coordinate system. Based on the known size and number of chessboard squares, the world coordinates of all corner points, i.e., P, are constructed. i Cal (X i ,Y i Z i ), i = 1, 2, ..., n;

[0024] By mapping the pixel coordinates of all corner points to their world coordinates and combining them with the intrinsic parameter matrix K obtained from camera calibration, the PnP algorithm is used to calculate the calibration board coordinate system O under the hand-eye calibration posture. Cal Relative to camera coordinate system O Cam1 The rigid transformation matrix under The expression is:

[0025]

[0026] The checkerboard calibration board is pre-defined with m corner points as feature points, and the coordinates of these m corner points in the checkerboard calibration board coordinate system are pre-defined.

[0027] According to the rigid transformation matrix The coordinates of m corner points in the coordinate system of the chessboard calibration board. Transform to camera coordinate system to obtain

[0028] The point set is obtained by using the tool's end pose data as it sequentially pokes the checkerboard feature points using the obtained TCP endpoint.

[0029] based on Hedianji The constructed matrix relationship is as follows:

[0030]

[0031] The matrix relationship is solved using the SVD eigenvalue decomposition method to obtain the required hand-eye matrix.

[0032] Using the SVD method to obtain the required hand-eye matrix Perform UV eigenvalue decomposition to obtain the rotation matrix.

[0033]

[0034] Substitute the center point of the point set into the calculation to obtain the position. The expression is:

[0035] in, The centroid of the feature point in the calibration plate coordinate system. The centroid of the feature point in the camera coordinate system.

[0036] Preferably, in step S3, the method for calculating the workstation calibration parameters by teaching the robotic arm to the workpiece grasping pose, and having the in-box camera capture the checkerboard and Aruco QR code images before and after teaching, includes:

[0037] Step S31: With the chassis of the composite robot remaining in the same position, control the end effector of the robotic arm to return to the hand-eye coordinate positioning posture, and the end effector camera and the camera inside the box will respectively capture images of the chessboard calibration board;

[0038] Step S32: Teach the robotic arm to reach above the object, adjust the end effector pose according to the appropriate grasping posture, and have the camera inside the box take a picture of the Aruco QR code at the end of the robotic arm.

[0039] Step S33: Calculate the station calibration parameters based on the checkerboard calibration board image captured by the two cameras, the TCP end pose data when the robotic arm moves to the workstation, and the Aruco QR code image.

[0040] Preferably, in step S33, the method for calculating the workstation calibration parameters based on the checkerboard calibration board image captured by the two cameras, the TCP end-effector pose data when the robotic arm moves to the workstation pose, and the Aruco QR code image includes:

[0041] PnP algorithm is performed on the checkerboard images captured by two cameras under hand-eye pose conditions to obtain the checkerboard calibration plate O. Cal Rotation matrix relative to the coordinate systems of the end camera and the in-box camera and in It is a constant;

[0042] PnP decoding was performed on the Aruco QR code image captured by the camera inside the box under the calibration posture at the workstation to obtain the transformation matrix between the Aruco coordinate system and the camera coordinate system inside the box. The pose of the tool tip is acquired synchronously and represented by the workstation coordinate system O. Work Relative to the robot arm's base coordinate system O Base rigid transformation

[0043] Adjust the robotic arm to the hand-eye calibration posture and solve for the hand-eye matrix. Repeated application, based on rotation matrix and Transformation matrix and rigid transformation The coordinate system transformation formula for obtaining the workstation calibration pose is:

[0044] in, It is a constant rigid transformation matrix. The Aruco coordinate system calibrated for the workstation is relative to the standard pose of the camera inside the box.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] This invention significantly reduces the time required for hand-eye calibration of the composite robot, simplifies the calibration process, and minimizes the risk of collisions with environmental objects during pose acquisition. Furthermore, the hand-eye calibration maintains high accuracy. In addition, the workstation calibration method of this invention fully considers the rigid transformation between coordinate systems, allowing the workpiece to be fixed to the calibration plate in any orientation. The calibration method is entirely algorithmic, eliminating the need for high-precision auxiliary positioning devices. Moreover, by detecting the calibration plate's pose, this invention can calculate the repositioning pose error of the mobile chassis at the work point, solving the problem of workpiece position calculation errors caused by inaccurate navigation and greatly improving the accuracy of the composite robot's workpoint repositioning and workpiece grasping. Simultaneously, by detecting the Aruco code pose, this invention performs secondary repositioning calculations on the end effector of the robotic arm that has moved to the workpiece grasping pose, thus resolving occasional chassis body misalignment issues. The secondary repositioning results can be used to compensate for errors in the end effector of the robotic arm. Attached Figure Description

[0047] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram illustrating the components of an exemplary embodiment of the composite robot hand-eye calibration and workstation positioning device according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram illustrating a practical application of the composite robot's hand-eye calibration and workstation positioning in a secondary positioning embodiment of the present invention.

[0050] Figure 3 This is a calibration flowchart of an exemplary embodiment of the composite robot hand-eye calibration and workstation positioning method according to an embodiment of the present invention;

[0051] Figure 4 This is a flowchart illustrating the overall working application of the composite robot hand-eye calibration and workstation positioning method as an exemplary embodiment of the present invention.

[0052] Figure 5 This is a flowchart illustrating the calibration steps of an exemplary embodiment of the composite robot hand-eye calibration and workstation positioning method according to an embodiment of the present invention.

[0053] Figure 6 This is a flowchart illustrating the actual working steps of the vision-based composite robot workstation positioning method, which is an exemplary embodiment of the present invention.

[0054] Figure 7 This is a schematic diagram illustrating the acquisition of chessboard feature points during calibration in the composite robot hand-eye calibration and workstation positioning method of this embodiment of the invention.

[0055] Figure 8 This is a flowchart illustrating the specific calibration steps of an exemplary embodiment of the composite robot hand-eye calibration and workstation positioning method of the present invention.

[0056] Figure 9 This is a flowchart illustrating the specific steps of a vision-based composite robot workstation positioning method as an exemplary embodiment of the present invention in its practical application.

[0057] Explanation of the labels in the diagram: 1-Protective box, 2-Camera inside the box, 3-Aruco code, 4-End connector, 5-End camera, 6-TCP tool end effector, 7-Workpiece holder, 8-End gripper, 9-Checkerboard calibration plate, 10-Mobile chassis, 11-Robotic arm Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] Example 1

[0061] like Figure 1 , Figure 2 As shown, this embodiment provides a vision-based composite robot hand-eye calibration and workpiece positioning device, including: a composite robot and a vision gripper;

[0062] The composite robot includes a mobile chassis 10 and a robotic arm 11. The base of the robotic arm 11 is fixed on the mobile chassis 10 and moves with the mobile chassis 10.

[0063] The vision gripper includes a vision sensor, a TCP tool end effector 6, an end gripper 8, an Aruco code 3, an end connector 4, and an end camera 5. The end camera 5, the TCP tool end effector 6, and the end connector 4 are all bolted together, while the Aruco code 3 is fixed to the back plate of the end connector 4 by adhesive. The end gripper 8 and the end connector 4 are both bolted together to the end flange of the robotic arm 11.

[0064] It also includes: 1 protective case, 2 camera inside the case, 7 workpiece holding box, and 9 checkerboard calibration plate;

[0065] The protective box 1 is the outer shell of the workbench at the work point and is fixed to the workbench; the camera 2 inside the box is located at the top inside the protective box 1 and is fixed by bolts; the checkerboard calibration plate 9 is fixedly placed at the edge of the workbench, and the workpiece holding box 7 is arranged in sequence at the center of the workbench.

[0066] Specifically, the vision gripper is connected to the end flange of the robotic arm 11 by an "L"-shaped mounting piece; the end camera is bolted to one end of the "L"-shaped block, and the other end is bolted to the end flange of the robotic arm 11. The end gripper is installed along the vertical line of the flange face, and the TCP tool end 6 is set between the gripper tool and the camera.

[0067] Example 2

[0068] like Figure 3 , Figure 4As shown, this embodiment provides a vision-based method for hand-eye calibration and workpiece positioning of a composite robot, including the following steps: Figure 5 As shown:

[0069] Step S1: Perform camera calibration and end-effector TCP calibration for the two cameras according to the preset working height;

[0070] Step S2: Control the robotic arm 11 to reach the calibration position, collect the hand-eye calibration image data, control the end effector of the robotic arm to reach the feature point position of the calibration plate, collect the feature point data of the hand-eye calibration, and calculate the hand-eye calibration parameters.

[0071] Step S3: Teach the robotic arm 11 to the workpiece gripping pose, and the camera 2 inside the box collects the chessboard and Aruco QR code images before and after teaching, respectively, and calculates the station calibration parameters.

[0072] Specifically, such as Figure 8 As shown, in step S1, the working distance of the robotic arm 11 is preset. At this distance, the camera on the end effector is calibrated to obtain the camera's intrinsic parameters and distortion parameters. The optical center of the camera is then set as the coordinate system O of the end effector camera 5. Cam1 Simultaneously, the camera inside the protective box 1 is also calibrated. The calibration working distance is the height between the checkerboard calibration plate 9 and the camera, and the optical center of the camera inside the protective box 1 is set as the coordinate system O of the camera 2 inside the box. Cam2 The purpose of calibrating the two cameras separately is to ensure that the images captured by the subsequent cameras are clearly visible; the end effector calibration of the robotic arm is then performed, and the coordinate system of the calibrated composite robot end effector is set to O. Tool ;

[0073] The method for calibrating the camera at the end effector by first setting the working distance of the robotic arm 11 is as follows: first, roughly set the working distance of the camera, place a calibration plate on the working surface, move the robotic arm to raise the camera to the preset working distance, adjust the camera focal length so that the calibration plate is clearly visible in the camera, and then complete the camera calibration operation.

[0074] Internal parameters and distortion parameters refer to the data obtained after camera calibration.

[0075] End-effector calibration, also known as TCP calibration, involves the following steps: First, a needle tip is placed on the work surface and fixed in place. Then, a moving robotic arm is used to make the end-effector contact the needle tip in different postures, and the pose information at the contact is recorded synchronously. Then, TCP calibration calculations are performed, and finally, the rigidity transformation matrix data of the end-effector relative to the robotic arm flange is output. Here, the calibration refers to the robotic arm end-effector, which can also be understood as the end-effector of a composite robot, since the components of a composite robot are a moving chassis and a robotic arm. For the sake of clarity, it can also be referred to as the robotic arm end-effector.

[0076] Step S21: Perform mapping and set the working point position for workpiece gripping. After moving the composite robot to the working point, set the base coordinate system of the composite robot arm to O. Base The robotic arm 11 is controlled to reach approximately the camera calibration height position, and its end effector is adjusted to vertically capture the checkerboard calibration board 9. Fine adjustments are made to ensure the calibration board is centered in the image and occupies 2 / 3 of the image's field of view. The upper left feature point of the checkerboard calibration board 9 is set as the calibration board coordinate system O. Cal Simultaneously record the current pose of the robotic arm And record it as the hand-eye calibration posture of the robotic arm;

[0077] Step S22: Acquire images of the calibration board, control the end effector of the robotic arm to gradually reach the preset feature points of the calibration board, and simultaneously record the pose of the TCP end effector at the feature points;

[0078] Specifically, image acquisition of the checkerboard calibration board 9 needs to be performed while maintaining the robotic arm's hand-eye calibration posture as described in step S21. After capturing the calibration board data, refer to... Figure 7 The system collects 9 feature points from a checkerboard calibration board. The needle tip is instructed to sequentially poke the pre-set m checkerboard feature points in a clockwise direction, recording the m spatial coordinates in the robotic arm's base coordinate system during the poke process. The collected image data and tool end data at feature points are used in step S23 to calculate hand-eye calibration parameters.

[0079] Step S23: Calculate the hand-eye calibration parameters based on the acquired image and the feature point data of the TCP terminal tool reaching the calibration board;

[0080] Specifically, the hand-eye calibration parameter calculation process can be divided into two parts: The first part of the algorithm is as follows: First, corner point detection is performed on the image data of the chessboard calibration board 9 to obtain the pixel coordinates of n corner points, which is P. i Cam1 (u i v i), i = 1, 2, ..., n, with the top left corner of the calibration board as the origin of the calibration board coordinate system. Based on the known size and number of chessboard squares, the world coordinates of all corner points are constructed, i.e., P. i Cal (X i ,Y i Z i (i = 1, 2, ..., n), map these two types of coordinate points one-to-one and combine them with the intrinsic parameter matrix K obtained from camera calibration. Then, use the PnP algorithm to calculate the calibration board coordinate system O under the hand-eye calibration posture. Cal Relative to camera coordinate system O Cam1 The rigid transformation matrix under Its matrix relation is:

[0081]

[0082] The second part of the algorithm is as follows: Pre-set m corner points on the chessboard calibration board 9 as feature points, and know the coordinates of these points in the calibration board coordinate system. Based on the calculated extrinsic parameter matrix Transform this set to the camera coordinate system to obtain Based on the tool end pose data obtained in step S23 when the TCP end sequentially pokes the chessboard feature points, the point set is obtained. The constructed matrix relationship is as follows:

[0083]

[0084] Then, using the SVD eigenvalue decomposition method, we obtain linear solutions for these two point sets. These linear solutions are the required hand-eye matrix. The calculation process is as follows:

[0085] set up Where j = m, representing the m feature points of the chessboard calibration board in the chessboard coordinate system O. Cal Let the three-dimensional coordinates on the surface be... Where j = m, which represents the position of m feature points on the chessboard calibration board in the coordinate system O of the end camera 5. Cam1 First, determine the centroid of the feature point sets in these two coordinate systems using the three-dimensional coordinates:

[0086]

[0087] Decentralize the two sets of points, that is, subtract the center point from each set of points. The decentralized set of points still satisfies the matrix relationship mentioned above, i.e.:

[0088]

[0089] Calculate the covariance matrix between the two sets of points:

[0090]

[0091] The SVD method is used to perform UV eigenvalue decomposition on the covariance matrix, and the rotation matrix of the two point sets is calculated.

[0092]

[0093] Finally, the center point of the point set and the rotation matrix obtained above are used. Substitute into the following formula to calculate the position.

[0094]

[0095] Step S23, the calculation of hand-eye calibration parameters, requires two sets of data: image data of the chessboard calibration board under hand-eye calibration posture and tool end pose data when the TCP end sequentially pokes the chessboard feature points; the output of hand-eye calibration is the hand-eye calibration matrix. The hand-eye calibration matrix here is not the traditional camera coordinate system O with the eye on the hand. Cam1 Relative to the end effector coordinate system O of the robotic arm End Hand-eye matrix matrix The coordinate system of the composite robot camera is O. Cam1 Relative to the composite robot base coordinate system O Base The rigid transformation, which encompasses the transformation of the robot arm's end-effector coordinate system O End Indirect conversion.

[0096] Step S31 keeps the composite robot's position at the working point unchanged. Considering that in this method, the hand-eye calibration matrix is ​​represented by the composite robot's camera coordinate system as O... Cam1 Relative to the composite robot base coordinate system O Base The rigid transformation of the mechanical arm tool end is used to return it to the hand-eye coordinate system's positioning pose, ensuring that the relationship between the camera and the mechanical arm base remains unchanged. (Hand-eye matrix) It can be applied repeatedly, with both the end camera and the in-box camera 2 acquiring images of the checkerboard calibration plate;

[0097] Specifically, re-capturing the checkerboard image is to solve for new camera extrinsic parameters, that is, to calculate the calibration board coordinate system O under the hand-eye calibration posture when re-reaching the working point position. Cal Relative to the end camera 5 coordinate system O Cam1 The rigid transformation matrix under Before moving the robotic arm 11, the camera inside the box is opened to take a picture of the chessboard calibration plate 9. The purpose of this is to calculate the coordinate system O of the chessboard calibration plate. Cal Relative to the camera's 2-coordinate system OCam2 The rigid transformation matrix under Since the checkerboard calibration plate 9 is fixedly installed inside the protective case 1, and the camera 2 inside the case is also fixedly installed on top of the protective case 1, therefore In subsequent workstation calibration and actual positioning and capture, these are constants and are known;

[0098] Step S32: Fix an Aruco QR code to the end effector connector 4 of the robotic arm. The pose of the Aruco code 3 will change with the pose of the robotic arm end effector. Set the center of the planar QR code as the Aruco coordinate system O. Arc The teaching control robot arm 11 moves above the object and adjusts its end effector pose to a reasonable grasping posture. At the same time, the tip of the tool end is set as the workstation coordinate system O. Work Open the camera 2 inside the box and take a picture of the Aruco QR code at the end of the robotic arm;

[0099] Step S33 calculates the workstation calibration parameters based on the checkerboard image captured by the two cameras and the Aruco QR code image when the robotic arm 11 moves to the workstation pose.

[0100] Specifically, step S33 involves calculating the station calibration parameters, which is the process of obtaining the station calibration parameters: The process includes: firstly, performing PnP calculations on the checkerboard images captured by two cameras under hand-eye posture to obtain the checkerboard calibration plate O. Cal Rotation matrix relative to the coordinate systems of end camera 5 and in-box camera 2 and in As constant; continue to perform PnP calculation on the Aruco QR code image captured by camera 2 inside the box under the calibration posture of the workstation, and obtain the transformation matrix between the Aruco coordinate system and the coordinate system of camera 2 inside the box. The pose of the tool tip is acquired synchronously and represented by the workstation coordinate system O. Work Relative to the robot arm's base coordinate system O Base rigid transformation Since the robotic arm 11 has been adjusted to the hand-eye calibration posture in step S31, the hand-eye matrix solved in step S23 is... This can be applied repeatedly. The coordinate system transformation formula under the station calibration pose is derived as follows:

[0101]

[0102] By combining the solved hand-eye calibration parameters, we can proceed according to the formula. Solve therefore For known quantities, For known quantities, constant, It is a solveable quantity; the rotation matrix can be derived. Its representative workstation coordinate system O Work Relative to the Aruco coordinate system O Arc Due to the rigid transformation, the station calibration parameters have been obtained, including: The Aruco code 3 coordinate system calibrated for the workstation, relative to the standard pose of the camera 2 inside the box, is determined through matrix transformation relationships. The workstation coordinate system O can be obtained indirectly. Work Compared to the camera inside the box O Cam2 The pose data is obtained, while the pose relationship between the camera 2 inside the box and the workpiece remains unchanged. Therefore, subsequent steps only need to ensure... Once the workstation returns to its calibrated state, precise grasping can be achieved, and because... Since they are all constants, we only need to ensure that Returning to the workstation calibration state is sufficient, as this is crucial for minimizing errors when reaching the workpiece gripping pose after subsequent quantization and repositioning.

[0103] Furthermore, the actual workpiece gripping process includes: such as Figure 6 As shown:

[0104] Step S4: Navigate the chassis to the working point position and control the robotic arm 11 to the hand-eye calibration position;

[0105] Step S5: The end camera 5 captures a picture of the checkerboard calibration board and calculates the pose of the calibration board. The hand-eye matrix and the workstation calibration pose parameters are combined to calculate the pose of the workstation coordinate system.

[0106] In step S6, the end effector of the robotic arm is controlled to reach the calculated workstation pose. The camera 2 inside the box takes pictures of the Aruco code 3 at the end effector of the robotic arm again and calculates the pose, and calculates the error value between the Aruco code 3 pose and the workstation calibration.

[0107] Step S7 processes the difference between the actual Aruco code 3-position. If the error exceeds the offset error threshold, it is considered that the vehicle body has been offset and needs to be recalculated.

[0108] Step S8 continues to determine whether the difference between the 3-position Aruco code and the precise grasping error threshold is met. If it is met, the workpiece can be precisely grasped; otherwise, the end-effector pose is adjusted within a small range to compensate for the error.

[0109] Specifically, such as Figure 9 As shown, the actual workstation grasping operation of the vision-based composite robot includes the following steps:

[0110] Step S41: The composite robot navigates to the working point set in the map, adjusts the robotic arm posture to the hand-eye positioning posture, and ensures that the relationship between the end-effector camera 5 and the robotic arm base remains unchanged.

[0111] Step S51: Open the end-effector camera 5 to acquire images of the checkerboard calibration board. Preprocess the image using algorithms such as grayscale conversion, threshold segmentation, and corner detection to obtain a new set of checkerboard corner pixel coordinates. Then, recalculate the pose of the checkerboard calibration board 9 relative to the camera using the PnP algorithm.

[0112] The workpiece position calibration parameters relative to the calibration plate obtained from step S52, which involves calibrating the workpiece together, are as follows: Perform pose calculation of the workstation coordinate system relative to the robot arm base;

[0113] Specifically, the calculated pose matrix is: As can be seen from the workstation calibration in step 32, among which... All are constants. These are the standard quantities for workstation calibration, where the only variable is the chassis working point repositioning error caused by navigation errors. This type of error can be determined according to step S51. To quantify, therefore, based on the coordinate system transfer relationship:

[0114]

[0115] Once the pose of the repositioned workstation coordinate system in the robot arm base coordinate system is obtained, workpiece positioning is achieved.

[0116] Step S61: Control the robotic arm tool end effector to reach the calculated workstation grasping pose, open the camera 2 inside the box to take another picture of the Aruco code 3 on the robotic arm end effector in the workstation grasping pose state, and calculate the pose of the Aruco code 3 coordinate system in the camera coordinate system when reaching the workstation calibration pose this time. Simultaneously calculate its relationship with the standard pose during workstation calibration. The difference between them;

[0117] Step S71 first presets the offset threshold, and then... When the workstation is marked The rotational component is converted into Euler angles. The deviations of the Euler angles and position data are checked to see if they are within a preset offset threshold. If they exceed this threshold, it is considered that after recalculating the workstation coordinate system, during the process of controlling the robotic arm 11 to reach the workstation and grasp the pose, the vehicle body became unstable and shifted. This accidental shift caused the pose of the checkerboard calibration plate 9 relative to the camera to be affected. If a change occurs, the robotic arm 11 is adjusted back to the hand-eye calibration posture, the end effector camera 5 is reopened to photograph the checkerboard calibration board 9, and the actual offset is calculated. After updating the rotation relationship, continue to perform new station gripping pose calculations by combining the station calibration parameters;

[0118] Step S81: When the difference meets the offset threshold, it is considered that the car body has not moved. To ensure accurate grasping, an accurate grasping threshold needs to be set, mainly to address the error caused by the repetition error of the robotic arm 11 and other factors. Continue to judge whether the deviation value is within the preset accurate grasping threshold range. If it exceeds the threshold, adjust the end pose of the robotic arm according to the Euler difference and position difference feedback, and perform error compensation on the actual workstation pose. After the adjustment is completed, the camera 2 inside the box repeats the detection process for secondary positioning. When the threshold requirement is met, the accurate grasping of the workpiece is completed.

[0119] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A vision-based method for hand-eye calibration and workpiece positioning in a composite robot, characterized in that, The method includes: Step S1: Perform camera calibration for the end camera and the camera inside the box, as well as TCP tool end calibration, according to the preset working height; Step S2: Control the robotic arm to reach the calibration position and collect the hand-eye calibration image data; control the TCP tool end to reach the feature point position of the chessboard calibration board and collect the feature point data of the hand-eye calibration; calculate the hand-eye calibration parameters based on the image data and the feature point data. Step S3: Teach the robotic arm to the workpiece grasping pose, and the camera inside the box captures the chessboard and Aruco QR code images before and after teaching, respectively, and calculates the station calibration parameters; In step S2, the robotic arm is controlled to reach the calibration position and collect image data for hand-eye calibration; the TCP tool end effector is controlled to reach the feature point position of the chessboard calibration board and collect feature point data for hand-eye calibration; the method for calculating hand-eye calibration parameters based on the image data and the feature point data includes: Step S21: Move the composite robot to the working point, control the end of the robotic arm to reach the position of the camera calibration height, and adjust the end posture to vertically shoot the calibration board. Record the current posture of the robotic arm as the hand-eye calibration posture. Under the hand-eye calibration posture, acquire the image of the chessboard calibration board. Step S22: Control the TCP tool tip to reach the corner feature point positions of the chessboard calibration board, and use the needle tip to sequentially poke the preset m feature point positions of the chessboard calibration board in a clockwise order. During the poke process, record the m spatial coordinates in the robot arm base coordinate system. ; Step S23: Calculate the hand-eye calibration parameters based on the collected image of the chessboard calibration board and the feature point data of the chessboard calibration board; In step S23, the method for calculating hand-eye calibration parameters based on the acquired image and the feature point data of the chessboard calibration board includes: Corner detection is performed on the image data of the chessboard calibration board to obtain the pixel coordinates of n corner points. ; The top left corner of the calibration board is set as the origin of the calibration board coordinate system. Based on the known size and number of chessboard squares, the world coordinates of all corner points are constructed. ; By mapping the pixel coordinates of all corner points to their world coordinates and combining them with the intrinsic parameter matrix K obtained from camera calibration, the PnP algorithm is used to calculate the calibration board coordinate system under the hand-eye calibration posture. Relative to camera coordinate system The rigid transformation matrix under The expression is: ； The checkerboard calibration board is pre-defined with m corner points as feature points, and the coordinates of these m corner points in the checkerboard calibration board coordinate system are pre-defined. ; According to the rigid transformation matrix Set the coordinates of m corner points in the coordinate system of the chessboard calibration board. Transform to camera coordinate system to obtain ; Based on the TCP tool terminal pose data obtained when it sequentially pokes the chessboard feature points, the point set is obtained. ; based on Hedianji The constructed matrix relationship is as follows: ； The matrix relationship is solved using the SVD eigenvalue decomposition method to obtain the required hand-eye matrix. , Using the SVD method to obtain the required hand-eye matrix Perform UV eigenvalue decomposition to obtain the rotation matrix. : ； Substitute the center point of the point set into the calculation to obtain the position. The expression is: ,in, The centroid of the feature point in the calibration plate coordinate system. The centroid of the feature point in the camera coordinate system.

2. The vision-based composite robot hand-eye calibration and workpiece positioning method according to claim 1, characterized in that, In step S3, the method for calculating the workstation calibration parameters includes teaching the robotic arm to the workpiece grasping pose, and having the in-box camera capture chessboard and Aruco QR code images before and after teaching. Step S31: With the chassis of the composite robot remaining in the same position, control the TCP tool end effector to return to the hand-eye coordinate positioning pose, and both the end effector camera and the camera inside the box will acquire images of the chessboard calibration board. Step S32: Teach the robotic arm to reach above the object, adjust the end effector pose according to the appropriate grasping posture, and have the camera inside the box take a picture of the Aruco QR code at the end of the robotic arm. Step S33: Calculate the station calibration parameters based on the checkerboard calibration board image collected by the end-effector camera and the in-box camera, the TCP tool end-effector pose data when the robotic arm moves to the workstation pose, and the Aruco QR code image.

3. The vision-based composite robot hand-eye calibration and workpiece positioning method according to claim 2, characterized in that, In step S33, the method for calculating the workstation calibration parameters based on the checkerboard calibration board image collected by the end-effector camera and the in-box camera respectively, the TCP tool end-effector pose data when the robotic arm moves to the workstation pose, and the Aruco QR code image includes: PnP algorithm is performed on the checkerboard images acquired by the end-effector camera and the in-box camera under hand-eye posture to obtain the checkerboard calibration board. Rotation matrix relative to the coordinate systems of the end camera and the in-box camera and ,in It is a constant; PnP decoding was performed on the Aruco QR code image captured by the camera inside the box under the calibration posture at the workstation to obtain the transformation matrix between the Aruco coordinate system and the camera coordinate system inside the box. The pose of the TCP tool's terminal needle tip is synchronously acquired and represented as a workstation coordinate system. Relative to the robot arm's base coordinate system rigid transformation ; Adjust the robotic arm to the hand-eye calibration posture and solve for the hand-eye matrix. Repeated application, based on rotation matrix and Transformation matrix and rigid transformation The coordinate system transformation formula under the workstation calibration pose is: ,in, It is a constant rigid transformation matrix. The Aruco coordinate system calibrated for the workstation is positioned relative to the standard pose of the camera inside the box.

4. The vision-based composite robot hand-eye calibration and workpiece positioning method according to claim 1, characterized in that, The method is implemented based on a vision-based composite robot hand-eye calibration and workpiece positioning device, the device comprising: a composite robot and a vision gripper; The composite robot includes a mobile chassis and a robotic arm. The base of the robotic arm is fixed to the mobile chassis and moves with the mobile chassis. The vision gripper includes a TCP tool end, an end gripper, an Aruco QR code, an end connector, and an end camera. The end camera, TCP tool end, and end connector are all bolted together, while the Aruco QR code is fixed to the back plate of the end connector by adhesive. The end gripper and end connector are bolted to the end flange of the robotic arm. It also includes: protective case, camera inside the case, workpiece holder, and checkerboard calibration plate; The protective case is the outer shell of the workbench at the work point and is fixed to the workbench; the camera inside the case is located at the top inside the protective case and is fixed by bolts; the checkerboard calibration plate is fixedly placed at the edge of the workbench, and the workpiece holders are arranged in sequence at the center of the workbench.

5. The vision-based composite robot hand-eye calibration and workpiece positioning method according to claim 1, characterized in that, The vision gripper is connected to the end flange of the robotic arm by an "L"-shaped mounting bracket; the end camera is bolted to one end of the "L"-shaped mounting bracket, and the end gripper is installed along the vertical line of the end flange face of the robotic arm. The TCP tool end is positioned between the end gripper and the end camera.

Images