Vision-based composite robot hand-eye calibration and workpiece positioning device and method

Abstract

The invention discloses a vision-based composite robot hand-eye calibration and workpiece positioning device and method. The method comprises the following steps: S1, performing camera calibration and end tool TCP calibration on two cameras according to a preset operation height; s2, the mechanical arm is controlled to reach a calibration position, and image data of hand-eye calibration are collected; the tail end of the mechanical arm tool is controlled to reach the characteristic point position of the checkerboard calibration plate, and characteristic point data of hand-eye calibration are collected; calculating hand-eye calibration parameters based on the image data and the feature point data; and S3, the mechanical arm is demonstrated to reach the workpiece grabbing pose, the camera in the box collects the checkerboard image and the Aruco two-dimensional code image before and after demonstration, and station calibration parameter calculation is carried out. Hand-eye calibration is completed in a simpler, more convenient and quicker mode, the problem that the vision-guided workpiece grabbing precision is reduced due to movement of the chassis is solved, and meanwhile post-processing is conducted on subsequent workpiece grabbing poses.

Description

Technical Field

[0001] The invention belongs to the field of visual positioning and industrial robots, in particular to hand-eye calibration and visual workpiece positioning technology, and specifically relates to a composite robot hand-eye calibration and workpiece positioning device and method based on vision. Background Art

[0002] At present, under the premise of considering flexibility, the hand-eye calibration technology of composite robots generally adopts the eye-in-hand vision solution, that is, the camera sensor will move with the operation of the robot. If the hand-eye calibration and workstation positioning process of ordinary fixed-base robots is considered: usually it is necessary to move the robotic arm to different postures first, and synchronously collect the image data of the high-precision calibration plate and the posture data of the robotic arm, so as to calculate the calibration parameter data of the hand-eye posture; subsequent workpiece positioning also requires setting up a working area, the position of the working area and the calibration plate is fixed, and the actual three-dimensional offset between the working area and the calibration plate needs to be accurately obtained, that is, the workstation calibration; in actual work, it is only necessary to re-shoot the image of the calibration plate, according to the transformation of the hand-eye and robotic arm matrix, and then add the known workstation calibration offset, the robot can accurately grasp the workpiece.

[0003] For composite robots, if the hand-eye calibration is performed at the work point using the method of sampling postures, it is necessary to adjust the end of the robotic arm to multiple sets of different postures at the work point, which requires a larger workspace. However, in actual production conditions, the calibration plate needs to be placed in a small, enclosed environment. Multiple sampling postures may collide with the protective wall of the box where the workpiece is stored. At the same time, the calibration method of multiple sampling postures is often time-consuming and complex. At the same time, the composite robot needs one more step before actually performing visual workstation positioning, that is, the mobile chassis needs to be navigated to the work point before the grasping task can be performed. Due to the influence of radar or sensor errors and the uneven ground environment at the work point, the posture of the mobile chassis when it arrives at the work point will be different from the position and posture at the time of calibration. Since the traditional workpiece positioning method is to read the TCP offset before and after teaching, it only considers the translation in three-dimensional space, and does not consider the rotation. If the offset is added according to the posture at the time of calibration, the posture error caused by inaccurate navigation will be ignored. , which will eventually lead to a deviation between the calculated workpiece position and the actual workpiece position, greatly reducing the grasping accuracy of the robot arm; at the same time, considering the interference of environmental factors such as the ground at the actual work point, the state of the chassis is not stable after it stops at the work point. After the robot arm reaches the calculated workpiece positioning point, the chassis will occasionally offset, and its body offset will introduce errors to the robot arm again. Current positioning technologies cannot handle such occasional error problems. At the same time, it is also necessary to evaluate the accuracy of the grasping posture of the workstation reached by the robot arm. Current technologies assume that they have reached the workstation accurately, and most of them do not consider the state detection of the grasping posture of the robot arm when it reaches the workstation.

[0004] In the prior art, Publication (Announcement) No.: CN 110842928 A, Publication (Announcement) Date: 2020.02.28: It relates to a composite robot vision-guided positioning device and method, in which the parameters of the station calibration are the TCP offset before and after teaching, which only considers the position and cannot autonomously deal with the error introduced by the navigation offset. Therefore, it is necessary to add a variety of end-point posture detection sensors, which increases the hardware consumption and algorithm processing complexity; Publication (Announcement) No.: CN 115609591 A, Public (Announcement) Date: 2023.01.17 It involves a visual positioning method and system based on 2DMarker. The calibration method of the workstation adopts a teaching method to reach the workstation grasping posture, and obtains the rigid transformation matrix of the workstation coordinate system relative to the Marker coordinate system by reading the TCP end posture. The matrix is ​​subsequently used for the repositioning and grasping of the workpoint. In theory, this scheme can accurately obtain the repositioned workstation coordinate system posture, but in the actual control of the movement of the manipulator arm of the composite robot, due to the environmental factors of the workpoint in actual production, the body of the robot will occasionally shake slightly. The workstation coordinate system calculated using the pre-calibrated rigid transformation matrix will have an error with the actual one, and this method cannot handle such errors; and this method cannot know whether the manipulator arm after moving has actually reached the workpiece grasping posture during calibration, and does not consider the problem of quantifying the posture accuracy between the workpiece grasping posture actually reached and the workpiece grasping posture calibrated by the workpiece. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a vision-based composite robot hand-eye calibration and workpiece positioning device and method, which completes the hand-eye calibration in a simpler and faster way, and solves the problem of decreased accuracy in vision-guided workpiece grasping caused by chassis movement, while post-processing the subsequent workpiece grasping posture.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A vision-based composite robot hand-eye calibration and workpiece positioning device, comprising: a composite robot and a visual gripper;

[0008] The composite robot includes a mobile chassis and a mechanical arm, wherein the base of the mechanical arm is fixed on the mobile chassis and moves with the mobile chassis;

[0009] The visual gripper includes a visual sensor, a TCP tool end and an end gripper, an Aruco code, an end connector, and an end camera. The end camera, the TCP tool end, and the end connector are all bolted together, and the Aruco code is fixed to the back plate of the end connector by pasting. The end gripper and the end connector are both bolted to the end flange of the robot arm.

[0010] It also includes: a protective box, a camera inside the box, a workpiece receiving box, and a chessboard calibration plate;

[0011] The protective box is the outer shell of the workbench at the working point and is fixed to the workbench; the camera inside the box is located at the top of the protective box and is fixed by bolts; the chessboard calibration plate is fixed at the edge of the workbench, and the workpiece receiving boxes are arranged in sequence at the center of the workbench.

[0012] Preferably, the visual gripper is connected to the end flange of the robot arm with an "L"-shaped mounting piece; the end camera is installed on one end of the "L"-shaped block with bolts, and the other end is installed on the end flange of the robot arm with bolts, and the end clamp is installed along the vertical line of the flange surface, and the end of the TCP tool is set between the clamp tool and the camera.

[0013] The present invention also provides a vision-based composite robot hand-eye calibration and workpiece positioning method, which is implemented by applying the vision-based composite robot hand-eye calibration and workpiece positioning device, and includes:

[0014] Step S1: calibrate the two cameras and the TCP of the end tool according to the preset working height;

[0015] Step S2: Control the robot arm to reach the calibration position and collect image data for hand-eye calibration; control the end of the robot arm tool to reach the feature point position of the checkerboard calibration plate and collect feature point data for hand-eye calibration; calculate the hand-eye calibration parameters based on the image data and the feature point data;

[0016] Step S3: Teach the robot arm to the workpiece grasping posture, and the camera in the box collects the chessboard and Aruco QR code images before and after teaching, and calculates the workstation calibration parameters.

[0017] Preferably, in step S2, the robot arm is controlled to reach the calibration position and the image data of the hand-eye calibration is collected; the tool end of the robot arm is controlled to reach the feature point position of the checkerboard calibration board and the feature point data of the hand-eye calibration is collected; and the method for solving the hand-eye calibration parameters based on the image data and the feature point data includes:

[0018] Step S21: Move the composite robot to the working point, control the end of the robot arm to reach the position of the camera calibration height, and adjust the end posture to vertically shoot the calibration plate, record the current robot arm posture as the hand-eye calibration posture, and collect the image of the checkerboard calibration plate under the hand-eye calibration posture;

[0019] Step S22: Control the end of the robot tool to reach the corner feature point position of the checkerboard calibration plate, and use the end of the needle tip to poke the preset m feature point positions of the checkerboard calibration plate in a clockwise order, and record the m spatial coordinates in the robot base coordinate system in the process of poking points.

[0020] Step S23: Calculate the hand-eye calibration parameters based on the acquired image of the checkerboard calibration plate and the feature point data of the checkerboard calibration plate.

[0021] Preferably, in step S23, the method for calculating the hand-eye calibration parameters according to the acquired image and the feature point data of the checkerboard calibration plate includes:

[0022] Perform corner point detection on the image data of the checkerboard calibration plate to obtain the pixel coordinates of n corner points, namely P i Cam1 (u i v i ),i=1,2,..,n;

[0023] The upper left corner of the calibration plate is preset as the origin of the calibration plate coordinate system. The world coordinates of all corner points, namely P, are constructed based on the known size and number of chessboard grids. i Cal (X i ,Y i ,Z i ),i=1,2,..,n;

[0024] The pixel coordinates and world coordinates of all corner points are matched one by one and the intrinsic parameter matrix K obtained by parallel camera calibration is calculated by PnP algorithm to obtain the calibration plate coordinate system O under the hand-eye calibration posture. Cal Relative to the camera coordinate system O Cam1 The rigid transformation matrix under The expression is:

[0025]

[0026] Preset m corner points on the chessboard calibration plate as feature points, and preset the coordinates of m corner points in the coordinate system of the chessboard calibration plate

[0027] According to the rigid transformation matrix The coordinates of m corner points in the coordinate system of the chessboard calibration plate Switch to the camera coordinate system and get

[0028] According to the obtained TCP end poking the checkerboard feature points in sequence, the tool end posture data is obtained, that is, the point set

[0029] based on Hedianji The constructed matrix relationship is:

[0030]

[0031] The matrix relationship is solved using the SVD eigenvalue decomposition method to obtain the required hand-eye matrix

[0032] Use the SVD method to obtain the required hand-eye matrix Perform UV eigenvalue decomposition to obtain the rotation matrix

[0033]

[0034] Substitute the center point of the point set into the calculation to get the position The expression is:

[0035] in, is the centroid of the feature point in the calibration plate coordinate system, is the centroid of the feature point in the camera coordinate system.

[0036] Preferably, in step S3, the robot arm is taught to grasp the workpiece position, and the camera in the box respectively collects the chessboard and Aruco two-dimensional code images before and after teaching, and the method for calculating the station calibration parameters includes:

[0037] Step S31: The position of the composite robot chassis remains unchanged, and the end of the robot arm tool is controlled to return to the hand-eye calibration posture, and the camera at the end and the camera inside the box respectively collect images of the chessboard calibration plate;

[0038] Step S32: Teaching and controlling the robotic arm to reach above the object, adjusting the end position according to the reasonable grasping posture, and the camera in the box taking a picture of the Aruco QR code at the end of the robotic arm;

[0039] Step S33: Calculate the workstation calibration parameters based on the checkerboard calibration plate images collected by the two cameras, the TCP end posture data when the robot arm moves to the workstation posture, and the Aruco two-dimensional code image.

[0040] Preferably, in step S33, the method for calculating the station calibration parameters according to the checkerboard calibration plate images respectively collected by the two cameras, the TCP end posture data when the robot arm moves to the station posture, and the Aruco two-dimensional code image includes:

[0041] Perform PnP solution on the checkerboard images collected by the two cameras under the hand-eye posture to obtain the checkerboard calibration plate O Cal The rotation matrix of the two coordinate systems relative to the end camera and the camera inside the box and in is a constant;

[0042] Perform PnP solution on the Aruco QR code image taken by the camera in the box under the workstation calibration posture to obtain the transformation matrix between the Aruco coordinate system and the camera coordinate system in the box The position of the tool tip is synchronously acquired and represented as the workstation coordinate system O Work Relative to the robot base coordinate system O Base Rigid transformation of

[0043] Adjust the robot arm to the hand-eye calibration posture and solve the hand-eye matrix Repeatedly apply and based on the rotation matrix and Transformation Matrix And rigid transformation The coordinate system transformation relationship under the workstation positioning posture is obtained as follows:

[0044] in, is the constant rigid transformation matrix, The Aruco code coordinate system calibrated for the workstation is relative to the standard pose of the camera inside the box.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention significantly reduces the time required for hand-eye calibration of the composite robot, simplifies the process of hand-eye calibration, and minimizes the risk of collision with environmental objects when collecting postures for hand-eye calibration, while the hand-eye calibration also has a high accuracy guarantee; in addition, the workstation calibration method of the present invention fully considers the rigid transformation between coordinate systems, so the workpiece can be fixed to the calibration plate in any posture, and the calibration method is completely implemented by an algorithm, without the need to process a high-precision auxiliary positioning device; and the present invention can calculate the repositioning posture error of the mobile chassis at the working point by detecting the posture of the calibration plate, solves the problem of workpiece position calculation error caused by inaccurate navigation, and greatly improves the accuracy of the composite robot's repositioning and grasping of the workpiece at the working point; at the same time, the present invention performs secondary repositioning calculation on the end of the robot arm moved to the workpiece grasping posture by detecting the Aruco code posture, thereby solving the problem of occasional chassis and body offset, and the error compensation of the end of the robot arm can be performed according to the secondary repositioning result. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 It is a schematic diagram of an exemplary embodiment of a composite robot hand-eye calibration and workstation positioning device according to an embodiment of the present invention;

[0049] Figure 2 It is a schematic diagram of an embodiment of the composite robot hand-eye calibration and workstation positioning actual work application secondary positioning;

[0050] Figure 3 A calibration flow chart of an exemplary embodiment of a composite robot hand-eye calibration and workstation positioning method according to an embodiment of the present invention;

[0051] Figure 4 It is an overall flow chart of the actual working application of an exemplary embodiment of the composite robot hand-eye calibration and workstation positioning method according to an embodiment of the present invention;

[0052] Figure 5 A flowchart of the calibration steps of an exemplary embodiment of a composite robot hand-eye calibration and workstation positioning method according to an embodiment of the present invention;

[0053] Figure 6 This is a flowchart of the actual working application steps of an exemplary embodiment of the vision-based composite robot workstation positioning method of the present invention.

[0054] Figure 7 Schematic diagram of obtaining chessboard feature points during calibration of the composite robot hand-eye calibration and workstation positioning method according to an embodiment of the present invention

[0055] Figure 8 A flowchart of specific calibration steps of an exemplary embodiment of a composite robot hand-eye calibration and workstation positioning method according to an embodiment of the present invention;

[0056] Fig. 9 This is a flowchart of specific steps for the actual working application of an exemplary embodiment of the vision-based composite robot workstation positioning method of an embodiment of the present invention.

[0057] Explanation of the numbers in the figure: 1-protective box, 2-camera inside the box, 3-Aruco code, 4-end connector, 5-end camera, 6-TCP tool end, 7-workpiece receiving box, 8-end clamp, 9-chessboard calibration plate, 10-mobile chassis, 11-robotic arm DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] Embodiment 1

[0061] like Figure 1 , Figure 2 As shown, this embodiment provides a vision-based composite robot hand-eye calibration and workpiece positioning device, including: a composite robot and a visual gripper;

[0062] The composite robot comprises a mobile chassis 10 and a mechanical arm 11. The base of the mechanical arm 11 is fixed on the mobile chassis 10 and moves with the mobile chassis 10.

[0063] The visual gripper includes a visual sensor and a TCP tool end 6 and an end clamp 8, an Aruco code 3, an end connector 4, and an end camera 5; the end camera 5, the TCP tool end 6 and the end connector 4 are all bolted, and the Aruco code 3 is fixed on the back plate of the end connector 4 by pasting; and the end clamp 8 and the end connector 4 are both bolted to the end flange of the robot arm 11;

[0064] It also includes: a protective box 1, a camera in the box 2, a workpiece receiving box 7, and a chessboard calibration plate 9;

[0065] The protective box 1 is the outer shell of the workbench at the working point and is fixed to the workbench; the camera 2 inside the box is located at the top of the protective box 1 and is fixed by bolts; the checkerboard calibration plate 9 is fixedly placed at the edge of the workbench, and the workpiece receiving box 7 is arranged in sequence at the center of the workbench.

[0066] Specifically, the visual gripper is connected to the end flange of the robot arm 11 with an "L"-shaped mounting piece; the end camera is installed on one end of the "L"-shaped block with bolts, and the other end is installed on the end flange of the robot arm 11 with bolts, and the end clamp is installed along the vertical line of the flange surface, and the TCP tool end 6 is set between the clamp tool and the camera.

[0067] Embodiment 2

[0068] like Figure 3 , Figure 4As shown, this embodiment provides a method for hand-eye calibration and workpiece positioning of a composite robot based on vision, comprising the following steps: Figure 5 As shown:

[0069] Step S1: calibrate the two cameras and the TCP of the end tool according to the preset working height;

[0070] Step S2: Control the robot arm 11 to reach the calibration position, collect image data for hand-eye calibration, control the end of the robot arm tool to reach the feature point position of the calibration plate, collect feature point data for hand-eye calibration, and calculate the hand-eye calibration parameters.

[0071] Step S3: Teach the robot arm 11 to the workpiece grasping posture, and the camera 2 in the box collects the chessboard and Aruco two-dimensional code images before and after the teaching, and calculates the station calibration parameters.

[0072] Specifically, Figure 8 As shown, step S1 first presets the working distance of the robot arm 11, calibrates the camera on the end at this distance, obtains the camera intrinsic parameters and distortion parameters, and sets the camera optical center as the end camera 5 coordinate system O Cam1 , the camera in the protective box 1 is also calibrated synchronously, and the calibrated working distance is the height between the checkerboard calibration plate 9 and the camera, and the optical center of the camera in the protective box 1 is set as the coordinate system O of the camera 2 in the box Cam2 The purpose of calibrating the two cameras separately is to ensure that the images taken by the subsequent cameras are clear and visible; continue to calibrate the end tool of the robot arm, and set the calibrated composite robot tool end coordinate system as O Tool ;

[0073] Among them, the working distance of the robot arm 11 is preset first, and the method for calibrating the camera on the end at this distance is: first roughly preset the working distance of the camera, place a calibration plate on the working surface, move the robot arm to lift the camera to the preset working distance, adjust the camera focal length so that the calibration plate is clearly visible in the camera, and then complete the camera calibration operation.

[0074] Intrinsic parameters and distortion parameters refer to the data obtained after camera calibration is completed.

[0075] End tool calibration is also called TCP calibration. The calibration process is as follows: first, place a needle tip on the work surface and fix it in place. Then, use a mobile robotic arm to make the end tool contact the needle tip in different postures, and synchronously record the posture information at the time of contact. Then, perform TCP calibration calculations, and finally output the rigid transformation matrix data of the end tool relative to the robotic arm flange. What is calibrated here is the robotic arm end tool, which can also be understood as the end tool of a composite robot, because the components of a composite robot are the mobile chassis and the robotic arm. For the sake of clarity, it can also be called the robotic arm end tool.

[0076] Step S21: Build a map and set the work point position for workpiece grabbing. After moving the composite robot to the work point, set the base coordinate system of the composite robot arm to O Base , control the end of the robot arm 11 to reach the approximate camera calibration height position, adjust the end posture so that it can vertically shoot the checkerboard calibration plate 9, fine-tune so that the calibration plate is located in the middle of the image and ensure that the size of the calibration plate is 2 / 3 of the image field of view, set the upper left feature point of the checkerboard calibration plate 9 as the calibration plate coordinate system O Cal , synchronously record the current robot arm position And recorded as the robot arm hand-eye calibration posture;

[0077] Step S22: Collect the calibration plate image, control the end tool of the robot arm to gradually reach the preset calibration plate feature points, and synchronously record the position and posture of the TCP end tool when it reaches the feature points;

[0078] Specifically, it is necessary to perform image acquisition of the checkerboard calibration plate 9 while maintaining the robot arm hand-eye calibration posture in step S21. After taking the calibration plate data, refer to Figure 7 The 9 feature points of the chessboard calibration plate are collected. The tip of the teaching needle pokes the preset m chessboard feature points in clockwise order. During the poking process, the m spatial coordinates in the robot arm base coordinate system are recorded in sequence. The collected image data and the tool end data at the feature points are used to calculate the hand-eye calibration parameters in step S23;

[0079] Step S23: Calculate the hand-eye calibration parameters according to the collected image and the feature point data of the TCP end tool reaching the calibration plate;

[0080] Specifically, the hand-eye calibration parameter solution process can be divided into two parts: The first part of the algorithm is: firstly, the image data of the chessboard calibration plate 9 is corner detected to obtain the pixel coordinates of n corner points, that is, P i Cam1 (u i v i),i=1,2,..,n, the upper left corner of the calibration plate is preset as the origin of the calibration plate coordinate system, and the world coordinates of all corner points are constructed according to the known size and number of chessboard grids, that is, P i Cal (X i ,Y i ,Z i ), i = 1, 2, .., n, the two types of coordinate points are matched one by one and the intrinsic parameter matrix K obtained by parallel camera calibration is calculated by PnP algorithm to obtain the calibration plate coordinate system O under the hand-eye calibration posture Cal Relative to the camera coordinate system O Cam1 The rigid transformation matrix under Its matrix relationship is:

[0081]

[0082] The second part of the algorithm is as follows: m corner points on the preset chessboard calibration plate 9 are used as feature points, and the coordinates of these points in the calibration plate coordinate system are known. According to the calculated external parameter matrix Convert this group to the camera coordinate system to get According to the tool end posture data obtained in step S23 when the TCP end sequentially pokes the checkerboard feature points, the point set The matrix relationship constructed is:

[0083]

[0084] Then use the SVD eigenvalue decomposition method to obtain the linear solution of these two sets of points, which is the required hand-eye matrix The calculation process is:

[0085] set up Where j = m, represents the m feature points of the checkerboard calibration plate in the checkerboard coordinate system O Cal The three-dimensional coordinates on Where j = m, which represents the m feature points of the checkerboard calibration plate in the coordinate system O of the end camera 5 Cam1 The three-dimensional coordinates on the coordinate system are first obtained by finding the centroid of the feature point sets of these two sets of coordinate systems:

[0086]

[0087] Decentralize the two sets of points, that is, subtract the center point from each set of points. The decentralized point sets still satisfy the above matrix relationship, namely:

[0088]

[0089] Compute the covariance matrix between two sets of points:

[0090]

[0091] The SVD method is used to perform UV eigenvalue decomposition on the covariance matrix and calculate the rotation matrix of the two sets of point sets.

[0092]

[0093] Finally, the center point of the point set and the rotation matrix solved above are Substitute the following formula to calculate and get the position

[0094]

[0095] Step S23 The calculation process of the hand-eye calibration parameters requires two sets of data: the image data of the checkerboard calibration board under the hand-eye calibration posture and the tool end posture data when the TCP end sequentially pokes the checkerboard feature points; the output result of the hand-eye calibration is the hand-eye calibration matrix The hand-eye calibration matrix here is not the traditional eye-on-hand camera coordinate system O Cam1 Relative to the robot end coordinate system O End The Hand-Eye Matrix matrix Represents the composite robot camera coordinate system is O Cam1 Relative to the composite robot base coordinate system O Base The rigid transformation includes the coordinate system O of the end of the robot End indirect conversion.

[0096] Step S31 keeps the composite robot at the working point unchanged. Considering that the hand-eye calibration matrix in this method is represented by the composite robot camera coordinate system O Cam1 Relative to the composite robot base coordinate system O Base The rigid transformation of the hand-eye matrix is ​​used to control the end of the robot tool to return to the hand-eye calibration posture. The purpose is to ensure that the relationship between the camera and the robot base remains unchanged. It can be applied repeatedly, and the camera at the end and the camera 2 in the box both collect images of the checkerboard calibration plate;

[0097] Specifically, the purpose of reshooting the checkerboard image is to solve the new camera external parameters, that is, to calculate the calibration plate coordinate system O under the hand-eye calibration posture when the working point is reached again. Cal Relative to the end camera 5 coordinate system O Cam1 The rigid transformation matrix under Before moving the robot arm 11, the camera in the box is turned on to take a picture of the checkerboard calibration plate 9, the purpose of which is to calculate the coordinate system O of the checkerboard calibration plate. Cal Relative to the coordinate system O of camera 2 in the boxCam2 The rigid transformation matrix under Since the checkerboard calibration plate 9 is fixedly installed in the protective box 1, and the camera 2 in the box is also fixedly installed above the protective box 1, They are constant and known in the subsequent station calibration and actual positioning and grasping.

[0098] Step S32: An Aruco QR code is fixed on the end connector 4 of the robot arm. The posture of the subsequent Aruco code 3 will change with the posture of the end of the robot arm. The center of the plane QR code is set as the Aruco coordinate system O. Arc , teach and control the robot arm 11 to reach above the object, and adjust the end position to a reasonable grasping position, and set the tip of the tool end as the workstation coordinate system O Work , turn on the camera 2 in the box and take a picture of the Aruco QR code at the end of the robotic arm;

[0099] Step S33 calculates the station calibration parameters according to the checkerboard images collected by the two cameras and the Aruco two-dimensional code image when the robot arm 11 moves to the station posture;

[0100] Specifically, the station calibration parameter solution process of step S33 is to obtain the station calibration parameters: The process includes: firstly, performing PnP solution on the chessboard images collected by the two cameras under the hand-eye posture to obtain the chessboard calibration plate O Cal The rotation matrix of the two coordinate systems relative to the end camera 5 and the camera 2 in the box and in is a constant; continue to perform PnP solution on the Aruco QR code image taken by the camera 2 in the box under the station calibration posture, and obtain the transformation matrix between the Aruco coordinate system and the coordinate system of the camera 2 in the box The position of the tool tip is obtained synchronously, which is represented by the workstation coordinate system O Work Relative to the robot base coordinate system O Base Rigid transformation of Since the robot arm 11 has been adjusted to the hand-eye calibration posture in step S31, the hand-eye matrix solved in step S23 It can be applied repeatedly, and the coordinate system transformation relationship under the workstation calibration posture is derived as follows:

[0101]

[0102] Combined with the solved hand-eye calibration parameters, we can use the formula Solution therefore is a known quantity, is a known quantity, constant, is a quantity that can be calculated; the rotation matrix can be derived It represents the workstation coordinate system O Work Relative to the Aruco coordinate system O Arc The rigid transformation of , thus the station calibration parameters have been obtained, which include: The Aruco code 3 coordinate system calibrated for the workstation is relative to the standard position of the camera 2 in the box, through the matrix conversion relationship The workstation coordinate system O can be obtained indirectly Work Relative to the camera O in the box Cam2 The pose data of the camera 2 in the box and the workpiece placement pose relationship remain unchanged, so the subsequent operation only needs to ensure Return to the state of the workstation calibration, and accurate grasping can be completed. are all constants, so we only need to ensure Just return to the workstation calibration state, which is also the key to the error when reaching the workpiece grasping position after subsequent quantitative repositioning.

[0103] Furthermore, the actual workpiece grasping process includes: Figure 6 As shown:

[0104] Step S4: Navigate the chassis to the working point and control the robot arm 11 to the position during hand-eye calibration;

[0105] Step S5: the terminal camera 5 takes a picture of the checkerboard calibration plate, calculates the position and posture of the calibration plate, combines the hand-eye matrix and the workstation calibration posture parameters, and performs the workstation coordinate system posture calculation;

[0106] Step S6 controls the end of the robot tool to reach the calculated workstation posture, and the camera 2 in the box takes a picture of the Aruco code 3 at the end of the robot again and calculates the posture, and calculates the error value of the Aruco code 3 posture during the workstation calibration;

[0107] Step S7 is to process the difference of the actual Aruco code 3 posture. If the error exceeds the offset error threshold, it is considered that the vehicle body has been offset and needs to be recalculated.

[0108] Step S8 continues to determine whether the difference in the Aruco code 3 poses meets the accurate grasping error threshold. If it meets the threshold, it is considered that the workpiece can be accurately grasped. Otherwise, the end pose is adjusted within a small range to compensate for the error.

[0109] Specifically, Fig. 9 As shown, the actual workstation grasping of the vision-based composite robot includes the following steps:

[0110] Step S41: The composite robot navigates to the work point set in the map, adjusts the robot arm posture to the hand-eye calibration posture, and ensures that the relationship between the end camera 5 and the robot arm base remains unchanged;

[0111] Step S51: Open the terminal camera 5 to collect the chessboard calibration plate image, pre-process the image according to grayscale, threshold segmentation, corner point detection and other algorithms, obtain a new set of chessboard corner point pixel coordinates, and use the PnP algorithm to recalculate the chessboard calibration plate 9 relative to the camera.

[0112] Step S52 combines the workpiece calibration to obtain the workpiece relative to the calibration plate station calibration parameters, namely: Calculate the position and posture of the workstation coordinate system relative to the robot arm base;

[0113] Specifically, the calculated pose matrix is: From the station calibration in step 32, it can be seen that are all constants, is the standard quantity for station calibration, in which the only variable is the relocation error of the chassis working point caused by the navigation error. This type of error can be calculated based on the error in step S51. to quantify, so according to the coordinate system transfer relationship:

[0114]

[0115] The position and posture of the workstation coordinate system in the robot base coordinate system after repositioning are obtained, thus realizing the positioning of the workpiece;

[0116] Step S61 controls the end of the robot tool to reach the calculated workstation grasping posture, turns on the camera 2 in the box to take another photo of the Aruco code 3 on the end of the robot in the workstation grasping posture state, and calculates the posture of the Aruco code 3 coordinate system in the camera coordinate system when it reaches the workstation marking posture this time, namely: Synchronously calculate the standard posture relationship between it and the workstation calibration The difference between

[0117] Step S71 first presets the offset threshold. With station calibration The rotation component is converted into the Euler angle form, and it is determined whether the deviation values ​​of the two data, the Euler angle and the position quantity, are within the preset offset threshold range. If it exceeds, it is considered that: after recalculating the workstation coordinate system, in the process of controlling the robot arm 11 to reach the workstation to grasp the posture, the body of the car is unstable and deviates. This accidental deviation causes the posture of the chessboard calibration plate 9 relative to the camera If the robot arm 11 is adjusted back to the hand-eye calibration posture, the end camera 5 is turned on again to shoot the checkerboard calibration plate 9, and the actual offset is calculated. After updating the rotation relationship, continue to combine the station calibration parameters to calculate the new station grasping posture;

[0118] In step S81, when the difference meets the offset threshold, it is considered that the vehicle body has not moved. In order to consider accurate grasping, a precise grasping threshold needs to be set, mainly for the error problems caused by the repetitive error of the robot arm 11 and interference from other factors. Continue to judge whether the deviation value is within the preset precise grasping threshold range. If it exceeds: adjust the posture of the end of the robot arm according to the Euler difference and the position difference feedback, perform error compensation on the actual workstation posture, and after the adjustment, repeat the detection process of the camera 2 in the box for secondary positioning. When the threshold requirement is met, the precise grasping of the workpiece is completed.

[0119] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A vision-based composite robot hand-eye calibration and workpiece positioning device, characterized in that: include: Composite robots and vision grippers; The composite robot includes a mobile chassis and a mechanical arm, wherein the base of the mechanical arm is fixed on the mobile chassis and moves with the mobile chassis; The visual gripper includes a visual sensor, a TCP tool end and an end gripper, an Aruco code, an end connector, and an end camera. The end camera, the TCP tool end, and the end connector are all bolted together, and the Aruco code is fixed to the back plate of the end connector by pasting. The end gripper and the end connector are both bolted to the end flange of the robot arm. It also includes: a protective box, a camera inside the box, a workpiece receiving box, and a chessboard calibration plate; The protective box is the outer shell of the workbench at the working point and is fixed to the workbench; the camera inside the box is located at the top of the protective box and is fixed by bolts; the chessboard calibration plate is fixed at the edge of the workbench, and the workpiece receiving boxes are arranged in sequence at the center of the workbench.

2. The vision-based composite robot hand-eye calibration and workpiece positioning device according to claim 1, characterized in that: The visual gripper is connected to the end flange of the robot arm with an "L"-shaped mounting piece; the end camera is installed on one end of the "L"-shaped block with bolts, and the other end is installed on the end flange of the robot arm with bolts, and the end clamp is installed along the vertical line of the flange surface, and the end of the TCP tool is set between the clamp tool and the camera.

3. A vision-based composite robot hand-eye calibration and workpiece positioning method, characterized in that: The vision-based composite robot hand-eye calibration and workpiece positioning device according to any one of claims 1 to 2 is implemented, comprising: Step S1: calibrate the two cameras and the TCP of the end tool according to the preset working height; Step S2: Control the robot arm to reach the calibration position and collect image data for hand-eye calibration; control the end of the robot arm tool to reach the feature point position of the checkerboard calibration plate and collect feature point data for hand-eye calibration; calculate the hand-eye calibration parameters based on the image data and the feature point data; Step S3: Teach the robot arm to the workpiece grasping posture, and the camera in the box collects the chessboard and Aruco QR code images before and after teaching, and calculates the workstation calibration parameters.

4. The method for hand-eye calibration and workpiece positioning of a composite robot based on vision according to claim 3, characterized in that: In the step S2, the robot arm is controlled to reach the calibration position and the image data of the hand-eye calibration is collected; the tool end of the robot arm is controlled to reach the feature point position of the checkerboard calibration plate and the feature point data of the hand-eye calibration is collected; the method for solving the hand-eye calibration parameters based on the image data and the feature point data includes: Step S21: Move the composite robot to the working point, control the end of the robot arm to reach the position of the camera calibration height, and adjust the end posture to vertically shoot the calibration plate, record the current robot arm posture as the hand-eye calibration posture, and collect the image of the checkerboard calibration plate under the hand-eye calibration posture; Step S22: Control the end of the robot tool to reach the corner feature point position of the checkerboard calibration plate, and use the end of the needle tip to poke the preset m feature point positions of the checkerboard calibration plate in a clockwise order, and record the m spatial coordinates in the robot base coordinate system in the process of poking points. Step S23: Calculate the hand-eye calibration parameters based on the acquired image of the checkerboard calibration plate and the feature point data of the checkerboard calibration plate.

5. The method for hand-eye calibration and workpiece positioning of a composite robot based on vision according to claim 4, characterized in that: In step S23, the method for calculating the hand-eye calibration parameters according to the acquired image and the feature point data of the checkerboard calibration plate includes: Perform corner point detection on the image data of the chessboard calibration plate to obtain the pixel coordinates of n corner points, namely The upper left corner of the calibration plate is preset as the origin of the calibration plate coordinate system. The world coordinates of all corner points, namely P, are constructed based on the known size and number of chessboard grids. i Cal (X i ,Y i ,Z i ),i=1,2,..,n; The pixel coordinates and world coordinates of all corner points are matched one by one and the intrinsic parameter matrix K obtained by parallel camera calibration is calculated by PnP algorithm to obtain the calibration plate coordinate system O under the hand-eye calibration posture. Cal Relative to the camera coordinate system O Cam1 The rigid transformation matrix under The expression is: Preset m corner points on the chessboard calibration plate as feature points, and preset the coordinates of m corner points in the coordinate system of the chessboard calibration plate According to the rigid transformation matrix The coordinates of m corner points in the coordinate system of the chessboard calibration plate Switch to the camera coordinate system and get According to the obtained TCP end poking the checkerboard feature points in sequence, the tool end posture data is obtained, that is, the point set based on Hedianji The constructed matrix relationship is: The matrix relationship is solved using the SVD eigenvalue decomposition method to obtain the required hand-eye matrix Use the SVD method to obtain the required hand-eye matrix Perform UV eigenvalue decomposition to obtain the rotation matrix Substitute the center point of the point set into the calculation to get the position The expression is: in, is the centroid of the feature point in the calibration plate coordinate system, is the centroid of the feature point in the camera coordinate system.

6. The method for hand-eye calibration and workpiece positioning of a composite robot based on vision according to claim 5, characterized in that: In step S3, the robot arm is taught to grasp the workpiece position, and the camera in the box collects the chessboard and Aruco two-dimensional code images before and after the teaching, respectively. The method for calculating the station calibration parameters includes: Step S31: The position of the composite robot chassis remains unchanged, and the end of the robot arm tool is controlled to return to the hand-eye calibration posture, and the camera at the end and the camera inside the box respectively collect images of the chessboard calibration plate; Step S32: Teaching and controlling the robotic arm to reach above the object, adjusting the end position according to the reasonable grasping posture, and the camera in the box taking a picture of the Aruco QR code at the end of the robotic arm; Step S33: Calculate the workstation calibration parameters based on the checkerboard calibration plate images collected by the two cameras, the TCP end posture data when the robot arm moves to the workstation posture, and the Aruco two-dimensional code image.

7. The method for hand-eye calibration and workpiece positioning of a composite robot based on vision according to claim 6, characterized in that: In step S33, the method for calculating the station calibration parameters according to the checkerboard calibration plate images respectively collected by the two cameras, the TCP end posture data when the robot arm moves to the station posture, and the Aruco two-dimensional code image includes: Perform PnP solution on the checkerboard images collected by the two cameras under the hand-eye posture to obtain the checkerboard calibration plate O Cal The rotation matrix of the two coordinate systems relative to the end camera and the camera inside the box and in is a constant; Perform PnP solution on the Aruco QR code image taken by the camera in the box under the workstation calibration posture to obtain the transformation matrix between the Aruco coordinate system and the camera coordinate system in the box The position of the tool tip is synchronously acquired and represented as the workstation coordinate system O Work Relative to the robot base coordinate system O Base Rigid transformation of Adjust the robot arm to the hand-eye calibration posture and solve the hand-eye matrix Repeatedly apply and based on the rotation matrix and Transformation Matrix And rigid transformation The coordinate system transformation relationship under the workstation positioning posture is obtained as follows: in, is the constant rigid transformation matrix, The Aruco code coordinate system calibrated for the workstation is relative to the standard pose of the camera inside the box.

Images