Robot hand-eye calibration method, control device and robot system

Abstract

The application discloses a robot hand-eye calibration method, a control device and a robot system, and relates to the technical field of robots. The robot hand-eye calibration method unifies multiple pixel coordinates obtained through dynamic shooting to a reference pixel coordinate system consistent with the direction of the camera coordinate system at the reference position, simultaneously performs localization processing on the world coordinates of the end effector corresponding to each sampling position relative to the world coordinates of the end effector at the reference position, eliminates the interference caused by global coordinate offset and camera orientation difference, and expands the application range of the hand-eye calibration method. In addition, the singular value decomposition algorithm is used to solve the mapping relationship between the pixel coordinates in the reference pixel coordinate system and the local world coordinates, so that the rotation matrix obtained can satisfy the orthogonality constraint, the hand-eye transformation process strictly conforms to the robot motion model with translation and rotation motion, and the calibration accuracy is improved. The calibration process of the application can be automatically calculated, and the calibration efficiency is relatively high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a robot hand-eye calibration method, a control device and a robot system. BACKGROUND

[0002] In automated production, robots are often combined with vision systems to guide precise grasping, positioning and assembly tasks through cameras. To achieve this function, hand-eye calibration must be performed to determine the transformation relationship between the camera coordinate system and the robot coordinate system. For example, when a camera is fixedly installed on the small arm of a SCARA (Selective Compliance Assembly Robot Arm) robot, since the relative position relationship between the camera and the robot small arm is fixed, the core of hand-eye calibration is to solve the transformation matrix between the camera coordinate system and the SCARA robot end.

[0003] At present, the calibration method of eye on hand usually adopts the following steps: a calibration board (such as a chessboard) is fixed in the workspace, and the camera is controlled to move above the calibration board to take a picture to obtain the pixel coordinates of the feature points. Then the robot end (or a feature point) is controlled to move to multiple feature points (such as corner points) on the calibration board, and the robot world coordinates corresponding to each feature point are recorded. However, this method requires manual intervention to accurately guide the robot end to align with the feature points on the calibration board point by point, and based on the corresponding relationship between the collected multiple sets of pixel coordinates and world coordinates, the hand-eye transformation matrix is solved by means of least squares method, which has the problems of low automation degree, poor calibration efficiency and difficult to guarantee the accuracy.

[0004] The above content is only used to assist in understanding the technical solutions of the present application, and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a robot hand-eye calibration method, which aims to solve the problems of low automation degree, poor calibration efficiency and difficult to guarantee the accuracy existing in the existing calibration method.

[0006] To achieve the above purpose, the present application provides a robot hand-eye calibration method, the robot comprising a first rotating arm, a second rotating arm and an end effector, a camera parallel to the end effector being installed on the second rotating arm, the steps of the method comprising:

[0007] S1, control the robot to move, so that the camera moves to an initial position above a calibration object, record the corresponding position of the robot as a reference position, take a picture of the calibration object at the reference position to obtain pixel coordinates, and record the world coordinates of the end effector in the base coordinate system and the joint angles of the first rotating arm and the second rotating arm;

[0008] S2, planning a regular grid motion path containing a plurality of sampling points with the reference position as the center, controlling the robot to move to the rest of the plurality of sampling positions according to the regular grid motion path except the reference position, respectively taking pictures of the calibration object to obtain corresponding pixel coordinates, and recording the world coordinates of the end effector and the joint angles of the first and second rotary arms at each position;

[0009] S3, based on the pixel coordinates obtained at the rest of the plurality of sampling positions, the joint angles of the first and second rotary arms, and the pixel coordinates obtained at the reference position, the joint angles of the first and second rotary arms, performing inverse rotation transformation on the rest of the plurality of pixel coordinates to correct them to the reference pixel coordinate system consistent with the direction of the camera coordinate system at the reference position;

[0010] S4, based on the world coordinates of the end effector at each position and the world coordinates of the end effector at the reference position, calculating the local world coordinates of each position with the reference position as the origin;

[0011] S5, based on the pixel coordinates of each position in the reference pixel coordinate system and the corresponding local world coordinates, calculating the rotation matrix and translation vector of the hand-eye transformation matrix using the singular value decomposition algorithm.

[0012] In an embodiment, step S5 comprises:

[0013] centering the pixel coordinates of each position in the reference pixel coordinate system and the corresponding local world coordinates to obtain a set of center pixel coordinates and a set of center world coordinates; wherein

[0014]

[0015]

[0016]

[0017] the set of center pixel coordinates, the pixel coordinates of each position in the reference pixel coordinate system, the center pixel coordinates, the number of the rest of the plurality of sampling positions, the set of center world coordinates, the local world coordinates of each position, the center world coordinates;

[0018] calculating the covariance matrix between the set of center pixel coordinates and the set of center world coordinates ;

[0019] wherein

[0020]

[0021] performing singular value decomposition on the covariance matrix to obtain a rotation matrix ;

[0022] wherein

[0023] ,

[0024] , , is a left singular vector matrix, a diagonal singular value matrix and a right singular vector matrix, respectively;

[0025] based on the rotation matrix, the center pixel coordinate and the center world coordinate, calculating the translation vector;

[0026] wherein

[0027] ,

[0028] is the translation vector, is the hand-eye transformation matrix.

[0029] In an embodiment, before the step of calculating the translation vector based on the rotation matrix, the center pixel coordinate and the center world coordinate, the method further comprises:

[0030] if the determinant of the rotation matrix is less than 0, multiplying the last column of the right singular vector matrix by -1, and recalculating the rotation matrix according to .

[0031] In an embodiment, step S2 comprises:

[0032] based on a preset horizontal translation distance and a vertical translation distance, planning a nine-square motion path centered on the reference position;

[0033] controlling the robot to sequentially translate to the remaining 8 sampling positions according to the planned path, respectively taking pictures to obtain corresponding pixel coordinates, and recording the world coordinates of the end effector and the joint angles of the first and second rotary arms at each position.

[0034] In one embodiment, step S3 includes:

[0035] Calculate the joint angle changes of the remaining multiple sampling positions relative to the reference position;

[0036] in

[0037]

[0038] This represents the change in joint angle relative to the reference position at the corresponding sampling position. This represents the joint angle of the first rotating arm at the corresponding sampling position. This represents the joint angle of the second rotating arm at the corresponding sampling position. The joint angle of the first rotating arm at the reference position. The joint angle of the second rotating arm at the reference position;

[0039] The pixel coordinates collected at each sampling position are rotated in the opposite direction around the reference position to obtain the pixel coordinates in the reference pixel coordinate system;

[0040] in

[0041]

[0042] This represents the change in joint angle relative to the reference position at the corresponding sampling position. These are the pixel coordinates of the corresponding sampling position. The corresponding sampling position is the pixel coordinate in the reference pixel coordinate system. These are the pixel coordinates corresponding to the reference position.

[0043] In one embodiment, step S4 includes:

[0044] Set the world coordinates P of each location wi (x i ,y i Subtract the world coordinates P of the reference position w0 (x0, y0) is used to obtain the local world coordinates with the reference position as the origin;

[0045] in

[0046] .

[0047] In addition, to achieve the above objectives, this application also proposes a control device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot hand-eye calibration method as described above.

[0048] Furthermore, to achieve the above object, the application further provides a robot system, comprising a robot and the control device as described above, wherein the robot has a first rotating arm, a second rotating arm and an end effector, and a camera is installed on the second rotating arm and is parallel to the end effector.

[0049] In an embodiment, the control device is further configured to:

[0050] control the robot to move the camera to any one position, denoted as a target position, take a picture of a target object at the target position to obtain target pixel coordinates, and read the world coordinates of the end effector in the base coordinate system; and

[0051] based on the target pixel coordinates, the hand-eye transformation matrix and the world coordinates of the end effector in the base coordinate system, calculate the global world coordinates of the target object in the base coordinate system obtained by the camera when the camera moves to the target position.

[0052] In an embodiment, the control device is particularly configured to:

[0053] correct the target pixel coordinates to the reference pixel coordinate system to obtain pixel coordinates of the target pixel coordinates in the reference pixel coordinate system;

[0054] wherein

[0055]

[0056]

[0057] is a joint angle change amount of the target position relative to the reference position, is the target pixel coordinates, is pixel coordinates of the target pixel coordinates in the reference pixel coordinate system, is pixel coordinates corresponding to the reference position, is a joint angle of the first rotating arm at the target position, is a joint angle of the second rotating arm at the target position, is a joint angle of the first rotating arm at the reference position, is a joint angle of the second rotating arm at the reference position;

[0058] based on the pixel coordinate of the target pixel coordinate in the reference pixel coordinate system, the hand-eye transformation matrix and the world coordinate of the end effector in the base coordinate system, the global world coordinate of the target object in the base coordinate system obtained by the camera when the camera moves to the target position and takes a picture is calculated;

[0059] wherein

[0060]

[0061] (x, y) is the world coordinate of the end effector in the base coordinate system at the corresponding position, , respectively the rotation matrix and the translation vector, is the global world coordinate of the target object in the base coordinate system obtained by the camera when the camera moves to the target position and takes a picture.

[0062] The robot hand-eye calibration method provided in the application comprises the following steps: S1, controlling the robot to move so that the camera moves to an initial position above the calibration object, recording the position of the robot as a reference position, taking a photo of the calibration object at the reference position to obtain pixel coordinates, and recording the world coordinates of the end effector in the base coordinate system and the joint angles of the first rotating arm and the second rotating arm; S2, planning a regular grid motion path containing a plurality of sampling points with the reference position as the center, controlling the robot to move to the plurality of sampling positions except the reference position in turn according to the regular grid motion path, taking a photo of the calibration object at each position to obtain corresponding pixel coordinates, and recording the world coordinates of the end effector at each position and the joint angles of the first rotating arm and the second rotating arm; S3, based on the pixel coordinates obtained at the plurality of sampling positions, the joint angles of the first rotating arm and the second rotating arm, and the pixel coordinates obtained at the reference position, the joint angles of the first rotating arm and the second rotating arm, performing inverse rotation transformation on the plurality of pixel coordinates to correct them to the reference pixel coordinate system which is consistent with the direction of the camera coordinate system at the reference position; S4, based on the world coordinates of the end effector at each position and the world coordinates of the end effector at the reference position, calculating the local world coordinates of each position with the reference position as the origin; and S5, based on the pixel coordinates of each position in the reference pixel coordinate system and the corresponding local world coordinates, calculating the rotation matrix and the translation vector of the hand-eye transformation matrix by using the singular value decomposition algorithm. In the application, the plurality of pixel coordinates obtained by dynamic shooting are corrected to the reference pixel coordinate system which is consistent with the direction of the camera coordinate system at the reference position, and the world coordinates of the end effector at each sampling position are localized relative to the world coordinates of the end effector at the reference position, so as to focus on the mapping relationship constructed by the relative displacement and the attitude change. This strategy not only eliminates the interference of global coordinate offset and camera orientation difference, but also makes the calibration model more suitable for different working areas and motion trajectories, effectively expanding the application range of the hand-eye calibration method. On this basis, the singular value decomposition algorithm is used to solve the mapping relationship between the pixel coordinates in the reference pixel coordinate system and the local world coordinates, so that the rotation matrix satisfies the orthogonality constraint, and the hand-eye transformation strictly conforms to the robot motion model with translation and rotation relationship, thereby improving the calibration accuracy of the hand-eye calibration. Moreover, the calibration process of the application can be automatically calculated, which can improve the automation degree and the calibration efficiency of the hand-eye calibration. BRIEF DESCRIPTION OF DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and serve to explain the principles of the application.

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings are only for the purpose of illustrating the embodiments of the present application, and for the person of ordinary skill in the art, other drawings can be obtained without creative labor.

[0065] Figure 1 The flowchart provided for an embodiment of the robot hand-eye calibration method of the present application;

[0066] Figure 2 The flowchart provided for an embodiment of the robot system of the present application;

[0067] Figure 3 The SCARA robot structure diagram provided for an embodiment of the robot hand-eye calibration method of the present application;

[0068] Figure 4 The nine-square path planning diagram provided for an embodiment of the robot hand-eye calibration method of the present application;

[0069] Figure 5 The ab diagram of the correction process of correcting the pixel coordinates to the reference pixel coordinate system provided for an embodiment of the robot hand-eye calibration method of the present application;

[0070] Figure 6 The conversion process diagram of converting the world coordinates to the local world coordinates provided for an embodiment of the robot hand-eye calibration method of the present application.

[0071] The purpose of the present application, the functional characteristics and the advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0072] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0073] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail with reference to the accompanying drawings and the specific embodiments.

[0074] The main solution of the embodiment of the present application is to provide a robot hand-eye calibration method, the robot comprising a first rotating arm, a second rotating arm and an end effector, a camera parallel to the end effector being installed on the second rotating arm, the steps of the method comprising: step S1, controlling the robot to move so that the camera moves to an initial position above a calibration object, recording the position of the robot as a reference position, taking a picture of the calibration object at the reference position to obtain pixel coordinates, and recording the world coordinates of the end effector in the base coordinate system and the joint angles of the first rotating arm and the second rotating arm. Step S2, centering on the reference position, a regular grid motion path containing a plurality of sampling points is planned, the robot is controlled to move to the plurality of sampling positions except the reference position in turn according to the regular grid motion path, the calibration object is photographed at each position to obtain corresponding pixel coordinates, and the world coordinates of the end effector at each position and the joint angles of the first rotating arm and the second rotating arm are recorded. Step S3, based on the pixel coordinates obtained at the plurality of sampling positions, the joint angles of the first rotating arm and the second rotating arm, and the pixel coordinates obtained at the reference position, the joint angles of the first rotating arm and the second rotating arm, the pixel coordinates at the plurality of positions are inversely rotated and transformed to be corrected to the reference pixel coordinate system consistent with the direction of the camera coordinate system at the reference position. Step S4, based on the world coordinates of the end effector at each position and the world coordinates of the end effector at the reference position, the local world coordinates of each position with the reference position as the origin are calculated. Step S5, based on the pixel coordinates of each position in the reference pixel coordinate system and the corresponding local world coordinates, the rotation matrix and the translation vector of the hand-eye transformation matrix are calculated by using the singular value decomposition algorithm.

[0075] The technical terms involved in the embodiment of the present application:

[0076] Hand-eye calibration: refers to determining the spatial transformation relationship (i.e. hand-eye transformation matrix) between the camera (i.e. corresponding to the eye) installed on the end of the robot (i.e. corresponding to the hand) and the robot base coordinate system. The transformation usually includes a rotation matrix R and a translation vector T, which is used to convert the points in the camera coordinate system into the robot base coordinate system, or vice versa.

[0077] Calibration object: an object with a known geometric structure (such as a checkerboard, etc.), the positions of the feature points in its own coordinate system being accurately known, which is used to provide the corresponding relationship between the pixel coordinates and the real world coordinates.

[0078] Base coordinate system: the fixed reference coordinate system of the robot body, usually located at the robot base, all joint movements and end poses are described based on this.

[0079] Pixel coordinates: the positions of feature points in the camera image, with the upper left corner of the image as the origin, and the unit being pixels.

[0080] Local world coordinate: A world coordinate system established with a certain position (such as a reference position) as the origin, used to simplify calculations and eliminate the effects of global offset.

[0081] Singular value decomposition: Singular value decomposition (SVD) is a mathematical method of decomposing any real number matrix into the product of three specific matrices, with the advantages of stable calculation and strong anti-noise ability.

[0082] SCARA: Selective Compliance Assembly Robot Arm, a selective compliance assembly robot arm, used for automated assembly tasks, usually with a rotatable size arm, a first joint for vertical movement, and a second joint rotating around the Z axis.

[0083] In this embodiment, for ease of description, the following describes the control device for identifying the robot hand-eye calibration method as the execution subject.

[0084] Currently, the calibration method of eye on hand usually adopts the following steps: fixing a calibration board (such as a chessboard) in the workspace, controlling the camera to move above the calibration board to take pictures to obtain the pixel coordinates of the feature points. Then control the robot end (or a feature point) to move to multiple feature points (such as corner points) on the calibration board, and record the corresponding robot world coordinates of each feature point. Through multiple sets of corresponding points of “pixel coordinates-world coordinates”, the hand-eye transformation matrix is calculated by using the least squares method.

[0085] However, this method has significant drawbacks. First, the method is tedious and inefficient, requiring manual accurate control of the robot end to align with each feature point on the calibration board, which is time-consuming and labor-intensive. Second, the calibration accuracy is greatly affected by human factors, and the accuracy of manual point determination directly determines the calibration accuracy, with unavoidable human errors, making it difficult to guarantee high repeatability and high accuracy. Finally, for the case where the camera optical axis is parallel to the robot end effector, the hand-eye matrix calculated using the least squares method cannot guarantee that its rotation matrix is orthogonal, which to some extent limits the accuracy of the calibration.

[0086] The application provides a solution, which unifies a plurality of pixel coordinates obtained by dynamic shooting to a reference pixel coordinate system consistent with the direction of the camera coordinate system at the reference position, and simultaneously performs localization processing on the world coordinates of the end effector corresponding to each sampling position relative to the world coordinates of the end effector at the reference position, so as to focus the hand-eye calibration problem on the mapping relationship constructed by the relative displacement and attitude change. This strategy not only eliminates the interference brought by the global coordinate offset and the camera orientation difference, but also makes the calibration model more suitable for different working areas and motion trajectories, effectively expanding the application range of the hand-eye calibration method. On this basis, the singular value decomposition algorithm is used to solve the mapping relationship between the pixel coordinates in the reference pixel coordinate system and the local world coordinates, so that the rotation matrix solved can meet the orthogonality constraint, and then the hand-eye transformation strictly conforms to the robot motion model with translation and rotation relationship, thereby improving the calibration accuracy of the hand-eye calibration. Moreover, the calibration process of the application can be automatically calculated, so as to improve the automation degree and the calibration efficiency of the hand-eye calibration.

[0087] Based on this, the application embodiment provides a robot hand-eye calibration method. The robot comprises a first rotating arm, a second rotating arm and an end effector, a camera parallel to the end effector is installed on the second rotating arm, and the SCARA robot is taken as an example, referring to Figure 3 , Figure 3 The SCARA robot structure schematic diagram provided by the first embodiment of the robot hand-eye calibration method of the application is shown in the figure, wherein J1, J2 and J4 are respectively joints of the robot, L1 and L2 are respectively a large arm (i.e. the first rotating arm) and a small arm (i.e. the second rotating arm), J3 is an installation joint of the camera optical axis, the camera and J3 only have rotation and translation, x b o b y b is the world coordinate system of the robot base, x c o c y c is the pixel coordinate system of the camera. Referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the robot hand-eye calibration method of the application is shown in the figure.

[0088] In this embodiment, the robot hand-eye calibration method comprises steps S1-S5:

[0089] In step S1, the robot motion is controlled, so that the camera moves to an initial position above the calibration object, the corresponding position of the robot is recorded as a reference position, the calibration object is photographed at the reference position to obtain pixel coordinates, and the world coordinates of the end effector in the base coordinate system and the joint angles of the first rotating arm and the second rotating arm are recorded.

[0090] In this step, the calibration object is placed in the workspace of the robot, and then the motion of the robot is controlled by the control device to move the camera on the second rotating arm (i.e., the small arm) to the initial position of the calibration object. A unified reference point is established based on the initial position as the reference position, which is used as the comparison reference for all subsequent sampling points. The position defines the reference camera orientation and observation angle, and the world coordinates of the end effector at this time and the joint angles of the large and small arms are recorded.

[0091] In step S2, a regular grid motion path containing a plurality of sampling points is planned centered on the reference position. The robot is controlled to move to the remaining plurality of sampling positions except the reference position in turn according to the regular grid motion path. The calibration object is photographed to obtain corresponding pixel coordinates, and the world coordinates of the end effector and the joint angles of the first and second rotating arms at each position are recorded.

[0092] It should be noted that due to the scale and direction ambiguity of a single perspective, the hand-eye transformation cannot be uniquely determined. In this step, a regular grid (such as a 3x3 or 5x5 point array) is planned with the reference position as the reference point. The robot is controlled to move to these sampling points in turn, and the pixel coordinates, world coordinates of the end effector, and joint angles of the large and small arms at the corresponding positions are recorded each time.

[0093] In step S3, based on the pixel coordinates obtained at the remaining plurality of sampling positions, the joint angles of the first and second rotating arms, and the pixel coordinates obtained at the reference position, the joint angles of the first and second rotating arms, the remaining plurality of pixel coordinates are inversely rotated and transformed to be uniformly corrected to the reference pixel coordinate system consistent with the direction of the camera coordinate system at the reference position.

[0094] It should be noted that when the robot moves in translation, it will cause the camera coordinate system to rotate, resulting in inconsistent directions of the pixel coordinate systems collected at different positions, i.e., the angle between the camera coordinate system and the robot base coordinate is not fixed. To solve this problem, in this step, the pixel coordinates are inversely rotated and transformed according to the camera rotation angle of each position relative to the reference position, so that they are uniformly transformed to the same direction as the camera coordinate system at the reference position, i.e., to the same reference pixel coordinate system.

[0095] In step S4, based on the world coordinates of the end effector at each position and the world coordinates of the end effector at the reference position, the local world coordinates of each position with the reference position as the origin are calculated. In step S5, based on the pixel coordinates of each position in the reference pixel coordinate system and the corresponding local world coordinates, the rotation matrix and translation vector of the hand-eye transformation matrix are calculated using the singular value decomposition algorithm.

[0096] It should be noted that since the camera is mounted on the robot arm, the camera moves with the robot, and the position of the photograph is not fixed. If the hand-eye matrix is directly calibrated using the coordinates in the robot base coordinate system, the relationship between the camera and the base is obtained. In order to obtain the correct application, it is necessary to limit the camera to take pictures at the initial position of the calibration object each time. Otherwise, it will be wrong, which limits the application range of the robot. In this step, the local world coordinates of each position are calculated with the reference position as the origin. The local world coordinates are parallel to the x-axis and y-axis directions of the base coordinate system, which can eliminate the absolute offset of the world coordinates in the base coordinate system and focus on the relative displacement to establish the mapping relationship.

[0097] In step S5, based on the pixel coordinates of each position in the reference pixel coordinate system and the corresponding local world coordinates, the singular value decomposition algorithm is used to calculate the rotation matrix and translation vector of the hand-eye transformation matrix.

[0098] It should be noted that the commonly used calibration calculation method is to regard the pixel coordinates and the world coordinates as a coordinate mapping problem, and to calculate the transformation matrix by using the least square method. However, for the robot structure in which the camera coordinate system and the robot base coordinate system have a rotation relationship, for example, for the SCARA robot, the camera is mounted on the robot arm, and the camera optical axis is parallel to the robot end effector installation. At this time, the camera coordinate system and the robot base coordinate system have both rotation and translation relationship. The rotation component of the transformation matrix calculated by using the least square method is not necessarily orthogonal, which limits the accuracy of the solution matrix to some extent. In this step, the singular value decomposition algorithm is used for calibration, which can improve the accuracy and numerical stability of the calibration.

[0099] In this embodiment, the plurality of pixel coordinates obtained by dynamic shooting are uniformly corrected to the reference pixel coordinate system which is consistent with the direction of the camera coordinate system at the reference position, and the world coordinates of the end effector corresponding to each sampling position are localized with respect to the world coordinates of the end effector at the reference position, so as to focus on the mapping relationship constructed by the relative displacement and the attitude change. This strategy not only eliminates the interference of global coordinate offset and camera orientation difference, but also makes the calibration model more suitable for different working areas and motion trajectories, effectively expanding the application range of the hand-eye calibration method. On this basis, the singular value decomposition algorithm is used to solve the mapping relationship between the pixel coordinates in the reference pixel coordinate system and the local world coordinates, so that the rotation matrix satisfies the orthogonality constraint. Then, the hand-eye transformation strictly conforms to the robot motion model with translation and rotation relationship, and the calibration accuracy of the hand-eye calibration is improved. Moreover, the calibration process of the embodiment can be automatically calculated, which can improve the automation degree and the calibration efficiency of the hand-eye calibration.

[0100] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and the subsequent will not be described. On this basis, step S2 includes steps S21-S22:

[0101] Step S21, based on the preset horizontal translation interval and vertical translation interval, a 3x3 grid motion path is planned with the reference position as the center.

[0102] Step S22, control the robot to move to the remaining 8 sampling positions in turn according to the planned path, respectively take pictures to obtain the corresponding pixel coordinates, and record the world coordinates of the end effector at each position and the joint angles of the first and second rotating arms.

[0103] Reference Figure 4 , Figure 4 The nine-grid path planning schematic diagram provided by the first embodiment of the robot hand-eye calibration method of the present application. In this embodiment, the camera moves above the calibration object as the initial reference position P0, and then a 3x3 nine-grid motion path is planned with P0 as the center, which contains 9 photographing positions (including P0 itself), the horizontal translation interval and the vertical translation interval of each position and the adjacent position are the same, which are the preset horizontal translation interval and the vertical translation interval. In this embodiment, first, according to the translation interval of the X axis △X, control the robot to move to three positions P1, P2, P3 in the X direction and take pictures respectively, and then move in the Y direction according to the translation interval of the Y axis △Y, and then continue to move in the X direction to three positions P4, P0, P5 and take pictures of the calibration object respectively, until the nine-grid motion path is completed, and the corresponding information is photographed and recorded. Among them, the joint angle of the first rotating arm is the rotation angle of joint J1, and the joint angle of the second rotating arm is the rotation angle of joint J2.

[0104] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and the subsequent will not be described. On this basis, step S3 includes steps S31-S32:

[0105] Step S31, calculate the joint angle change of the remaining multiple sampling positions relative to the reference position;

[0106] Among them

[0107]

[0108] For the joint angle change of the corresponding sampling position relative to the reference position, For the joint angle of the first rotating arm at the corresponding sampling position, For the joint angle of the second rotating arm at the corresponding sampling position, The joint angle of the first rotating arm at the reference position. This represents the joint angle of the second rotating arm in the reference position.

[0109] Step S32: Rotate the pixel coordinates collected at each sampling position in the reverse direction around the reference position to obtain the pixel coordinates in the reference pixel coordinate system.

[0110] in

[0111]

[0112] This represents the change in joint angle between the sampling position and the reference position. These are the pixel coordinates of the corresponding sampling position. The corresponding sampling position is the pixel coordinate in the reference pixel coordinate system. These are the pixel coordinates corresponding to the reference position.

[0113] In this embodiment, the camera position at the reference position is used as a reference to calculate the change in joint angle relative to the reference position at each subsequent position. This refers to the rotation of the camera coordinate system. The pixel coordinates acquired at each location are rotated in the opposite direction around the feature point reference position. Obtain the rotated and corrected pixel coordinates. .like Figure 5 As shown in Figure a, the pixel position P in the camera coordinate system uov is obtained by taking a picture of the calibration object at the reference position P0. p0 Then, based on the 3x3 grid path at position P i The pixel position P in the camera coordinate system uoiv is obtained by taking a picture of the calibrated object. pi Because the robot's manipulator rotates the camera during translation, the camera coordinate system uoiv rotates and translates relative to uov, thus causing P... p0 and P pi They are not in the same camera coordinate system, and their rotation angle is... Therefore, it is necessary to change the coordinate P. pi Transform to P p0 Within the same coordinate system. For example... Figure 5 As shown in b, make P pi Around the center of rotation P p0 Rotation - Obtain the pixel coordinates P in the reference pixel coordinate system pi ' .

[0114] Based on the first embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above description, and the subsequent will not be described in detail. On this basis, step S4 includes step S41:

[0115] The world coordinates P wi (x i ,y i ) of each position are subtracted from the world coordinates P w0 (x0,y0) of the reference position to obtain the local world coordinates with the reference position as the origin;

[0116] Wherein

[0117] .

[0118] In this embodiment, please refer to Figure 6 , the construction of the local world coordinate system x l o l y l takes the robot end as the origin of the coordinate system, and the x and y axes are parallel to the x and y axes of the robot base coordinate system xoy. Align the robot end with the reference position P0, and subtract the world coordinates P w0 (x0,y0) of the reference position P0 from the robot world coordinates P wi (x i ,y i ) recorded at all 9 positions to obtain the local world coordinates with the reference position as the origin.

[0119] Based on the first embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above description, and the subsequent will not be described in detail. On this basis, step S5 includes steps S51-S54:

[0120] Step S51, the pixel coordinates of each position in the reference pixel coordinate system and the corresponding local world coordinates are centralized to obtain the central pixel coordinate point set and the central world coordinate point set; wherein

[0121] Formula one,

[0122]

[0123]

[0124]

[0125] The central pixel coordinate point set is , the pixel coordinates of each position in the reference pixel coordinate system are , and the central pixel coordinates are the number of the rest of the sampling positions, the set of central world coordinates, the local world coordinates of each position, the central world coordinates.

[0126] Step S52, calculate the covariance matrix between the set of central pixel coordinates and the set of central world coordinates ;

[0127] wherein

[0128] Equation Two,

[0129]

[0130] Step S53, singular value decomposition is performed on the covariance matrix to obtain the rotation matrix ;

[0131] wherein

[0132] Equation Three,

[0133] ,

[0134] , , are the left singular vector matrix, the diagonal singular value matrix and the right singular vector matrix, respectively.

[0135] Step S54, based on the rotation matrix, the central pixel coordinates and the central world coordinates, a translation vector is calculated;

[0136] wherein

[0137] Equation Four,

[0138] ,

[0139] is the translation vector, is the hand-eye transformation matrix.

[0140] In an embodiment, before step S54, the robot hand-eye calibration method further comprises step S5401:

[0141] If the determinant of the rotation matrix is less than 0, the last column of the right singular vector matrix is multiplied by -1, and the rotation matrix is recalculated according to .

[0142] It should be noted that since the transformation matrix calculated by using the least square method cannot guarantee that its rotation component is orthogonal, this limits the accuracy of the solution matrix to some extent. In this embodiment, taking the nine-square motion path as an example, a total of 9 position information is sampled, n in formula one and formula two is 8, and the mapping problem of the pixel coordinates and the world coordinate point set is regarded as the registration problem between two point sets as shown in formula one:

[0143] Formula five,

[0144]

[0145]

[0146]

[0147] wherein, is the pixel coordinate of the 9 positions in the reference pixel coordinate system, is the local world coordinate of the 9 positions, is the rotation matrix to be solved, is the translation vector to be solved. In this embodiment, the centering process is performed by formula one, which helps to reduce the data difference between the pixel coordinates and the local world coordinates, and facilitates the subsequent calculation process. According to formula two to formula four, the rotation matrix and the translation vector can be calculated.

[0148] wherein, based on step S5401, the rotation matrix is checked, if the determinant of is less than 0, i.e.

[0149]

[0150] the last column of the right singular vector matrix is multiplied by-1, and the rotation matrix is recalculated according to . In this way, the solved rotation matrix can meet the orthogonality constraint, so that the hand-eye transformation strictly meets the robot motion model containing translation and rotation relationship, and further improves the automation degree, calibration efficiency and calibration accuracy of the hand-eye calibration.

[0151] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the robot hand-eye calibration method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0152] The application further provides a control device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot hand-eye calibration method as described above. The control device provided by the application adopts the robot hand-eye calibration method in the above embodiments, and can solve the technical problems of low automation, poor calibration efficiency, and difficult-to-ensure accuracy of existing calibration methods. Compared with the prior art, the control device provided by the application has the same beneficial effects as the robot hand-eye calibration method provided by the above embodiments, and other technical features in the control device are the same as the features disclosed in the above embodiments, which will not be described here.

[0153] The application further provides a robot system, comprising a robot and the control device as described above, wherein the robot has a first rotating arm, a second rotating arm, and an end effector, and a camera parallel to the end effector is installed on the second rotating arm. The robot system provided by the application adopts the robot hand-eye calibration method in the above embodiments, and can solve the technical problems of low automation, poor calibration efficiency, and difficult-to-ensure accuracy of existing calibration methods. Compared with the prior art, the robot system provided by the application has the same beneficial effects as the robot hand-eye calibration method provided by the above embodiments, and other technical features in the robot system are the same as the features disclosed in the above embodiments, which will not be described here.

[0154] In an embodiment, referring to Figure 2 The control device is further configured to implement steps S61-S62.

[0155] In step S61, the robot is controlled to move the camera to an arbitrary position, denoted as a target position, and a target pixel coordinate is obtained by taking a picture of a target object at the target position, and a world coordinate of the end effector in the base coordinate system is read.

[0156] In step S62, based on the target pixel coordinate, the hand-eye transformation matrix, and the world coordinate of the end effector in the base coordinate system, a global world coordinate of the target object in the base coordinate system obtained by the camera taking a picture when the camera moves to the target position is calculated.

[0157] In an embodiment, step S62 specifically comprises:

[0158] correcting the target pixel coordinate to a reference pixel coordinate system to obtain a pixel coordinate of the target pixel coordinate in the reference pixel coordinate system;

[0159] wherein

[0160] .

[0161]

[0162] is a joint angle variation of the target position relative to the reference position, is a target pixel coordinate, is a pixel coordinate of the target pixel coordinate in the reference pixel coordinate system, is a pixel coordinate corresponding to the reference position, is a joint angle of the first rotating arm at the target position, is a joint angle of the second rotating arm at the target position, is a joint angle of the first rotating arm at the reference position, is a joint angle of the second rotating arm at the reference position;

[0163] Based on the pixel coordinate of the target pixel coordinate in the reference pixel coordinate system, the hand-eye transformation matrix, and the world coordinate of the end effector in the base coordinate system, the global world coordinate of the target object in the base coordinate system obtained by the camera photographing when the camera moves to the target position is calculated.

[0164] wherein

[0165]

[0166] (x, y) is the world coordinate of the end effector in the base coordinate system at the corresponding position, , are a rotation matrix and a translation vector, respectively, is the global world coordinate of the target object in the base coordinate system obtained by the camera photographing when the camera moves to the target position.

[0167] In the embodiment, the robot is no longer limited to identification and operation in a fixed workspace, but can freely move in a large workspace and dynamically perceive the environment. In the embodiment, the camera is installed on the second rotating arm. After the hand-eye relationship between the camera and the local world coordinate is calibrated, in actual application, the robot moves the camera to an arbitrary position (denoted as a target position), the camera photographs the target object at the target position to obtain a target pixel coordinate, then the target pixel coordinate is converted to a reference pixel coordinate system, and the pixel coordinate in the reference pixel coordinate system is converted to a local world coordinate using the calibrated hand-eye matrix, and then the read world coordinate of the end effector in the base coordinate system is added or subtracted from the local world coordinate at this time (the addition or subtraction is determined according to whether the direction of the reference pixel coordinate system of the camera and the local world coordinate system is the same or opposite), to obtain the global world coordinate of the target object in the base coordinate system. In this way, the robot manipulator can be guided to realize precise movement and grabbing.

[0168] It should be understood that portions of the application disclosed can be implemented in hardware, software, firmware, or combinations thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner without departing from the scope of the present application.

[0169] The above merely provides part of embodiments of the present application and does not limit the patent scope of the present application. Any equivalent structure transformation made by using the content of the present application specification and drawings or directly / indirectly applied in other related technical fields under the technical concept of the present application is included in the patent protection scope of the present application.

Claims

1. A method for robot hand-eye calibration, characterized in that, The robot includes a first rotating arm, a second rotating arm, and an end effector. A camera parallel to the end effector is mounted on the second rotating arm. The steps of the method include: S1. Control the robot to move, so that the camera moves to the initial position above the calibration object, record the position corresponding to the robot as the reference position, take a picture of the calibration object at the reference position to obtain pixel coordinates, and record the world coordinates of the end effector in the base coordinate system and the joint angle between the first rotating arm and the second rotating arm. S2. Using the reference position as the center, plan a regular grid motion path containing multiple sampling points, control the robot to move sequentially to multiple sampling positions other than the reference position according to the regular grid motion path, take pictures of the calibration object to obtain the corresponding pixel coordinates, and record the world coordinates of the end effector and the joint angle between the first rotating arm and the second rotating arm at each position. S3. Based on the pixel coordinates obtained from the other multiple sampling positions, the joint angles of the first rotating arm and the second rotating arm, and the pixel coordinates obtained from the reference position, the other multiple pixel coordinates are subjected to reverse rotation transformation to uniformly correct them to the reference pixel coordinate system that is consistent with the direction of the camera coordinate system at the reference position. S4. Based on the world coordinates of the end effector at each location and the world coordinates of the end effector at the reference location, calculate the local world coordinates of each location with the reference location as the origin; S5. Based on the pixel coordinates of each position in the reference pixel coordinate system and the corresponding local world coordinates, the rotation matrix and translation vector of the hand-eye transformation matrix are calculated using the singular value decomposition algorithm.

2. The method as described in claim 1, characterized in that, Step S5 includes: Centering the pixel coordinates and corresponding local world coordinates of each location in the reference pixel coordinate system yields a set of central pixel coordinate points and a set of central world coordinate points; where... For the set of center pixel coordinates, These are the pixel coordinates of each position in the reference pixel coordinate system. The center pixel coordinates, For the number of the remaining multiple sampling locations, For the set of central world coordinate points, For each location, the local world coordinates are... Using the world coordinates as the center; Calculate the covariance matrix between the set of center pixel coordinates and the set of center world coordinates. ; in For the covariance matrix Perform singular value decomposition to obtain the rotation matrix. ; in ， , , These are the left singular vector matrix, the diagonal singular value matrix, and the right singular vector matrix, respectively. The translation vector is calculated based on the rotation matrix, the center pixel coordinates, and the center world coordinates; in ， Let be the translation vector. Let be the hand-eye transformation matrix.

3. The method as described in claim 2, characterized in that, Before the step of calculating the translation vector based on the rotation matrix, the center pixel coordinates, and the center world coordinates, the method further includes: If the rotation matrix If the determinant is less than 0, then the right singular vector matrix will be... Multiply the last column by -1, according to Recalculate the rotation matrix .

4. The method as described in claim 1, characterized in that, Step S2 includes: Based on the preset horizontal and vertical translation intervals, a nine-square grid movement path centered on the reference position is planned. The robot is controlled to move sequentially to the remaining 8 sampling positions along the planned path, take pictures to obtain the corresponding pixel coordinates, and record the world coordinates of the end effector and the joint angle between the first and second rotating arms at each position.

5. The method as described in claim 1, characterized in that, Step S3 includes: Calculate the joint angle changes of the remaining multiple sampling positions relative to the reference position; in This represents the change in joint angle relative to the reference position at the corresponding sampling position. This represents the joint angle of the first rotating arm at the corresponding sampling position. This refers to the joint angle of the second rotating arm at the corresponding sampling position. The joint angle of the first rotating arm at the reference position. The joint angle of the second rotating arm at the reference position; The pixel coordinates collected at each sampling position are rotated in the opposite direction around the reference position to obtain the pixel coordinates in the reference pixel coordinate system; in This represents the change in joint angle relative to the reference position at the corresponding sampling position. These are the pixel coordinates of the corresponding sampling position. The corresponding sampling position is the pixel coordinate in the reference pixel coordinate system. These are the pixel coordinates corresponding to the reference position.

6. The method as described in claim 1, characterized in that, Step S4 includes: Set the world coordinates P of each location wi (x i ,y i Subtract the world coordinates P of the reference position w0 (x0, y0) is used to obtain the local world coordinates with the reference position as the origin; in 。 7. A control device, characterized in that, The control device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot hand-eye calibration method as described in any one of claims 1 to 6.

8. A robot system, characterized in that, The system includes a robot and the control device as described in claim 7, wherein the robot has a first rotating arm, a second rotating arm and an end effector, and a camera is mounted on the second rotating arm parallel to the end effector.

9. The robot system as described in claim 8, characterized in that, The control device is also used for: Control the robot's movement to move the camera to any position, which is recorded as the target position. Take a picture of the target object at the target position to obtain the target pixel coordinates, and read the world coordinates of the end effector in the base coordinate system. as well as Based on the target pixel coordinates, the hand-eye transformation matrix, and the world coordinates of the end effector in the base coordinate system, the global world coordinates of the target object in the base coordinate system obtained by the camera taking a picture when the camera moves to the target position are calculated.

10. The robot system as described in claim 9, characterized in that, The control device is also specifically used for: The target pixel coordinates are corrected to the reference pixel coordinate system to obtain the pixel coordinates of the target pixel coordinates in the reference pixel coordinate system; in The change in joint angle between the target position and the reference position. The target pixel coordinates, The target pixel coordinates are the pixel coordinates in the reference pixel coordinate system. The pixel coordinates corresponding to the reference position. The joint angle of the first rotating arm at the target position. The joint angle of the second rotating arm at the target position. The joint angle of the first rotating arm at the reference position. The joint angle of the second rotating arm at the reference position; Based on the pixel coordinates of the target pixel coordinates in the reference pixel coordinate system, the hand-eye transformation matrix and the world coordinates of the end effector in the base coordinate system, the global world coordinates of the target object in the base coordinate system obtained by the camera when the camera moves to the target position are calculated; in (x, y) represent the world coordinates of the end effector at the corresponding position in the base coordinate system. , The rotation matrix and the translation vector are respectively, The global world coordinates of the target object in the base coordinate system obtained by taking a picture of the target object when the camera moves to the target position.

Images