Robot hand-eye calibration method and device, computer device and storage medium

Abstract

The application relates to a robot hand-eye calibration method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a plurality of hand-eye calibration data sets, selecting each hand-eye calibration data set as a verification data set in turn, taking the remaining hand-eye calibration data sets as training data sets, obtaining a plurality of sets of calibration data division results, for each set of calibration data division result, performing hand-eye calibration on the robot according to the training data set, obtaining a hand-eye calibration matrix, verifying the hand-eye calibration matrix according to the verification data set, obtaining a hand-eye calibration error, screening out the smallest first hand-eye calibration error, and the corresponding hand-eye calibration matrix is a first hand-eye calibration matrix, correcting the first hand-eye calibration matrix based on the first hand-eye calibration error, a second hand-eye calibration error except the first hand-eye calibration error and a second hand-eye calibration matrix, and determining a target hand-eye calibration matrix. The method can improve the hand-eye calibration precision.

Description

Technical Field

[0001] This application relates to the field of robot calibration technology, and in particular to a robot hand-eye calibration method, apparatus, computer equipment, storage medium and computer program product. Background Technology

[0002] With the rapid development of industrial automation and intelligent manufacturing, robots are being used more and more widely in various production and operational tasks. In order for robots to perform precise movements based on visual information, hand-eye calibration is required before using them.

[0003] In traditional solutions, a planar calibration board, such as a black and white checkerboard, is typically used. The camera at the end of the robotic arm is controlled to capture images of the calibration board in different postures of the robotic arm, and the robot's hand-eye calibration is performed using a hand-eye calibration formula.

[0004] However, traditional hand-eye calibration schemes require collecting the different poses of the robotic arm end effector relative to the robot's base coordinate system. In most cases, the robotic arm end effector needs to be equipped with tools, such as grippers. Whether it is manual collection or algorithm calibration, the pose of the robotic arm end effector will have a large error, resulting in low hand-eye calibration accuracy. Summary of the Invention

[0005] Therefore, it is necessary to provide a robot hand-eye calibration method, device, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of hand-eye calibration, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a method for robot hand-eye calibration. The method includes:

[0007] Acquire multiple hand-eye calibration datasets for robots;

[0008] Each hand-eye calibration dataset is selected in turn as the validation dataset, and the remaining hand-eye calibration datasets other than the selected ones are used as the training datasets to obtain multiple sets of calibration data partitioning results;

[0009] For each set of calibration data partitioning results, the robot is calibrated using the training dataset in the calibration data partitioning results to obtain a hand-eye calibration matrix. The hand-eye calibration matrix is ​​then validated using the validation dataset in the calibration data partitioning results to obtain the hand-eye calibration error corresponding to the calibration data partitioning results.

[0010] The smallest first hand-eye calibration error is selected from the hand-eye calibration errors corresponding to the multiple sets of calibration data division results, and the hand-eye calibration matrix corresponding to the first hand-eye calibration error is determined as the first hand-eye calibration matrix of the robot.

[0011] Based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error, the first hand-eye calibration matrix is ​​corrected to determine the target hand-eye calibration matrix.

[0012] Secondly, this application also provides a robot hand-eye calibration device. The device includes:

[0013] The data acquisition module is used to acquire multiple hand-eye calibration datasets for the robot;

[0014] The data partitioning module is used to select each hand-eye calibration dataset as the validation dataset in turn, and use the remaining hand-eye calibration datasets other than the selected hand-eye calibration datasets as the training datasets, so as to obtain multiple sets of calibration data partitioning results;

[0015] The hand-eye calibration module is used to perform hand-eye calibration on the robot based on the training dataset in each set of calibration data division results, to obtain a hand-eye calibration matrix, and to verify the hand-eye calibration matrix based on the verification dataset in the calibration data division results, to obtain the hand-eye calibration error corresponding to the calibration data division results.

[0016] The first optimization module is used to select the smallest first hand-eye calibration error from the hand-eye calibration errors corresponding to the division results of multiple sets of calibration data, and to determine the hand-eye calibration matrix corresponding to the first hand-eye calibration error as the first hand-eye calibration matrix of the robot.

[0017] The second optimization module is used to correct the first hand-eye calibration matrix and determine the target hand-eye calibration matrix based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error.

[0018] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps described in the robot hand-eye calibration method embodiments above.

[0019] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps in the above-described embodiments of the robot hand-eye calibration method.

[0020] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps described in the embodiments of the robot hand-eye calibration method.

[0021] The aforementioned robot hand-eye calibration method, apparatus, computer equipment, storage medium, and computer program product first acquire multiple hand-eye calibration datasets for the robot. These datasets are then used in turn as validation datasets, with the remainder used as training datasets, resulting in different data partitioning results. Further, for each data partitioning result, the robot's hand-eye calibration is performed using the training dataset to obtain a hand-eye calibration matrix. This matrix is ​​then validated using the validation datasets, and the hand-eye calibration error is calculated. Finally, the smallest first hand-eye calibration error is selected from the multiple hand-eye calibration errors, and the corresponding hand-eye calibration matrix is ​​determined as the robot's first hand-eye calibration matrix. Based on the first hand-eye calibration error, the second hand-eye calibration error (excluding the first hand-eye calibration error), and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error, the first hand-eye calibration matrix is ​​further modified to determine the target hand-eye calibration matrix. Unlike traditional solutions that rely on a single hand-eye calibration method, this solution utilizes dual verification and cross-verification to fully leverage existing data resources and evaluate and analyze robot hand-eye calibration from multiple perspectives and combinations. This reduces random errors that may occur during a single calibration, thereby improving the accuracy of robot hand-eye calibration. Attached Figure Description

[0022] Figure 1 This is an application environment diagram of the robot hand-eye calibration method in one embodiment;

[0023] Figure 2 This is a flowchart illustrating a robot hand-eye calibration method in one embodiment;

[0024] Figure 3 This is a flowchart illustrating the optimization of the first-hand eye calibration matrix in one embodiment;

[0025] Figure 4 A flowchart illustrating the optimization of the first-hand eye calibration matrix in another embodiment;

[0026] Figure 5 This is a flowchart illustrating the robot hand-eye calibration method in another embodiment;

[0027] Figure 6 This is a flowchart illustrating a robot hand-eye calibration method in a detailed embodiment;

[0028] Figure 7 This is a structural block diagram of a robot hand-eye calibration device in one embodiment;

[0029] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] The robot hand-eye calibration method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0032] Specifically, staff can upload multiple hand-eye calibration datasets for the robot to server 104 via terminal 102. Then, server 104 sequentially selects each hand-eye calibration dataset as a validation dataset and uses the remaining datasets as training datasets, obtaining multiple sets of calibration data partitioning results. For each set of partitioning results, hand-eye calibration is performed on the robot based on the training dataset, resulting in a hand-eye calibration matrix. The hand-eye calibration matrix is ​​then validated using the validation dataset, yielding the hand-eye calibration error corresponding to the partitioning results. Further, server 104 selects the smallest first hand-eye calibration error from the multiple sets of calibration data partitioning errors and determines the first hand-eye calibration matrix corresponding to this error as the robot's first hand-eye calibration matrix. Finally, server 104 corrects the first hand-eye calibration matrix based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error, and determines the target hand-eye calibration matrix.

[0033] Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0034] In one embodiment, such as Figure 2 As shown, a robot hand-eye calibration method is provided, which can be applied to... Figure 1Taking server 104 as an example, the following steps are included:

[0035] S100: Obtain multiple hand-eye calibration datasets for the robot, select each hand-eye calibration dataset as the validation dataset in turn, and use the remaining hand-eye calibration datasets other than the selected hand-eye calibration datasets as the training datasets to obtain multiple sets of calibration data partitioning results.

[0036] Hand-eye calibration is the process of determining the spatial position and posture transformation relationship between the robot's end effector (hand) and the camera (eye). In a robot system, the information acquired by the camera is based on the camera coordinate system, while the actions of the end effector are based on the end effector coordinate system. To enable the robot to perform precise actions based on visual information, it is necessary to first determine the transformation relationship between the camera coordinate system and the end effector coordinate system, i.e., to perform hand-eye calibration on the robot. The hand-eye calibration dataset is a collection of data used for hand-eye calibration, containing the pose data of the end effector (hereinafter referred to as end effector pose) and the data acquired by the camera. The end effector pose is the position and posture information of the robot's end effector in the robot's base coordinate system. The data acquired by the camera can be a calibration image of a calibration board or other object taken by the camera, the pixel coordinates of the calibration point in the camera coordinate system, etc. It should be noted that this method can be used for hand-eye calibration regardless of whether the eye is on the hand or outside the hand.

[0037] Specifically, after obtaining the aforementioned hand-eye calibration dataset for the robot, the server uses a cross-validation approach to select each hand-eye calibration dataset as a validation dataset in turn. For example, assuming there are five hand-eye calibration datasets, numbered 1 to 5, the first time, dataset 1 is used as the validation dataset, while the remaining datasets 2 to 5 become the training datasets. The second time, dataset 2 is used as the validation dataset, while datasets 1, 3, 4, and 5 use dataset 1 as the training dataset, and so on, until every hand-eye calibration dataset has a chance to be selected as a validation dataset. This results in multiple different sets of calibration data partitioning results, each corresponding to a specific combination of validation and training datasets. Subsequent calibration and validation operations can then be performed based on these different partitioning results.

[0038] S200: For each set of calibration data partitioning results, perform hand-eye calibration on the robot based on the training dataset in the calibration data partitioning results to obtain the hand-eye calibration matrix. Then, verify the hand-eye calibration matrix based on the verification dataset in the calibration data partitioning results to obtain the hand-eye calibration error corresponding to the calibration data partitioning results.

[0039] The hand-eye calibration matrix represents the result of robot hand-eye calibration, characterizing the transformation relationship between the camera coordinate system and the robot arm end effector coordinate system. Following the steps above, for each set of calibration data partitioning results, a training dataset can be extracted. As mentioned earlier, the training dataset contains the pose information of the robot's end effector in the robot's base coordinate system, as well as calibration images of the calibration board or other objects captured by the camera, and the pixel coordinates of the calibration points in the camera coordinate system.

[0040] Furthermore, based on a pre-defined hand-eye calibration mathematical model (e.g. Using mathematical models, the hand-eye calibration equations can be constructed by using the robotic arm end-effector pose and the calibration images captured by the camera in the training dataset. This allows the construction of the hand-eye calibration matrix, which can then be solved using multiple sets of data points (different robotic arm end-effector poses and corresponding calibration images captured by the camera).

[0041] Then, the validation dataset is taken from the same set of calibration data division results. It also contains the end-effector pose of the robotic arm and the calibration images captured by the camera. Using the hand-eye calibration matrix obtained based on the training dataset and the end-effector pose of the robotic arm in the validation dataset, the calibration images that the camera should theoretically capture under the current calibration results are calculated according to the principle of hand-eye coordinate transformation. The calibration images that should theoretically be captured are compared with the calibration images that are actually captured in the validation dataset. The difference between the two is the hand-eye calibration error.

[0042] S300 selects the smallest first hand-eye calibration error from the hand-eye calibration errors corresponding to the multiple sets of calibration data division results, and determines the hand-eye calibration matrix corresponding to the first hand-eye calibration error as the robot's first hand-eye calibration matrix.

[0043] By performing hand-eye calibration and verification on the results of the multiple sets of calibration data as described above, and obtaining the corresponding hand-eye calibration errors, the smallest error (i.e., the first hand-eye calibration error) can be selected from among them. This allows us to find the calibration result (i.e., the first hand-eye calibration matrix) that most accurately reflects the real spatial position and attitude transformation relationship between the robot end effector and the camera under many different data combinations and calibration methods.

[0044] In the set of hand-eye calibration errors corresponding to the multiple sets of calibration data division results already obtained, the various hand-eye calibration errors are compared. For example, suppose there are 5 sets of calibration data division results, and the corresponding hand-eye calibration errors are error 1, error 2, error 3, error 4, and error 5. The magnitudes of these error values ​​are compared one by one, and the error with the smallest value is determined and defined as the first hand-eye calibration error. Then, the first hand-eye calibration matrix corresponding to the first hand-eye calibration error is traced back.

[0045] S400, based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error, corrects the first hand-eye calibration matrix and determines the target hand-eye calibration matrix.

[0046] Following the steps above, after selecting the first hand-eye calibration matrix with the smallest calibration error, further optimization can be performed. For example, introducing the second hand-eye calibration error (in addition to the first hand-eye calibration error) and its corresponding second hand-eye calibration matrix to correct the first hand-eye calibration matrix can integrate more calibration information, further optimize the calibration results, and ultimately obtain a more accurate and reliable target hand-eye calibration matrix.

[0047] The aforementioned robot hand-eye calibration method first acquires multiple hand-eye calibration datasets for the robot. These datasets are then used in turn as validation datasets, with the remainder used as training datasets, resulting in different data partitioning outcomes. Further, for each data partitioning outcome, the robot's hand-eye calibration is performed using the training dataset to obtain a hand-eye calibration matrix. This matrix is ​​then validated using the validation datasets, and the hand-eye calibration error is calculated. Finally, the smallest first hand-eye calibration error is selected from the multiple errors, and the corresponding hand-eye calibration matrix is ​​determined as the robot's first hand-eye calibration matrix. Based on the first hand-eye calibration error, the second hand-eye calibration error (excluding the first error), and the second hand-eye calibration matrix corresponding to the second error, the first hand-eye calibration matrix is ​​further refined to determine the target hand-eye calibration matrix. Unlike traditional solutions that rely on a single hand-eye calibration method, this solution utilizes dual verification and cross-verification to fully leverage existing data resources and evaluate and analyze robot hand-eye calibration from multiple perspectives and combinations. This reduces random errors that may occur during a single calibration, thereby improving the accuracy of robot hand-eye calibration.

[0048] In one embodiment, such as Figure 3 As shown, S400 includes:

[0049] S410 uses the first-hand eye calibration error, each second-hand eye calibration error, and the second-hand eye calibration matrix corresponding to each second-hand eye calibration error as a set of correction data.

[0050] S420 corrects the first-hand eye calibration matrix using each set of correction data to obtain the corrected third-hand eye calibration matrix.

[0051] S430, Based on the validation dataset corresponding to the first-hand eye calibration matrix, validate the third-hand eye calibration matrix to obtain the third-hand eye calibration error.

[0052] S440: Select the target hand-eye calibration error that is smaller than the first hand-eye calibration error from multiple third hand-eye calibration errors, and determine the third hand-eye calibration matrix corresponding to the target hand-eye calibration error as the target hand-eye calibration matrix.

[0053] Following the above embodiments, the first hand-eye calibration matrix initially selected is further modified by integrating multiple sets of relevant data, thereby determining the matrix with the smallest error and better than the first hand-eye calibration error as the target hand-eye calibration matrix.

[0054] Specifically, each calibration data division result corresponds to a hand-eye calibration error. Except for the smallest first hand-eye calibration error, the remaining hand-eye calibration errors are marked as second hand-eye calibration errors. The first hand-eye calibration error, each second hand-eye calibration error, and the second hand-eye calibration matrix corresponding to each second hand-eye calibration error are used as a set of correction data, thus forming multiple different sets of correction data.

[0055] For each set of correction data, the first-hand-eye calibration matrix must be corrected accordingly to obtain the corrected third-hand-eye calibration matrix. Specifically, the error of the first-hand-eye calibration matrix can be adjusted based on a preset bias correction rate. Furthermore, different weights can be set for translation and rotation errors according to the actual calibration situation, resulting in different correction magnitudes. After obtaining each corrected third-hand-eye calibration matrix, it can be validated using the validation dataset corresponding to the first-hand-eye calibration matrix. Similar to the cross-validation mentioned above, the robot arm end-effector pose from the validation dataset can be substituted into the third-hand-eye calibration matrix, and the theoretical calibration image can be calculated according to the hand-eye coordinate transformation principle. This theoretical image is then compared with the actual calibration image in the validation dataset; the error between the two is the third-hand-eye calibration error.

[0056] Then, from multiple third-hand-eye calibration errors, the goal is to find the target hand-eye calibration error that is both the smallest and smaller than the first-hand-eye calibration error. This means selecting the calibration result with the best error performance and further optimization compared to the initial first-hand-eye calibration matrix from among numerous corrected and verified calibration results. Furthermore, after selecting the target hand-eye calibration error, its corresponding third-hand-eye calibration matrix can be traced back and identified as the target hand-eye calibration matrix.

[0057] It should be noted that the above optimization process can also be based on the first-hand eye calibration matrix as the current calibration result, taking any set of correction data, and correcting the first-hand eye calibration matrix based on the correction data to obtain the corrected third-hand eye calibration matrix and the third-hand eye calibration error. It is then determined whether the third-hand eye calibration error is less than the calibration error corresponding to the current calibration result (which is the first-hand eye calibration error at this time). If so, the current calibration result is updated to the third-hand eye calibration matrix, and then a different set of correction data is used, and the correction operation is performed again. If not, a different set of correction data is used, and the correction operation is performed again. This process continues until all correction data has been used in the optimization process of the first-hand eye calibration matrix.

[0058] In this embodiment, by combining the first hand-eye calibration error, each second hand-eye calibration error, and the corresponding second hand-eye calibration matrix into a correction data set, and through continuous correction, verification, and filtering, the first hand-eye calibration matrix selected through cross-validation is further optimized. This maximizes the use of the collected and calculated hand-eye calibration data, thereby improving the accuracy of the hand-eye calibration matrix without adding too much data.

[0059] In one embodiment, such as Figure 4 As shown, the first hand-eye calibration matrix includes a first rotation matrix and a first translation matrix, the second hand-eye calibration matrix includes a second rotation matrix and a second translation matrix, and S420 includes:

[0060] S421, based on the first hand-eye calibration error, the second hand-eye calibration error and the preset bias correction rate, determine the rotation bias weight of the first rotation matrix and the translation bias weight of the first translation matrix.

[0061] S422, determine the rotation correction matrix based on the first rotation matrix, the second rotation matrix, and the rotation bias weights, and determine the translation correction matrix based on the first translation matrix, the second translation matrix, and the translation bias weights.

[0062] S423, determine the third hand-eye calibration matrix based on the rotation correction matrix and translation correction matrix.

[0063] Among them, the bias correction rate This is an additional control parameter, which can be any value from 0 to 1, used to adjust different correction magnitudes during rotation and translation. The rotation bias weight is set to... The translation bias weight is As can be seen from the above embodiments, there are k-1 sets of correction data. The rotation bias weight can be determined based on the second-hand eye calibration error in these k-1 sets of correction data. Translation bias weights Specifically, the second-hand eye calibration error in the nth set of corrected data is denoted as... The first-hand eye calibration error is denoted as Rotational bias weights Translation bias weights It can be determined based on equations (1) and (2) respectively:

[0064] (1)

[0065] (2)

[0066] Furthermore, the first-hand eye calibration matrix can be determined using the xyz order Euler angle calculation method. Decomposed into the first rotation matrix and the first translation matrix Second-hand eye calibration matrix Decomposed into the second rotation matrix Second translation matrix The first rotation matrix can be modified based on equations (3) and (4) respectively. and the first translation matrix :

[0067] (3)

[0068] (4)

[0069] In the above formula To correct the obtained rotation correction matrix, This is the translation correction matrix obtained after correction. Further, the rotation correction matrix is ​​then... Translation Correction Matrix The corrected hand-eye calibration matrix is ​​obtained by merging. Then repeat the above correction steps until each set of corrected data participates in the optimization of the first hand-eye calibration matrix, thus obtaining the target hand-eye calibration matrix.

[0070] In this embodiment, by combining the rotation correction matrix and the translation correction matrix, and by setting different weights for the rotation matrix and the translation matrix, the over-correction situation is reduced. Therefore, compared with the original first hand-eye calibration matrix, the third hand-eye calibration matrix can more accurately describe the spatial position and posture transformation relationship between the corrected hand and eye, providing a more reliable coordinate transformation basis for the robot to perform precise operations based on visual information.

[0071] In one embodiment, each hand-eye calibration dataset includes multiple sets of calibration data. Each set of calibration data includes calibration images captured by a camera located at the end effector of the robotic arm, and the end effector pose of the robotic arm in a robot coordinate system, such as... Figure 5 As shown, S200 includes:

[0072] S210. Based on multiple sets of calibration images in the training dataset from the calibration data division results, as well as multiple sets of robot arm end poses under robot base coordinate systems, perform hand-eye calibration on the robot to obtain the hand-eye calibration matrix.

[0073] Following the above embodiments, each hand-eye calibration dataset includes multiple sets of calibration data. For example, the camera at the end effector of the robotic arm can be controlled to acquire more than 50 valid calibration images, i.e., those calibration images that can detect the position of the calibration point on the calibration plate, and the pose of the robotic arm end effector when the camera acquires the valid calibration images. Then, the valid calibration images are randomly divided into k parts. Considering that hand-eye calibration generally requires more than 10 hand-eye calibration images to calibrate a relatively stable hand-eye calibration matrix, k can be set to 5, thus constructing 5 hand-eye calibration datasets. Each hand-eye calibration dataset includes 10 valid calibration images and the corresponding pose of the robotic arm end effector.

[0074] Furthermore, based on the mathematical model of hand-eye calibration (such as the common "AX = XB" model), a system of equations for hand-eye calibration is constructed using the robot arm end-effector pose in the training dataset and the calibration images captured by the camera in the robot arm end-effector pose. Specifically, when the robot arm end-effector transforms from one pose to another, this pose change can be described by a specific transformation matrix A; simultaneously, the camera relative to the calibration plate also undergoes a corresponding pose change during this process, which can be described by transformation matrix B. The robot's hand-eye relationship (which can be described by the hand-eye calibration matrix X) and the position of the calibration plate in the robot's base coordinate system remain constant throughout the process, thus allowing the construction of hand-eye calibration equations. Then, based on multiple sets of calibration data in the training dataset, the solution to the hand-eye calibration equations, i.e., the hand-eye calibration matrix, can be determined through iterative calculation and optimization.

[0075] Similarly, when the eye is outside the hand, a hand-eye calibration equation can be constructed, and the hand-eye calibration matrix can be solved based on multiple sets of calibration data in the training dataset.

[0076] In this embodiment, multiple sets of calibration data are used to construct a system of equations and solve the hand-eye calibration matrix. This utilizes the principle of multi-data optimization in statistics, which can effectively reduce various random errors such as minor deviations in mechanical movement and camera imaging noise in the actual hand-eye calibration process. This results in a calibration matrix that is closer to the real hand-eye relationship, improving the accuracy and reliability of hand-eye calibration.

[0077] In one embodiment, such as Figure 5 As shown, S200 includes:

[0078] S220, for each set of calibration images in the verification dataset in the calibration data division results, determine the position of the first calibration point in the robot arm coordinate system based on the hand-eye calibration matrix and the robot arm end pose corresponding to the calibration image.

[0079] S230, based on the calibration image, determine the position of the second calibration point in the robot arm coordinate system, and determine the distance error between the positions of the first and second calibration points.

[0080] S240 determines the mean distance error based on the distance error corresponding to multiple sets of calibration data, and uses the mean distance error as the hand-eye calibration error.

[0081] Specifically, the difference between the theoretical calibration point position (first calibration point position) calculated based on the hand-eye calibration matrix and the actual calibration point position obtained from the calibration image (second calibration point position) can be compared. This difference can be quantified by distance error, and then the average distance error of multiple sets of data can be obtained to determine the hand-eye calibration error, thereby measuring the accuracy of hand-eye calibration.

[0082] For example, let the hand-eye calibration matrix obtained by the i-th cross-validation calibration be... For the validation dataset used in this calibration, the calibration points in the j-th calibration image The position in the calibrated coordinate system is At this time, the position of the robotic arm's end effector is Assuming the camera intrinsics have been calibrated using OpenCV, the intrinsic matrix is ​​as follows: The distortion parameter is and hand-eye calibration matrix For ease of understanding, the corrected points are shown below. It can be expressed as shown in equation (5), where For distortion correction function:

[0083] (5)

[0084] For the k-th image in the calibration training dataset, the robotic arm's image-taking pose is as follows: Due to the projection transformation, the hand-eye calibration matrix can be... If the third column component (i.e., the z component) is removed and denoted as the projection transformation matrix A, then the following relationship (6) holds:

[0085] (6)

[0086] (7)

[0087] (8)

[0088] Based on equations (6) to (8), yes The position coordinates in the calibration board coordinate system are then determined by the robot arm pose in the validation dataset. and the hand-eye calibration matrix obtained from the training dataset. ,Will Projected onto the robotic arm's coordinate system, the final coordinates after removing the homogeneous term are: (i.e., the location of the first calibration point), combined with the calibration points in the j-th calibration image. The actual position in the calibrated coordinate system is (i.e., the location of the second calibration point). Since there may be multiple calibration points in the j-th calibration image, the average variance of the distance errors of these calibration points can be used as the distance error of the j-th calibration image, as shown in equation (9):

[0089] (9)

[0090] Where width and height represent the width and height of the calibration image, respectively, and are summed starting at c=0 and r=0, and can include all calibration points in the calibration image. The validation dataset of the i-th cross-validation contains multiple calibration images, so the hand-eye calibration error of the i-th cross-validation can be the average of the above distance errors, as shown in equation (10):

[0091] (10)

[0092] In this embodiment, through the above series of operations, the average variance of the positional errors of all calibration points in the j-th image can be obtained. Then, by calculating the mean, the hand-eye calibration error of the i-th cross-validation can be determined. This allows for a comprehensive and in-depth evaluation of the accuracy of the hand-eye calibration matrix, thus providing a quantifiable analytical basis for subsequent optimization of the hand-eye calibration matrix.

[0093] To provide a clearer explanation of the robot hand-eye calibration method provided in this application, the following is in conjunction with the appendix. Figure 6 and one A specific embodiment will be described, which includes the following steps:

[0094] S601, acquire multiple hand-eye calibration datasets for the robot, select each hand-eye calibration dataset as the verification dataset in turn, and use the remaining hand-eye calibration datasets other than the selected hand-eye calibration datasets as the training datasets to obtain multiple sets of calibration data partitioning results.

[0095] S602, for each set of calibration data division results, based on multiple sets of calibration images in the training dataset in the calibration data division results, and the end-effector poses of the robot arm under multiple robot base coordinate systems, perform hand-eye calibration on the robot to obtain the hand-eye calibration matrix.

[0096] S603, for each set of calibration images in the verification dataset in the calibration data division results, determine the position of the first calibration point in the robot arm coordinate system based on the hand-eye calibration matrix and the robot arm end pose corresponding to the calibration image, and determine the position of the second calibration point in the robot arm coordinate system based on the calibration image.

[0097] S604, determine the distance error between the first calibration point position and the second calibration point position, determine the mean distance error based on the distance error corresponding to multiple sets of calibration data, and determine the mean distance error as the hand-eye calibration error.

[0098] S605: Select the smallest first hand-eye calibration error from the hand-eye calibration errors corresponding to the multiple sets of calibration data division results, and determine the hand-eye calibration matrix corresponding to the first hand-eye calibration error as the robot's first hand-eye calibration matrix.

[0099] S606, based on the first hand-eye calibration error, the second hand-eye calibration error and the preset bias correction rate, determine the rotation bias weight of the first rotation matrix and the translation bias weight of the first translation matrix.

[0100] S607, determine the rotation correction matrix based on the first rotation matrix, the second rotation matrix, and the rotation bias weights, and determine the translation correction matrix based on the first translation matrix, the second translation matrix, and the translation bias weights.

[0101] S608. Based on the rotation correction matrix and translation correction matrix, determine the third hand-eye calibration matrix. Based on the verification dataset corresponding to the first hand-eye calibration matrix, verify the third hand-eye calibration matrix to obtain the third hand-eye calibration error.

[0102] S609: Select the target hand-eye calibration error that is the smallest among multiple third-hand-eye calibration errors and is smaller than the first-hand-eye calibration error, and determine the third-hand-eye calibration matrix corresponding to the target hand-eye calibration error as the target hand-eye calibration matrix.

[0103] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0104] Based on the same inventive concept, this application also provides a robot hand-eye calibration device for implementing the robot hand-eye calibration method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more robot hand-eye calibration device embodiments provided below can be found in the limitations of the robot hand-eye calibration method described above, and will not be repeated here.

[0105] In one embodiment, such as Figure 7 As shown, a robot hand-eye calibration device 700 is provided, including: a data acquisition module 710, a data division module 720, a hand-eye calibration module 730, a first optimization module 740, and a second optimization module 750, wherein:

[0106] The data acquisition module 710 is used to acquire multiple hand-eye calibration datasets for the robot.

[0107] The data partitioning module 720 is used to select each hand-eye calibration dataset as the validation dataset in turn, and use the remaining hand-eye calibration datasets other than the selected hand-eye calibration datasets as the training datasets, so as to obtain multiple sets of calibration data partitioning results.

[0108] The hand-eye calibration module 730 is used to perform hand-eye calibration on the robot based on the training dataset in each set of calibration data division results, to obtain the hand-eye calibration matrix, and to verify the hand-eye calibration matrix based on the verification dataset in the calibration data division results, so as to obtain the hand-eye calibration error corresponding to the calibration data division results.

[0109] The first optimization module 740 is used to select the smallest first hand-eye calibration error from the hand-eye calibration errors corresponding to the division results of multiple sets of calibration data, and to determine the hand-eye calibration matrix corresponding to the first hand-eye calibration error as the first hand-eye calibration matrix of the robot.

[0110] The second optimization module 750 is used to correct the first hand-eye calibration matrix and determine the target hand-eye calibration matrix based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error.

[0111] In one embodiment, the second optimization module 750 is used to take the first hand-eye calibration error, each second hand-eye calibration error, and the second hand-eye calibration matrix corresponding to each second hand-eye calibration error as a set of correction data, correct the first hand-eye calibration matrix through each set of correction data to obtain the corrected third hand-eye calibration matrix, verify the third hand-eye calibration matrix according to the verification dataset corresponding to the first hand-eye calibration matrix to obtain the third hand-eye calibration error, select the target hand-eye calibration error that is the smallest and smaller than the first hand-eye calibration error from multiple third hand-eye calibration errors, and determine the third hand-eye calibration matrix corresponding to the target hand-eye calibration error as the target hand-eye calibration matrix.

[0112] In one embodiment, the first hand-eye calibration matrix includes a first rotation matrix and a first translation matrix, and the second hand-eye calibration matrix includes a second rotation matrix and a second translation matrix. The second optimization module 750 is used to determine the rotation bias weight of the first rotation matrix and the translation bias weight of the first translation matrix based on the first hand-eye calibration error, the second hand-eye calibration error, and a preset bias correction rate; determine the rotation correction matrix based on the first rotation matrix, the second rotation matrix, and the rotation bias weight; determine the translation correction matrix based on the first translation matrix, the second translation matrix, and the translation bias weight; and determine the third hand-eye calibration matrix based on the rotation correction matrix and the translation correction matrix.

[0113] In one embodiment, each hand-eye calibration dataset includes multiple sets of calibration data. Each set of calibration data includes calibration images captured by a camera located at the end of the robotic arm and the end-effector pose of the robotic arm under the robot coordinate system. The hand-eye calibration module 730 is used to perform hand-eye calibration on the robot based on the multiple sets of calibration images and the multiple sets of end-effector poses of the robotic arm under the robot coordinate system in the training dataset of the calibration data division result, and obtain a hand-eye calibration matrix.

[0114] In one embodiment, the hand-eye calibration module 730 is used to determine the position of the first calibration point in the robotic arm coordinate system for each set of calibration images in the verification dataset of the calibration data segmentation results, based on the hand-eye calibration matrix and the robotic arm end pose corresponding to the calibration image, and to determine the position of the second calibration point in the robotic arm coordinate system based on the calibration image, and to determine the distance error between the first calibration point position and the second calibration point position, and to determine the mean distance error based on the distance errors corresponding to multiple sets of calibration data, and to determine the mean distance error as the hand-eye calibration error.

[0115] Each module in the aforementioned robot hand-eye calibration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0116] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as hand-eye calibration datasets. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a robot hand-eye calibration method.

[0117] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0118] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described robot hand-eye calibration method embodiment.

[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps in the above-described robot hand-eye calibration method embodiment.

[0120] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described robot hand-eye calibration method embodiment.

[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0122] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for robot hand-eye calibration, characterized in that, The method includes Acquire multiple hand-eye calibration datasets for robots; Each hand-eye calibration dataset is selected in turn as the validation dataset, and the remaining hand-eye calibration datasets other than the selected ones are used as the training datasets to obtain multiple sets of calibration data partitioning results; For each set of calibration data partitioning results, the robot is calibrated using the training dataset in the calibration data partitioning results to obtain a hand-eye calibration matrix. The hand-eye calibration matrix is ​​then validated using the validation dataset in the calibration data partitioning results to obtain the hand-eye calibration error corresponding to the calibration data partitioning results. The smallest first hand-eye calibration error is selected from the hand-eye calibration errors corresponding to the multiple sets of calibration data division results, and the hand-eye calibration matrix corresponding to the first hand-eye calibration error is determined as the first hand-eye calibration matrix of the robot. Based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error, the first hand-eye calibration matrix is ​​corrected to determine the target hand-eye calibration matrix.

2. The method according to claim 1, characterized in that, The step of correcting the first hand-eye calibration matrix and determining the target hand-eye calibration matrix based on the first hand-eye calibration error, the second hand-eye calibration error excluding the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error includes: The first hand-eye calibration error, each second hand-eye calibration error, and the second hand-eye calibration matrix corresponding to each second hand-eye calibration error are used as a set of correction data; The first hand-eye calibration matrix is ​​corrected by each set of correction data to obtain the corrected third hand-eye calibration matrix. The third hand-eye calibration matrix is ​​then verified based on the verification dataset corresponding to the first hand-eye calibration matrix to obtain the third hand-eye calibration error. The target hand-eye calibration error, which is smaller than the first hand-eye calibration error, is selected from multiple third hand-eye calibration errors. The third hand-eye calibration matrix corresponding to the target hand-eye calibration error is then determined as the target hand-eye calibration matrix.

3. The method according to claim 2, characterized in that, The first hand-eye calibration matrix includes a first rotation matrix and a first translation matrix, and the second hand-eye calibration matrix includes a second rotation matrix and a second translation matrix; The step of correcting the first hand-eye calibration matrix using each set of correction data to obtain the corrected third hand-eye calibration matrix includes: Based on the first hand-eye calibration error, the second hand-eye calibration error, and the preset bias correction rate, the rotation bias weight of the first rotation matrix and the translation bias weight of the first translation matrix are determined. Based on the first rotation matrix, the second rotation matrix, and the rotation bias weights, a rotation correction matrix is ​​determined; The translation correction matrix is ​​determined based on the first translation matrix, the second translation matrix, and the translation bias weight; The third hand-eye calibration matrix is ​​determined based on the rotation correction matrix and the translation correction matrix.

4. The method according to any one of claims 1 to 3, characterized in that, Each of the hand-eye calibration datasets includes multiple sets of calibration data. Each set of calibration data includes calibration images captured by a camera located at the end of the robotic arm, as well as the end-effector pose of the robotic arm under the robot base coordinate system. The step of performing hand-eye calibration on the robot based on the training dataset from the calibration data partitioning results to obtain a hand-eye calibration matrix includes: Based on multiple sets of calibration images and multiple sets of robot arm end poses under robot coordinate systems in the training dataset of the calibration data division results, hand-eye calibration of the robot is performed to obtain the hand-eye calibration matrix.

5. The method according to claim 4, characterized in that, The step of verifying the hand-eye calibration matrix based on the verification dataset in the calibration data partitioning results to obtain the hand-eye calibration error corresponding to the calibration data partitioning results includes: For each set of calibration images in the verification dataset of the calibration data division results, the first calibration point position in the robot arm coordinate system is determined according to the hand-eye calibration matrix and the robot arm end pose corresponding to the calibration image. The second calibration point position in the robot arm coordinate system is determined according to the calibration image. The distance error between the first calibration point position and the second calibration point position is determined. Based on the distance errors corresponding to multiple sets of calibration data, the mean distance error is determined, and the mean distance error is determined as the hand-eye calibration error.

6. A robot hand-eye calibration device, characterized in that, The device includes: The data acquisition module is used to acquire multiple hand-eye calibration datasets for the robot; The data partitioning module is used to select each hand-eye calibration dataset as the validation dataset in turn, and use the remaining hand-eye calibration datasets other than the selected hand-eye calibration datasets as the training datasets, so as to obtain multiple sets of calibration data partitioning results; The hand-eye calibration module is used to perform hand-eye calibration on the robot based on the training dataset in each set of calibration data division results, to obtain a hand-eye calibration matrix, and to verify the hand-eye calibration matrix based on the verification dataset in the calibration data division results, to obtain the hand-eye calibration error corresponding to the calibration data division results. The first optimization module is used to select the smallest first hand-eye calibration error from the hand-eye calibration errors corresponding to the division results of multiple sets of calibration data, and to determine the hand-eye calibration matrix corresponding to the first hand-eye calibration error as the first hand-eye calibration matrix of the robot. The second optimization module is used to correct the first hand-eye calibration matrix and determine the target hand-eye calibration matrix based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error.

7. The apparatus according to claim 6, characterized in that, Each hand-eye calibration dataset includes multiple sets of calibration data. Each set of calibration data includes calibration images captured by a camera located at the end of the robotic arm and the end-effector pose of the robotic arm under a robot coordinate system. The hand-eye calibration module is used to perform hand-eye calibration on the robot based on multiple sets of calibration images and multiple sets of end-effector poses of the robotic arm under a robot coordinate system in the training dataset of the calibration data division results, and obtain a hand-eye calibration matrix.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images