Robot hand-eye calibration method and device, computer equipment and storage medium

Abstract

The invention relates to a robot hand-eye calibration method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a plurality of hand-eye calibration data sets, selecting each hand-eye calibration data set as a verification data set in turn, taking the remaining hand-eye calibration data set as a training data set to obtain a plurality of groups of calibration data division results, and performing hand-eye calibration on the robot according to the training data set for each group of calibration data division result. Verifying the hand-eye calibration matrix according to the verification data set to obtain hand-eye calibration errors, screening out the minimum first hand-eye calibration error, and taking the corresponding hand-eye calibration matrix as the first hand-eye calibration matrix; and based on the first hand-eye calibration error, a second hand-eye calibration error except the first hand-eye calibration error, and a second hand-eye calibration matrix, correcting the first hand-eye calibration matrix, and determining a target hand-eye calibration matrix. By adopting the method, the hand-eye calibration precision can be improved.

Description

Technical Field

[0001] The present application relates to the field of robot calibration technology, and in particular to a robot hand-eye calibration method, device, computer equipment, storage medium and computer program product. Background Art

[0002] With the rapid development of industrial automation and intelligent manufacturing, robots are increasingly used in various production and operation tasks. In order to enable robots to perform precise movements based on visual information, hand-eye calibration is required before using the robot.

[0003] In traditional solutions, a flat calibration plate, such as a black and white checkerboard, is usually used to control the camera at the end of the robotic arm to collect images of the calibration plate under different postures of the robotic arm, and the robot's hand-eye calibration formula is used to calibrate the hand and eye of the robot.

[0004] However, traditional hand-eye calibration schemes require collecting different postures of the end of the robotic arm relative to the robot's base coordinate system. In most cases, the end of the robotic arm needs to be installed with tools, such as grippers. Whether it is manually collected or calibrated by algorithm, the posture of the end of the robotic arm obtained will have large errors, resulting in low hand-eye calibration accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide a robot hand-eye calibration method, device, computer equipment, computer-readable storage medium and computer program product that can improve the hand-eye calibration accuracy in response to the above technical problems.

[0006] In a first aspect, the present application provides a robot hand-eye calibration method. The method comprises:

[0007] Obtain multiple hand-eye calibration datasets for robots;

[0008] Each hand-eye calibration data set is selected as a verification data set in turn, and the remaining hand-eye calibration data sets other than the selected hand-eye calibration data set are used as training data sets to obtain multiple groups of calibration data division results;

[0009] For each group of calibration data division results, the robot is calibrated for hand-eye calibration according to the training data set in the calibration data division results to obtain a hand-eye calibration matrix, and the hand-eye calibration matrix is ​​verified according to the verification data set in the calibration data division results to obtain a hand-eye calibration error corresponding to the calibration data division results;

[0010] Selecting a minimum first hand-eye calibration error from the hand-eye calibration errors corresponding to the division results of the multiple groups of calibration data, and determining a hand-eye calibration matrix corresponding to the first hand-eye calibration error as a first hand-eye calibration matrix of the robot;

[0011] Based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error, the first hand-eye calibration matrix is ​​corrected to determine a target hand-eye calibration matrix.

[0012] In a second aspect, the present application also provides a robot hand-eye calibration device. The device comprises:

[0013] A data acquisition module, used to acquire multiple hand-eye calibration data sets for the robot;

[0014] A data partitioning module is used to select each hand-eye calibration data set as a verification data set in turn, and use the remaining hand-eye calibration data sets other than the selected hand-eye calibration data set as a training data set to obtain multiple groups of calibration data partitioning results;

[0015] A hand-eye calibration module is used to perform hand-eye calibration on the robot for each group of calibration data division results according to a training data set in the calibration data division results to obtain a hand-eye calibration matrix, and to verify the hand-eye calibration matrix according to a verification data set in the calibration data division results to obtain a hand-eye calibration error corresponding to the calibration data division results;

[0016] A first optimization module is used to select a minimum first hand-eye calibration error from the hand-eye calibration errors corresponding to the division results of multiple groups of calibration data, and determine the hand-eye calibration matrix corresponding to the first hand-eye calibration error as the first hand-eye calibration matrix of the robot;

[0017] The second optimization module is used to correct the first hand-eye calibration matrix and determine the target hand-eye calibration matrix based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error.

[0018] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above robot hand-eye calibration method embodiment when executing the computer program.

[0019] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the above-mentioned robot hand-eye calibration method embodiment are implemented.

[0020] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned robot hand-eye calibration method embodiment are implemented.

[0021] The above-mentioned robot hand-eye calibration method, device, computer equipment, storage medium and computer program product first obtain multiple hand-eye calibration data sets for the robot, and then take turns to use these hand-eye calibration data sets as verification data sets, and the rest as training data sets to form different data division results. Further, for each set of data division results, the training data set is used to calibrate the robot's hand and eye to obtain a hand-eye calibration matrix, and then the hand-eye calibration matrix is ​​verified through the verification data set to calculate the hand-eye calibration error. Finally, the smallest first hand-eye calibration error is selected from the multiple hand-eye calibration errors, and the corresponding hand-eye calibration matrix is ​​determined as the first hand-eye calibration matrix of the robot, and according to the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error, the first hand-eye calibration matrix is ​​further corrected to determine the target hand-eye calibration matrix. Different from the single hand-eye calibration in the traditional solution, this solution can make full use of existing data resources through double verification and cross-validation, evaluate and analyze the robot hand-eye calibration from multiple angles and different combinations, reduce the accidental errors that may occur in a single calibration, and thus improve the accuracy of the robot hand-eye calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A diagram showing an application environment of a robot hand-eye calibration method in one embodiment;

[0023] Figure 2 Schematic diagram of a flow chart of a robot hand-eye calibration method in one embodiment;

[0024] Figure 3 A schematic diagram of a process for optimizing a first hand-eye calibration matrix in one embodiment;

[0025] Figure 4 A schematic diagram of a flow chart of optimizing a first hand-eye calibration matrix in another embodiment;

[0026] Figure 5 A schematic diagram of a flow chart of a robot hand-eye calibration method in another embodiment;

[0027] Figure 6 A schematic diagram of a flow chart of a robot hand-eye calibration method in a detailed embodiment;

[0028] Figure 7 is a structural block diagram of a robot hand-eye calibration device in one embodiment;

[0029] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] The robot hand-eye calibration method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0032] Specifically, a staff member may upload multiple hand-eye calibration data sets for the robot to the server 104 through the terminal 102. Then, the server 104 selects each hand-eye calibration data set as a verification data set in turn, and uses the remaining hand-eye calibration data sets other than the selected hand-eye calibration data set as a training data set to obtain multiple groups of calibration data division results, and for each group of calibration data division results, performs hand-eye calibration on the robot according to the training data set in the calibration data division results to obtain a hand-eye calibration matrix, and verifies the hand-eye calibration matrix according to the verification data set in the calibration data division results to obtain the hand-eye calibration error corresponding to the calibration data division results. Further, the server 104 selects the smallest first hand-eye calibration error from the hand-eye calibration errors corresponding to the multiple groups of calibration data division results, and determines the hand-eye calibration matrix corresponding to the first hand-eye calibration error as the first hand-eye calibration matrix of the robot. Finally, server 104 corrects the first hand-eye calibration matrix based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error to determine the target hand-eye calibration matrix.

[0033] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0034] In one embodiment, Figure 2 As shown, a robot hand-eye calibration method is provided, which is applied to Figure 1Taking the server 104 in the example as an example, the following steps are included:

[0035] S100, obtaining multiple hand-eye calibration data sets for the robot, selecting each hand-eye calibration data set as a verification data set in turn, and using the remaining hand-eye calibration data sets other than the selected hand-eye calibration data set as a training data set to obtain multiple groups of calibration data division results.

[0036] Among them, hand-eye calibration is the process of determining the spatial position and posture conversion relationship between the robot's end arm (hand) and the camera (eye). In the robot system, the information obtained by the camera is based on the camera coordinate system, and the actuator action at the end of the robot arm is based on the end effector coordinate system. In order to enable the robot to perform accurate actions based on visual information, it is necessary to first determine the conversion relationship between the camera coordinate system and the end effector coordinate system, that is, to calibrate the robot's hand-eye. The hand-eye calibration dataset is a data set used for hand-eye calibration, which contains the pose data of the end effector of the robot arm (hereinafter referred to as the end effector pose) and the data obtained by the camera. The end effector pose is the position and pose information of the end effector of the robot's robot arm in the robot base coordinate system. The data obtained by the camera can be the calibration image of the calibration plate or other objects taken by the camera, the pixel coordinates of the calibration point in the camera coordinate system, and other information. It should be noted that whether the eye is on the hand or outside the hand, this scheme can be used for hand-eye calibration.

[0037] Specifically, after the server obtains the hand-eye calibration dataset for the robot, it selects each hand-eye calibration dataset as a verification dataset in turn according to the idea of ​​cross-validation. For example, assuming there are 5 hand-eye calibration datasets, numbered 1 to 5, the first time the hand-eye calibration dataset No. 1 is used as the verification dataset, and the remaining hand-eye calibration datasets No. 2 to 5 become the training datasets. The second time, the hand-eye calibration dataset No. 2 is used as the verification dataset, and No. 1, 3, 4, and 5 use the hand-eye calibration dataset No. 1 as the training dataset, and so on, until each hand-eye calibration dataset has the opportunity to be selected as the verification dataset. In this way, multiple groups of different calibration data division results will be obtained, and each group of calibration data division results corresponds to a specific combination of verification datasets and training datasets. Subsequently, corresponding calibration and verification operations can be performed based on different calibration data division results.

[0038] S200, for each group of calibration data division results, perform hand-eye calibration on the robot according to the training data set in the calibration data division results to obtain a hand-eye calibration matrix, verify the hand-eye calibration matrix according to the verification data set in the calibration data division results to obtain a hand-eye calibration error corresponding to the calibration data division results.

[0039] Among them, the hand-eye calibration matrix is ​​the result of the robot's hand-eye calibration, which characterizes the transformation relationship between the camera coordinate system and the robot end coordinate system or the end effector coordinate system. Following the above steps, for each set of calibration data division results, a training data set can be extracted from each set of calibration data division results. As mentioned above, the training data set contains the pose information of the robot's robot end effector in the robot base coordinate system, as well as the calibration image of the calibration plate or other objects taken by the camera, the pixel coordinates of the calibration point in the camera coordinate system and other information.

[0040] Furthermore, according to a preset hand-eye calibration mathematical model (e.g. The hand-eye calibration matrix can be constructed by constructing a set of hand-eye calibration equations using the end-arm postures and calibration images captured by the camera in the training data set. Then, the hand-eye calibration matrix can be solved through multiple sets of data points (different end-arm postures and calibration images captured by the camera at the corresponding postures).

[0041] Then, a verification data set is taken out from the same set of calibration data division results, which also contains the end posture of the robot arm and the calibration images captured by the camera. The hand-eye calibration matrix obtained based on the training data set and the end posture of the robot arm in the verification data set are used. According to the principle of hand-eye coordinate transformation, the calibration image that the camera should theoretically capture under the current calibration result is calculated. The calibration image that should be captured theoretically is compared with the calibration image actually captured in the verification data set. The difference between the two is the hand-eye calibration error.

[0042] S300, selecting the smallest first hand-eye calibration error from the hand-eye calibration errors corresponding to the division results of multiple groups of calibration data, and determining the hand-eye calibration matrix corresponding to the first hand-eye calibration error as the first hand-eye calibration matrix of the robot.

[0043] Through the above steps, hand-eye calibration and verification are performed on multiple groups of calibration data division results respectively. After obtaining the corresponding hand-eye calibration errors, the smallest error (i.e., the first hand-eye calibration error) is screened out. In this way, the calibration result (i.e., the first hand-eye calibration matrix) that most accurately reflects the real spatial position and posture conversion relationship between the robot end effector and the camera under many different data combinations and calibration methods can be found.

[0044] In the hand-eye calibration error sets corresponding to the multiple groups of calibration data division results that have been obtained, compare each hand-eye calibration error. For example, assuming there are 5 groups of calibration data division results, the corresponding hand-eye calibration errors are error 1, error 2, error 3, error 4 and error 5, and compare the values ​​of these errors one by one to determine the error with the smallest value, which is defined as the first hand-eye calibration error, and then traced back to the first hand-eye calibration matrix corresponding to the first hand-eye calibration error.

[0045] S400, based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error, correct the first hand-eye calibration matrix to determine the target hand-eye calibration matrix.

[0046] Following the above steps, after selecting the first hand-eye calibration matrix with the smallest hand-eye calibration error, it can be further optimized. For example, introducing a second hand-eye calibration error in addition to the first hand-eye calibration error and its corresponding second hand-eye calibration matrix to correct the first hand-eye calibration matrix can integrate more calibration information, further optimize the calibration results, and finally obtain a more accurate and reliable target hand-eye calibration matrix.

[0047] The above robot hand-eye calibration method first obtains multiple hand-eye calibration data sets for the robot, and then takes turns to use these hand-eye calibration data sets as verification data sets, and the rest are used as training data sets to form different data division results. Further, for each set of data division results, the training data set is used to calibrate the robot's hand-eye to obtain a hand-eye calibration matrix, and then the hand-eye calibration matrix is ​​verified through the verification data set to calculate the hand-eye calibration error. Finally, the smallest first hand-eye calibration error is selected from the multiple hand-eye calibration errors, and the corresponding hand-eye calibration matrix is ​​determined as the first hand-eye calibration matrix of the robot, and according to the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error, the first hand-eye calibration matrix is ​​further corrected to determine the target hand-eye calibration matrix. Different from the single hand-eye calibration in the traditional solution, this solution can make full use of existing data resources through double verification and cross-validation, evaluate and analyze the robot hand-eye calibration from multiple angles and different combinations, reduce the accidental errors that may occur in a single calibration, and thus improve the accuracy of the robot hand-eye calibration.

[0048] In one embodiment, Figure 3 As shown, S400 includes:

[0049] S410, taking the first hand-eye calibration error, each second hand-eye calibration error, and the second hand-eye calibration matrix corresponding to each second hand-eye calibration error as a set of correction data.

[0050] S420, correcting the first hand-eye calibration matrix using each set of correction data to obtain a corrected third hand-eye calibration matrix.

[0051] S430, verifying the third hand-eye calibration matrix according to the verification data set corresponding to the first hand-eye calibration matrix to obtain a third hand-eye calibration error.

[0052] S440, selecting a target hand-eye calibration error that is the smallest and smaller than the first hand-eye calibration error from multiple third hand-eye calibration errors, and determining a third hand-eye calibration matrix corresponding to the target hand-eye calibration error as the target hand-eye calibration matrix.

[0053] Following the above embodiment, the first hand-eye calibration matrix initially selected is further corrected by integrating multiple groups of relevant data, so as to determine a matrix with the smallest error and better than the first hand-eye calibration error as the target hand-eye calibration matrix.

[0054] Specifically, each calibration data division result corresponds to a hand-eye calibration error, and except for the smallest first hand-eye calibration error, the remaining hand-eye calibration errors are marked as second hand-eye calibration errors. The first hand-eye calibration error, each second hand-eye calibration error, and the second hand-eye calibration matrix corresponding to each second hand-eye calibration error are used as a set of correction data, thus forming multiple sets of different correction data.

[0055] For each set of correction data, the first hand-eye calibration matrix must be corrected based on it, so as to obtain a corrected third hand-eye calibration matrix. Specifically, the error of the first hand-eye calibration matrix can be adjusted based on a preset bias correction rate. In addition, different weights can be set for translation errors and rotation errors according to the actual calibration situation to take different correction amplitudes accordingly. After obtaining each corrected third hand-eye calibration matrix, the third hand-eye calibration matrix can be verified using the verification data set corresponding to the first hand-eye calibration matrix. Similar to the above cross-validation, the end position of the manipulator in the verification data set can be substituted into the third hand-eye calibration matrix, and the theoretical calibration image is calculated according to the hand-eye coordinate conversion principle, and then compared with the actual calibration image in the verification data set. The error between the two is the third hand-eye calibration error.

[0056] Then, a target hand-eye calibration error that is both the smallest and smaller than the first hand-eye calibration error is selected from among the multiple third hand-eye calibration errors. This means that among the many calibration results that have been corrected and verified, the calibration result with the best error performance and further optimized compared to the initial first hand-eye calibration matrix is ​​selected. Furthermore, after the target hand-eye calibration error is selected, the corresponding third hand-eye calibration matrix can be traced back and determined as the target hand-eye calibration matrix.

[0057] It should be noted that the above-mentioned optimization process can also be based on the first hand-eye calibration matrix as the current calibration result, and a set of correction data is randomly selected, and the first hand-eye calibration matrix is ​​corrected based on the correction data to obtain a corrected third hand-eye calibration matrix and a third hand-eye calibration error, and determine whether the third hand-eye calibration error is smaller than the calibration error corresponding to the current calibration result (the first hand-eye calibration error at this time). If so, the current calibration result is updated to the third hand-eye calibration matrix, and then a set of correction data is replaced, and the correction operation is performed again; if so, a set of correction data is directly replaced, and the correction operation is performed again; until all the correction data have participated in the optimization process of the first hand-eye calibration matrix.

[0058] In this embodiment, the first hand-eye calibration error, each second hand-eye calibration error and the corresponding second hand-eye calibration matrix are combined into a correction data group, and the first hand-eye calibration matrix screened by cross-validation is further optimized through continuous correction, verification and screening, so as to maximize the use of the hand-eye calibration data that has been collected and calculated, thereby improving the accuracy of the hand-eye calibration matrix without increasing too much data.

[0059] In one embodiment, Figure 4 As shown, the first hand-eye calibration matrix includes a first rotation matrix and a first translation matrix, the second hand-eye calibration matrix includes a second rotation matrix and a second translation matrix, and S420 includes:

[0060] S421, determining the rotation bias weight of the first rotation matrix and the translation bias weight of the first translation matrix based on the first hand-eye calibration error, the second hand-eye calibration error and a preset bias correction rate.

[0061] S422, determining a rotation correction matrix according to the first rotation matrix, the second rotation matrix, and the rotation offset weight, and determining a translation correction matrix according to the first translation matrix, the second translation matrix, and the translation offset weight.

[0062] S423, determining a third hand-eye calibration matrix according to the rotation correction matrix and the translation correction matrix.

[0063] Among them, the bias correction rate is an additional control parameter, which can be any value from 0 to 1, used to adjust the different correction amplitudes in the rotation and translation process, setting the rotation bias weight to , the translation bias weight is According to the above embodiment, there are k-1 sets of correction data. The rotation bias weight can be determined based on the second hand-eye calibration error in the k-1 sets of correction data. and translation bias weights Specifically, the second hand-eye calibration error in the nth set of correction data is recorded as , the first hand-eye calibration error is recorded as , rotation bias weight and translation bias weights It can be determined based on formula (1) and formula (2) respectively:

[0064] (1)

[0065] (2)

[0066] Furthermore, the first hand-eye calibration matrix can be obtained by using the Euler angle solution in xyz order. Decomposed into the first rotation matrix and the first translation matrix , the second hand-eye calibration matrix Decomposed into the second rotation matrix and the second translation matrix , the first rotation matrix can be modified based on equations (3) and (4) respectively and the first translation matrix :

[0067] (3)

[0068] (4)

[0069] In the above formula is the corrected rotation correction matrix, is the corrected translation correction matrix. Further, the rotation correction matrix Translation Correction Matrix Merge to get the corrected hand-eye calibration matrix Then repeat the above correction steps until each set of correction data participates in the optimization of the first hand-eye calibration matrix to obtain the target hand-eye calibration matrix.

[0070] In this embodiment, by combining the rotation correction matrix and the translation correction matrix, and setting different weights for the rotation matrix and the translation matrix, the over-correction situation is reduced. Therefore, compared with the original first hand-eye calibration matrix, the third hand-eye calibration matrix can more accurately describe the spatial position and posture conversion relationship between the corrected hand and eye, providing a more reliable coordinate transformation basis for the subsequent robot to perform precise operations based on visual information.

[0071] In one embodiment, each hand-eye calibration data set includes multiple sets of calibration data, each set of calibration data includes a calibration image captured by a camera at the end of the robotic arm, and the position and posture of the robotic arm end in the robot base coordinate system, such as Figure 5 As shown, S200 includes:

[0072] S210, performing hand-eye calibration on the robot according to multiple groups of calibration images in the training data set in the calibration data division result and multiple groups of robot arm end poses in the robot base coordinate system to obtain a hand-eye calibration matrix.

[0073] Following the above embodiment, each hand-eye calibration data set includes multiple sets of calibration data. For example, the camera at the end of the robotic arm can be controlled to collect more than about 50 valid calibration images, that is, those calibration images that can detect the position of the calibration point on the calibration plate, and record the posture of the end of the robotic arm when the camera collects the valid calibration image. Then, the valid calibration images are randomly divided into k parts. Considering that more than 10 hand-eye calibration images are required for general hand-eye calibration to calibrate a relatively stable hand-eye calibration matrix, k can be taken as 5, and 5 hand-eye calibration data sets are constructed. Each hand-eye calibration data set includes 10 valid calibration images and the posture of the end of the robotic arm corresponding to the valid calibration images.

[0074] Furthermore, according to the mathematical model of hand-eye calibration (such as the common "AX = XB" model, etc.), the end-arm posture in the training data set and the calibration image collected by the camera under the end-arm posture are used to construct the equation group of hand-eye calibration. Specifically, when the end-arm changes from one posture to another, the change of this posture can be described by a specific transformation matrix A; at the same time, the camera will also have a corresponding posture change relative to the calibration plate in this process, which can be described by the transformation matrix B, while the robot's hand-eye relationship (which can be described by the hand-eye calibration matrix X) and the position of the calibration plate in the robot's base coordinate system are fixed throughout the process, so the hand-eye calibration equation can be constructed. Then, based on multiple sets of calibration data in the training data set, the solution to the hand-eye calibration equation, that is, the hand-eye calibration matrix, can be determined by iterative calculation and optimization.

[0075] Similarly, the hand-eye calibration equation can also be constructed for the case where the eye is outside the hand, and the hand-eye calibration matrix can be solved based on multiple sets of calibration data in the training data set.

[0076] In this embodiment, multiple sets of calibration data are used to construct a system of equations and solve the hand-eye calibration matrix. The principle of multi-data optimization in statistics is applied, which can effectively reduce various accidental errors such as slight deviations in mechanical movement and camera imaging noise in the actual hand-eye calibration process, and obtain a calibration matrix that is closer to the actual hand-eye relationship, thereby improving the accuracy and reliability of hand-eye calibration.

[0077] In one embodiment, Figure 5 As shown, S200 includes:

[0078] S220, for each group of calibration images in the verification data set in the calibration data division result, determine the first calibration point position of the calibration point in the robotic arm coordinate system according to the hand-eye calibration matrix and the robotic arm end posture corresponding to the calibration image.

[0079] S230, determining the position of a second calibration point of the calibration point in the robot arm coordinate system according to the calibration image, and determining a distance error between the first calibration point position and the second calibration point position.

[0080] S240, determining a distance error mean based on distance errors corresponding to the multiple sets of calibration data, and determining the distance error mean as a hand-eye calibration error.

[0081] Specifically, the difference between the theoretical calibration point position (first calibration point position) calculated according to the hand-eye calibration matrix and the actual calibration point position (second calibration point position) obtained from the calibration image can be compared, and the difference can be quantified by distance error. Then, the distance errors of multiple sets of data are combined to obtain the mean distance error, and the hand-eye calibration error is determined to measure the accuracy of hand-eye calibration.

[0082] For example, let the hand-eye calibration matrix obtained by the i-th cross-validation calibration be , for the calibration validation data set of this calibration, the calibration point in the jth calibration image The position in the calibration coordinate system is , at this time the end position of the robot arm is , assuming that the camera intrinsic parameters have been calibrated using OpenCV, the intrinsic parameter matrix is , the distortion parameter is , and the hand-eye calibration matrix For ease of understanding, the corrected points It can be expressed as shown in formula (5), where is the distortion correction function:

[0083] (5)

[0084] For the kth image in the calibration training data set, the robotic arm’s photo-taking posture is , due to the relationship between the projection transformation, the hand-eye calibration matrix can be Remove the third column component (i.e., z component) in , recorded as the projection transformation matrix A, and then we have the following relationship (6):

[0085] (6)

[0086] (7)

[0087] (8)

[0088] Based on formula (6)-formula (8), yes The position coordinates in the calibration board coordinate system are then obtained from the robot arm posture in the verification data set. , and the hand-eye calibration matrix obtained based on the training data set ,Will Projected to the robot arm coordinates, the coordinates obtained after removing the homogeneous terms are (i.e. the position of the first calibration point), combined with the calibration point in the jth calibration image The actual position in the calibration coordinate system is (i.e., the position of the second calibration point). Since there may be multiple calibration points in the j-th calibration image, the average variance of the distance errors of these calibration points can be used as the distance error of the j-th calibration image, as shown in formula (9):

[0089] (9)

[0090] Where width and height represent the width and height of the calibration image, respectively. The sum is calculated starting from c=0 and r=0, and can include all calibration points in the calibration image. There are multiple calibration images in the validation data set of the i-th cross-validation. Therefore, the hand-eye calibration error of the i-th cross-validation can be taken as the mean of the above distance errors, as shown in formula (10):

[0091] (10)

[0092] In this embodiment, through the above series of operations, the average variance of the position errors of all calibration points in the jth image can be obtained, and then the hand-eye calibration error of the i-th cross-validation can be determined by taking the average value, which can comprehensively and deeply evaluate the accuracy of the hand-eye calibration matrix, thereby providing a quantifiable analysis basis for the subsequent optimization of the hand-eye calibration matrix.

[0093] In order to make a clearer explanation of the robot hand-eye calibration method provided in this application, the following is combined with the attached Figure 6 and one A specific embodiment is described below, and the specific embodiment includes the following steps:

[0094] S601, obtain multiple hand-eye calibration data sets for the robot, select each hand-eye calibration data set as a verification data set in turn, and use the remaining hand-eye calibration data sets other than the selected hand-eye calibration data set as a training data set to obtain multiple groups of calibration data division results.

[0095] S602, for each group of calibration data division results, the robot is calibrated with hand and eye according to multiple groups of calibration images in the training data set in the calibration data division results, and multiple groups of robot arm end postures in the robot base coordinate system to obtain a hand and eye calibration matrix.

[0096] S603, for each group of calibration images in the verification data set in the calibration data division result, determine the first calibration point position of the calibration point in the robotic arm coordinate system according to the hand-eye calibration matrix and the robotic arm end posture corresponding to the calibration image, and determine the second calibration point position of the calibration point in the robotic arm coordinate system according to the calibration image.

[0097] S604, determining the distance error between the first calibration point position and the second calibration point position, determining the mean of the distance errors based on the distance errors corresponding to the multiple sets of calibration data, and determining the mean of the distance errors as the hand-eye calibration error.

[0098] S605, selecting the smallest first hand-eye calibration error from the hand-eye calibration errors corresponding to the division results of the multiple groups of calibration data, and determining the hand-eye calibration matrix corresponding to the first hand-eye calibration error as the first hand-eye calibration matrix of the robot.

[0099] S606, determining a rotation bias weight of the first rotation matrix and a translation bias weight of the first translation matrix based on the first hand-eye calibration error, the second hand-eye calibration error and a preset bias correction rate.

[0100] S607, determining a rotation correction matrix according to the first rotation matrix, the second rotation matrix, and the rotation offset weight, and determining a translation correction matrix according to the first translation matrix, the second translation matrix, and the translation offset weight.

[0101] S608, determining a third hand-eye calibration matrix according to the rotation correction matrix and the translation correction matrix, verifying the third hand-eye calibration matrix according to a verification data set corresponding to the first hand-eye calibration matrix, and obtaining a third hand-eye calibration error.

[0102] S609, selecting a target hand-eye calibration error that is the smallest and smaller than the first hand-eye calibration error from multiple third hand-eye calibration errors, and determining a third hand-eye calibration matrix corresponding to the target hand-eye calibration error as the target hand-eye calibration matrix.

[0103] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0104] Based on the same inventive concept, the embodiment of the present application also provides a robot hand-eye calibration device for implementing the robot hand-eye calibration method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more robot hand-eye calibration device embodiments provided below can refer to the limitations of the robot hand-eye calibration method above, and will not be repeated here.

[0105] In one embodiment, Figure 7 As shown, a robot hand-eye calibration device 700 is provided, comprising: a data acquisition module 710, a data division module 720, a hand-eye calibration module 730, a first optimization module 740 and a second optimization module 750, wherein:

[0106] The data acquisition module 710 is used to acquire multiple hand-eye calibration data sets for the robot.

[0107] The data partitioning module 720 is used to select each hand-eye calibration data set as a verification data set in turn, and use the remaining hand-eye calibration data sets other than the selected hand-eye calibration data set as a training data set to obtain multiple groups of calibration data partitioning results.

[0108] The hand-eye calibration module 730 is used to perform hand-eye calibration on the robot for each group of calibration data division results according to the training data set in the calibration data division results to obtain a hand-eye calibration matrix, and to verify the hand-eye calibration matrix according to the verification data set in the calibration data division results to obtain a hand-eye calibration error corresponding to the calibration data division results.

[0109] The first optimization module 740 is used to select the smallest first hand-eye calibration error from the hand-eye calibration errors corresponding to the division results of multiple groups of calibration data, and determine the hand-eye calibration matrix corresponding to the first hand-eye calibration error as the first hand-eye calibration matrix of the robot.

[0110] The second optimization module 750 is used to correct the first hand-eye calibration matrix based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error, and determine the target hand-eye calibration matrix.

[0111] In one embodiment, the second optimization module 750 is used to use the first hand-eye calibration error, each second hand-eye calibration error, and the second hand-eye calibration matrix corresponding to each second hand-eye calibration error as a set of correction data, correct the first hand-eye calibration matrix through each set of correction data to obtain a corrected third hand-eye calibration matrix, verify the third hand-eye calibration matrix according to the verification data set corresponding to the first hand-eye calibration matrix to obtain the third hand-eye calibration error, select the smallest target hand-eye calibration error that is smaller than the first hand-eye calibration error from multiple third hand-eye calibration errors, and determine the third hand-eye calibration matrix corresponding to the target hand-eye calibration error as the target hand-eye calibration matrix.

[0112] In one embodiment, the first hand-eye calibration matrix includes a first rotation matrix and a first translation matrix, the second hand-eye calibration matrix includes a second rotation matrix and a second translation matrix, the second optimization module 750 is used to determine the rotation bias weight of the first rotation matrix and the translation bias weight of the first translation matrix based on the first hand-eye calibration error, the second hand-eye calibration error and a preset bias correction rate, determine the rotation correction matrix according to the first rotation matrix, the second rotation matrix, and the rotation bias weight, determine the translation correction matrix according to the first translation matrix, the second translation matrix, and the translation bias weight, and determine the third hand-eye calibration matrix according to the rotation correction matrix and the translation correction matrix.

[0113] In one embodiment, each hand-eye calibration data set includes multiple sets of calibration data, each set of calibration data includes a calibration image captured by a camera located at the end of the robotic arm, and the posture of the robotic arm end in the robot base coordinate system. The hand-eye calibration module 730 is used to perform hand-eye calibration on the robot according to the multiple sets of calibration images in the training data set in the calibration data division result, and the multiple sets of postures of the robotic arm end in the robot base coordinate system to obtain a hand-eye calibration matrix.

[0114] In one embodiment, the hand-eye calibration module 730 is used to determine, for each group of calibration images in the verification data set in the calibration data division result, a first calibration point position of the calibration point in the robotic arm coordinate system according to the hand-eye calibration matrix and the robotic arm end posture corresponding to the calibration image, determine a second calibration point position of the calibration point in the robotic arm coordinate system according to the calibration image, determine the distance error between the first calibration point position and the second calibration point position, determine a mean distance error based on the distance errors corresponding to multiple groups of calibration data, and determine the mean distance error as the hand-eye calibration error.

[0115] Each module in the above robot hand-eye calibration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0116] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as hand-eye calibration data sets. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a robot hand-eye calibration method is implemented.

[0117] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps in the above-mentioned robot hand-eye calibration method embodiment are implemented.

[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned robot hand-eye calibration method embodiment are implemented.

[0120] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps in the above-mentioned robot hand-eye calibration method embodiment.

[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0122] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0123] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A robot hand-eye calibration method, characterized in that: The method comprises Obtain multiple hand-eye calibration datasets for robots; Each hand-eye calibration data set is selected as a verification data set in turn, and the remaining hand-eye calibration data sets other than the selected hand-eye calibration data set are used as training data sets to obtain multiple groups of calibration data division results; For each group of calibration data division results, the robot is calibrated for hand-eye calibration according to the training data set in the calibration data division results to obtain a hand-eye calibration matrix, and the hand-eye calibration matrix is ​​verified according to the verification data set in the calibration data division results to obtain a hand-eye calibration error corresponding to the calibration data division results; Selecting a minimum first hand-eye calibration error from the hand-eye calibration errors corresponding to the division results of the multiple groups of calibration data, and determining a hand-eye calibration matrix corresponding to the first hand-eye calibration error as a first hand-eye calibration matrix of the robot; Based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error, the first hand-eye calibration matrix is ​​corrected to determine a target hand-eye calibration matrix.

2. The method according to claim 1, characterized in that The method of correcting the first hand-eye calibration matrix based on the first hand-eye calibration error, a second hand-eye calibration error other than the first hand-eye calibration error, and a second hand-eye calibration matrix corresponding to the second hand-eye calibration error to determine a target hand-eye calibration matrix includes: The first hand-eye calibration error, each of the second hand-eye calibration errors, and a second hand-eye calibration matrix corresponding to each of the second hand-eye calibration errors are used as a set of correction data; Correcting the first hand-eye calibration matrix by each set of correction data to obtain a corrected third hand-eye calibration matrix, and verifying the third hand-eye calibration matrix according to a verification data set corresponding to the first hand-eye calibration matrix to obtain a third hand-eye calibration error; A target hand-eye calibration error that is the smallest and smaller than the first hand-eye calibration error is selected from the multiple third hand-eye calibration errors, and a third hand-eye calibration matrix corresponding to the target hand-eye calibration error is determined as the target hand-eye calibration matrix.

3. The method according to claim 2, characterized in that The first hand-eye calibration matrix includes a first rotation matrix and a first translation matrix, and the second hand-eye calibration matrix includes a second rotation matrix and a second translation matrix; The step of correcting the first hand-eye calibration matrix by using each group of the correction data to obtain a corrected third hand-eye calibration matrix comprises: Determining a rotation bias weight of the first rotation matrix and a translation bias weight of the first translation matrix based on the first hand-eye calibration error, the second hand-eye calibration error and a preset bias correction rate; Determine a rotation correction matrix according to the first rotation matrix, the second rotation matrix, and the rotation bias weight; Determine a translation correction matrix according to the first translation matrix, the second translation matrix, and the translation bias weight; A third hand-eye calibration matrix is ​​determined according to the rotation correction matrix and the translation correction matrix.

4. The method according to any one of claims 1 to 3, characterized in that: Each of the hand-eye calibration data sets includes multiple sets of calibration data, each set of calibration data includes a calibration image captured by a camera located at the end of the robotic arm, and a position and posture of the end of the robotic arm in a robot base coordinate system; The method of performing hand-eye calibration on the robot according to the training data set in the calibration data division result to obtain a hand-eye calibration matrix includes: According to the multiple groups of calibration images in the training data set in the calibration data division result and the multiple groups of robot arm end postures in the robot base coordinate system, the robot is calibrated by hand and eye to obtain a hand and eye calibration matrix.

5. The method according to claim 4, characterized in that The step of verifying the hand-eye calibration matrix according to the verification data set in the calibration data division result to obtain the hand-eye calibration error corresponding to the calibration data division result includes: For each group of calibration images in the validation data set in the calibration data division result, determine a first calibration point position of the calibration point in the robotic arm coordinate system according to the hand-eye calibration matrix and the robotic arm end posture corresponding to the calibration image, determine a second calibration point position of the calibration point in the robotic arm coordinate system according to the calibration image, and determine a distance error between the first calibration point position and the second calibration point position; Based on the distance errors corresponding to the multiple groups of calibration data, a distance error mean is determined, and the distance error mean is determined as the hand-eye calibration error.

6. A robot hand-eye calibration device, characterized in that: The device comprises: A data acquisition module, used to acquire multiple hand-eye calibration data sets for the robot; A data partitioning module is used to select each hand-eye calibration data set as a verification data set in turn, and use the remaining hand-eye calibration data sets other than the selected hand-eye calibration data set as a training data set to obtain multiple groups of calibration data partitioning results; A hand-eye calibration module is used to perform hand-eye calibration on the robot for each group of calibration data division results according to a training data set in the calibration data division results to obtain a hand-eye calibration matrix, and to verify the hand-eye calibration matrix according to a verification data set in the calibration data division results to obtain a hand-eye calibration error corresponding to the calibration data division results; A first optimization module is used to select a minimum first hand-eye calibration error from the hand-eye calibration errors corresponding to the division results of multiple groups of calibration data, and determine the hand-eye calibration matrix corresponding to the first hand-eye calibration error as the first hand-eye calibration matrix of the robot; The second optimization module is used to correct the first hand-eye calibration matrix and determine the target hand-eye calibration matrix based on the first hand-eye calibration error, the second hand-eye calibration error other than the first hand-eye calibration error, and the second hand-eye calibration matrix corresponding to the second hand-eye calibration error.

7. The device according to claim 6, characterized in that Each of the hand-eye calibration data sets includes multiple groups of calibration data, each group of the calibration data includes a calibration image captured by a camera located at the end of the robotic arm, and the posture of the robotic arm end in the robot base coordinate system. The hand-eye calibration module is used to perform hand-eye calibration on the robot according to the multiple groups of calibration images in the training data set in the calibration data division result, and the multiple groups of postures of the robotic arm end in the robot base coordinate system to obtain a hand-eye calibration matrix.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images