Robot hand-eye calibration method, device, robot and storage medium

Abstract

The present application relates to a robot hand-eye calibration method, device, robot and storage medium. The method includes: obtaining a preset number of points to be calibrated; calibrating each point to be calibrated respectively, obtaining the first position corresponding to each point to be calibrated, the first position representing the position of the marker in the visual coordinate system after being guided to the corresponding point to be calibrated by the robotic arm; matching each point to be calibrated and its corresponding first position, obtaining the conversion relationship between the robotic arm coordinate system and the visual coordinate system; based on the conversion relationship, converting each first position to the robotic arm coordinate system, obtaining the estimated position corresponding to each point to be calibrated; obtaining the matching error of each point to be calibrated according to the deviation between each point to be calibrated and the corresponding estimated position, and recalibrating the corresponding point to be calibrated when there is a matching error that does not meet the preset accuracy requirements. The use of this method can improve the reliability and accuracy of the calibration results.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot hand-eye calibration method, device, robot, and storage medium. Background Art

[0002] In a robotic hand-eye system, the process of obtaining the transformation relationship between the visual sensor coordinate system and the robotic arm coordinate system is generally called extrinsic calibration. This calibration process is widely used in various robotics fields, especially in scenarios where vision technology is required to guide the robotic arm to complete tasks. The calibration results largely determine the accuracy and reliability of the robotic arm's operation.

[0003] Currently, the hand-eye system calibration process generally uses a nine-point calibration method. This involves obtaining the positions of nine spatial points in the manipulator coordinate system and the visual coordinate system, matching them, and then obtaining a transformation relationship as the external parameter calibration result. However, a problem with current calibration methods is that the calibration results are heavily dependent on the given nine spatial points. If individual calibration points are abnormal, such as being outside the visual sensor's field of view or being recognized incorrectly, the reliability and accuracy of the calibration results will be seriously affected. Summary of the Invention

[0004] Based on this, it is necessary to provide a robot hand-eye calibration method, device, robot and storage medium that can improve the reliability and accuracy of calibration results in response to the above technical problems.

[0005] A robot hand-eye calibration method, the method comprising:

[0006] Acquire a preset number of points to be calibrated, each of which represents a designated position of the marker in the robotic arm coordinate system;

[0007] Calibrate each of the points to be calibrated respectively to obtain a first position corresponding to each of the points to be calibrated, where the first position represents the position of the marker in the visual coordinate system after the marker is guided to the corresponding point to be calibrated by the robotic arm;

[0008] Matching each of the to-be-calibrated points and its corresponding first position to obtain a conversion relationship between the robotic arm coordinate system and the visual coordinate system;

[0009] Convert each of the first positions to the robotic arm coordinate system based on the conversion relationship to obtain an estimated position corresponding to each of the points to be calibrated;

[0010] According to the deviation between each of the points to be calibrated and the corresponding estimated position, the matching error of each of the points to be calibrated is obtained. When there is a matching error that does not meet the preset accuracy requirement, the corresponding point to be calibrated is recalibrated until the matching errors of all the points to be calibrated meet the preset accuracy requirement.

[0011] A robot hand-eye calibration device, comprising:

[0012] An acquisition module is used to acquire a preset number of points to be calibrated, wherein the points to be calibrated represent designated positions of the marker in the robotic arm coordinate system;

[0013] a calibration module, configured to calibrate each of the points to be calibrated, and obtain a first position corresponding to each of the points to be calibrated, wherein the first position represents the position of the marker in the visual coordinate system after the marker is guided to the corresponding point to be calibrated by the robotic arm;

[0014] A matching module, configured to perform matching based on each of the to-be-calibrated points and its corresponding first position, to obtain a conversion relationship between the robotic arm coordinate system and the visual coordinate system;

[0015] a conversion module, configured to convert each of the first positions into the manipulator coordinate system based on the conversion relationship, and obtain an estimated position corresponding to each of the points to be calibrated;

[0016] The correction module is used to obtain the matching error of each of the points to be calibrated based on the deviation between each of the points to be calibrated and the corresponding estimated position. When there is a matching error that does not meet the preset accuracy requirement, the corresponding point to be calibrated is recalibrated until the matching errors of all the points to be calibrated meet the preset accuracy requirement.

[0017] A robot comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Acquire a preset number of points to be calibrated, each of which represents a designated position of the marker in the robotic arm coordinate system;

[0019] Calibrate each of the points to be calibrated respectively to obtain a first position corresponding to each of the points to be calibrated, where the first position represents the position of the marker in the visual coordinate system after the marker is guided to the corresponding point to be calibrated by the robotic arm;

[0020] Matching each of the to-be-calibrated points and its corresponding first position to obtain a conversion relationship between the robotic arm coordinate system and the visual coordinate system;

[0021] Convert each of the first positions to the robotic arm coordinate system based on the conversion relationship to obtain an estimated position corresponding to each of the points to be calibrated;

[0022] According to the deviation between each of the points to be calibrated and the corresponding estimated position, the matching error of each of the points to be calibrated is obtained. When there is a matching error that does not meet the preset accuracy requirement, the corresponding point to be calibrated is recalibrated until the matching errors of all the points to be calibrated meet the preset accuracy requirement.

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0024] Acquire a preset number of points to be calibrated, each of which represents a designated position of the marker in the robotic arm coordinate system;

[0025] Calibrate each of the points to be calibrated respectively to obtain a first position corresponding to each of the points to be calibrated, where the first position represents the position of the marker in the visual coordinate system after the marker is guided to the corresponding point to be calibrated by the robotic arm;

[0026] Matching each of the to-be-calibrated points and its corresponding first position to obtain a conversion relationship between the robotic arm coordinate system and the visual coordinate system;

[0027] Convert each of the first positions to the robotic arm coordinate system based on the conversion relationship to obtain an estimated position corresponding to each of the points to be calibrated;

[0028] According to the deviation between each of the points to be calibrated and the corresponding estimated position, the matching error of each of the points to be calibrated is obtained. When there is a matching error that does not meet the preset accuracy requirement, the corresponding point to be calibrated is recalibrated until the matching errors of all the points to be calibrated meet the preset accuracy requirement.

[0029] The above-mentioned robot hand-eye calibration method, device, robot and storage medium obtain a preset number of points to be calibrated, where the points to be calibrated represent the specified positions of the markers in the robotic arm coordinate system; calibrate each point to be calibrated respectively to obtain the first position corresponding to each point to be calibrated, where the first position represents the position of the marker in the visual coordinate system after being guided to the corresponding point to be calibrated by the robotic arm; match each point to be calibrated and its corresponding first position to obtain a conversion relationship between the robotic arm coordinate system and the visual coordinate system; based on the conversion relationship, convert each first position to the robotic arm coordinate system to obtain an estimated position corresponding to each point to be calibrated; obtain the matching error of each point to be calibrated based on the deviation between each point to be calibrated and the corresponding estimated position; when there is a matching error that does not meet the preset accuracy requirement, recalibrate the corresponding point to be calibrated until the matching errors of all points to be calibrated meet the preset accuracy requirement. Based on this, by evaluating the matching error of the calibrated points, possible error points or abnormal points in the calibration process can be found. Then, by recalibrating until the matching error meets the preset accuracy requirements, the problems in the calibration process can be corrected, thereby improving the reliability and accuracy of the calibration results. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 1 is a flow chart of a robot hand-eye calibration method according to an embodiment;

[0031] Figure 2 A schematic diagram of a flow chart of the steps of obtaining a preset number of points to be calibrated in one embodiment;

[0032] Figure 3 Schematic diagram of the positions of the points to be calibrated in the field of view of the visual sensor in one embodiment;

[0033] Figure 4 A schematic diagram of a flow chart of the steps for correcting a calibrated point in one embodiment;

[0034] Figure 5 1 is a flow chart of a robot hand-eye calibration method according to an embodiment;

[0035] Figure 6 1 is a structural block diagram of a robot hand-eye calibration device in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0037] The robot hand-eye calibration method provided in this application can be applied to a robot. A visual sensor is installed on the robot. During the calibration process, the visual sensor maintains a rigid connection with the robot's robotic arm base, that is, the relative position remains unchanged. The purpose of the robot hand-eye calibration is to obtain the position conversion relationship between the visual sensor and the robotic arm base.

[0038] In one embodiment, Figure 1 As shown, a robot hand-eye calibration method is provided, including the following steps S102 to S110.

[0039] S102: Obtain a preset number of points to be calibrated, where the points to be calibrated represent designated positions of the marker in the robotic arm coordinate system.

[0040] The "uncalibrated points" are the spatial locations that require calibration during the calibration process. They can also be understood as the spatial locations where the marker passes and is photographed. Specifically, the marker can be mounted at the end of the robot's robotic arm. The uncalibrated points are sequentially transmitted to the robotic arm, which guides the marker through all of them. When the marker reaches a point, it is considered to have reached a specified position, meaning its position in the robotic arm's coordinate system at that point is considered to be that point.

[0041] The number and arrangement of points to be calibrated can be set according to actual needs and are not limited here, as long as the number of point pairs obtained subsequently is sufficient to solve the conversion relationship between the robot coordinate system and the visual coordinate system. A larger number of point pairs can easily improve calibration accuracy and reduce the interference of individual erroneous points on the final calibration results, but it will consume more calibration time. In one embodiment, the preset number is 9, which is conducive to balancing calibration accuracy and calibration time cost.

[0042] S104, calibrate each point to be calibrated respectively to obtain a first position corresponding to each point to be calibrated, where the first position represents the position of the marker in the visual coordinate system after the marker is guided to the corresponding point to be calibrated by the robotic arm.

[0043] During the calibration process, each time the robotic arm guides the marker to a point to be calibrated, the visual sensor begins capturing images. Specifically, this can involve taking a picture of the marker as it enters the field of view and capturing the marker's position from the image. This is the marker's position in the visual coordinate system, which is also the first position corresponding to the point to be calibrated. The correspondence between each point to be calibrated and its corresponding first position is recorded, forming multiple point pairs. Each point pair includes a point to be calibrated and its corresponding first position.

[0044] S106 , performing matching based on each to-be-calibrated point and its corresponding first position, and obtaining a conversion relationship between the robot coordinate system and the visual coordinate system.

[0045] The transformation relationship can be a transformation matrix between the manipulator coordinate system and the visual coordinate system. After obtaining multiple point pairs through the aforementioned calibration process, the relative posture relationship between the visual coordinate system and the manipulator coordinate system can be solved based on these multiple point pairs to obtain the transformation matrix. The transformation matrix can be solved using any currently available or future method, and is not limited here.

[0046] S108: Convert each first position to the robot arm coordinate system based on the conversion relationship to obtain an estimated position corresponding to each point to be calibrated.

[0047] After obtaining the conversion relationship between the manipulator coordinate system and the visual coordinate system, a reverse calculation can be performed based on the conversion relationship and each first position to convert each first position to the manipulator coordinate system to obtain the position corresponding to each first position in the manipulator coordinate system (expressed as an estimated position to distinguish it from the specified position mentioned above). It can be understood that since each first position corresponds to a point to be calibrated, for any first position, the point to be calibrated corresponding to the estimated position calculated based on the first position is the point to be calibrated corresponding to the first position.

[0048] Specifically, the reverse calculation formula can be as follows: ig =MC i , where C i represents the first position corresponding to the i-th point to be calibrated, M represents the transformation matrix, P ig Indicates the estimated position corresponding to the i-th point to be calibrated.

[0049] S110, obtaining the matching error of each point to be calibrated based on the deviation between each point to be calibrated and the corresponding estimated position. When there is a matching error that does not meet the preset accuracy requirement, the corresponding point to be calibrated is recalibrated until the matching errors of all points to be calibrated meet the preset accuracy requirement.

[0050] The conversion relationship is obtained based on all the points to be calibrated and their corresponding first positions, reflecting the overall matching situation. The calibration process of each point to be calibrated is relatively independent. Therefore, there may be a certain deviation between the estimated position obtained by reverse calculation based on the conversion relationship and each first position and its corresponding point to be calibrated, and the degree of deviation may be different.

[0051] The matching error of each point to be calibrated can be determined based on the deviation between each point to be calibrated and the corresponding estimated position. Specifically, the matching error calculation formula can be as follows: i =P ig –P i , where e i represents the matching error of the i-th point to be calibrated, Pig Indicates the estimated position corresponding to the i-th point to be calibrated, P i Indicates the i-th point to be calibrated.

[0052] The matching error of the point to be calibrated is used to characterize whether there is a problem in the calibration process of the point to be calibrated, and the preset accuracy requirement is used to evaluate the matching error. When the matching error does not meet the preset accuracy requirement, it is considered that there is a problem in the calibration process of the corresponding point to be calibrated (for example, misidentification by the visual algorithm, jitter of the marker caused by the movement of the robotic arm, recognition started before the robotic arm is in place, etc.). If there is a problem, the point to be calibrated is considered to be an error point or an abnormal point, and it needs to be recalibrated to adjust the local calibration content. After recalibrating the error point or abnormal point, the conversion relationship will also change, and then the matching error of all the points to be calibrated will be recalculated until the matching error of all the points to be calibrated meets the preset accuracy requirement. When the matching error of all the points to be calibrated meets the preset accuracy requirement, the conversion matrix at this time is output as the final external parameter calibration result.

[0053] In the above-mentioned robot hand-eye calibration method, a preset number of points to be calibrated are obtained, and the points to be calibrated represent the specified positions of the markers in the robotic arm coordinate system; each point to be calibrated is calibrated separately to obtain the first position corresponding to each point to be calibrated, and the first position represents the position of the marker in the visual coordinate system after being guided to the corresponding point to be calibrated by the robotic arm; matching is performed according to each point to be calibrated and its corresponding first position to obtain the conversion relationship between the robotic arm coordinate system and the visual coordinate system; based on the conversion relationship, each first position is converted to the robotic arm coordinate system to obtain the estimated position corresponding to each point to be calibrated; based on the deviation between each point to be calibrated and the corresponding estimated position, the matching error of each point to be calibrated is obtained. When there is a matching error that does not meet the preset accuracy requirement, the corresponding point to be calibrated is recalibrated until the matching errors of all points to be calibrated meet the preset accuracy requirement. Based on this, by evaluating the matching error of the calibrated points, possible error points or abnormal points in the calibration process can be found. Then, by recalibrating until the matching error meets the preset accuracy requirements, the problems in the calibration process can be corrected, thereby improving the reliability and accuracy of the calibration results.

[0054] In one embodiment, when there is a matching error greater than a threshold among the matching errors of the points to be calibrated, it is determined that there is a matching error that does not meet the preset accuracy requirement; when the matching errors of the points to be calibrated are all less than or equal to the threshold, it is determined that the matching errors of all the points to be calibrated meet the preset accuracy requirement; the step of recalibrating the corresponding points to be calibrated can specifically be: selecting the maximum matching error from the matching errors greater than the threshold, and recalibrating the points to be calibrated corresponding to the maximum matching error.

[0055] The points to be calibrated and their corresponding estimated positions are both positions under the coordinates of the robotic arm and can be represented by three-dimensional coordinate vectors. Correspondingly, the matching error can also be represented as a three-dimensional coordinate vector. A matching error greater than a threshold value may mean that the coordinates of any one or several dimensions of the three-dimensional coordinate vector are greater than the threshold value, or the modulus of the three-dimensional coordinate vector is greater than the threshold value, which is not limited here. A matching error less than or equal to a threshold value may mean that the coordinates of all dimensions in the three-dimensional coordinate vector are less than or equal to the threshold value, or the modulus of the three-dimensional coordinate vector is less than or equal to the threshold value, which is not limited here. In addition, the size of the threshold value can be set according to actual needs and is not limited here. It should be noted that there may be multiple matching errors greater than the threshold value in the matching errors of each point to be calibrated. The largest matching error among the multiple matching errors greater than the threshold value is found, and the point to be calibrated corresponding to the largest matching error is recalibrated.

[0056] In this embodiment, considering that the conversion relationship is obtained based on all the points to be calibrated and their corresponding first positions, there may be a situation where the calibration process of only one point to be calibrated is abnormal, but at the same time the matching errors of multiple points to be calibrated do not meet the preset accuracy requirements. Therefore, only the point to be calibrated corresponding to the maximum matching error is recalibrated in each iteration, which can avoid recalibrating the points to be calibrated that originally had no abnormalities in the calibration process, resulting in a larger matching error, thereby improving the correction efficiency and correction accuracy of the calibration results.

[0057] In one embodiment, Figure 2 As shown, the step of obtaining a preset number of points to be calibrated may specifically include the following steps S202 to S204.

[0058] S202, obtaining a designated center point and a preset number of designated relative positions, where the designated center point satisfies a condition that when the marker moves to the designated center point, the marker is in the center area of ​​the visual sensor's field of view.

[0059] As the robotic arm guides the marker to different positions, the marker's appearance in the visual sensor's field of view varies. It may be outside or inside the visual sensor's field of view. When the marker is inside the visual sensor's field of view, it may be in the center or at the edge. Therefore, when the marker is observed to be in the center of the visual sensor's field of view, its position in the robotic arm's coordinate system at that time is recorded as the designated center point. A designated relative position refers to a position relative to a designated center point. The number of designated relative positions is equal to the number of points to be calibrated.

[0060] S204: Add the designated center point to each designated relative position to obtain a preset number of points to be calibrated.

[0061] The designated center point and each designated relative position can be represented by a three-dimensional coordinate vector. The designated center point and the designated relative position are added together. Specifically, the coordinates of each dimension are added together. Different designated relative positions will result in different points to be calibrated. For example, assuming that the designated center point (P0) is represented by (X0, Y0, Z0), the designated relative position (ΔP i ) is expressed as (ΔX i ,ΔY i ,ΔZ i ), then the point to be calibrated (P i =P0+ΔP i ) is expressed as (X0+ΔX i ,Y0+ΔY i ,Z0+ΔZ i ).like Figure 3 As shown in the figure, a schematic diagram of the corresponding positions of each point to be calibrated in the field of view of the visual sensor is provided, which includes 9 points to be calibrated, namely P1 to P9. The designated center point is the position of P5, which is located in the central area of ​​the field of view of the visual sensor. P5 is the center point of the combination of the 9 points to be calibrated. The other points to be calibrated except P5 are surrounded by P5. During actual calibration, it is only necessary to ensure that P5 is located in the central area of ​​the field of view of the visual sensor, and the distribution of the other points to be calibrated in the field of view of the visual sensor is uniform.

[0062] In this embodiment, the points to be calibrated are obtained by specifying a center point and a relative position, so that the points to be calibrated based on the specified center point can be distributed as evenly as possible within the field of view of the visual sensor, and at the same time, the points to be calibrated can cover as large a range as possible within the field of view of the visual sensor, which is beneficial to improving the reliability and accuracy of the calibration results, and is beneficial to the subsequent adjustment of the points to be calibrated.

[0063] In one embodiment, the steps of calibrating each point to be calibrated separately and obtaining the first position corresponding to each point to be calibrated may specifically include: for each point to be calibrated, performing the following calibration operation: controlling the robotic arm to guide the marker to move toward the point to be calibrated, and when it is detected that the marker has moved into place, controlling the visual sensor to capture an image, and obtaining the position of the marker in the visual coordinate system based on the image as the first position corresponding to the point to be calibrated.

[0064] Taking 9 points to be calibrated as an example, the calibration order of the 9 points to be calibrated is pre-set, and the points to be calibrated are represented in sequence as P1 to P9. P1 to P9 are sent to the robotic arm in sequence, and the robotic arm guides the marker to move to P1 to P9 in sequence. Whenever the marker moves to a point to be calibrated, the visual sensor is controlled to collect an image, and the position of the marker in the visual coordinate system is obtained based on the image, thereby obtaining the first positions corresponding to P1 to P9, respectively, which are represented as C1 to C9.

[0065] In one embodiment, when it is detected that the marker has moved into place, the step of controlling the visual sensor to capture an image may specifically include the following steps: when it is detected that the marker has moved into place, determining whether the marker appears within the field of view of the visual sensor; when the marker appears within the field of view of the visual sensor, controlling the visual sensor to capture an image.

[0066] When the marker is detected to be in position, it may not appear in the vision sensor's field of view. Continuing calibration without making adjustments at this point may result in calibration anomalies, affecting the final calibration results. Therefore, only controlling the vision sensor to capture images when the marker appears in the vision sensor's field of view helps ensure the reliability and accuracy of the calibration results.

[0067] In one embodiment, if the marker does not appear in the field of view of the visual sensor, the point to be calibrated is corrected, and the robotic arm is controlled to guide the marker toward the corrected point to be calibrated until the marker appears in the field of view of the visual sensor.

[0068] When the marker is detected to be in place, if the marker does not appear in the field of view of the visual sensor, the possible reason is that the relative position coordinates of the point to be calibrated and the specified center point are too large, causing the marker to exceed the field of view of the sensor, such as Figure 3 As shown, P9 is outside the sensor's field of view. Correction is performed on P9, specifically by moving P9 closer to the center point P5. After correction, P9 becomes P9'. The robotic arm is controlled to guide the marker toward the corrected point to be calibrated until the marker appears in the vision sensor's field of view. This controls the vision sensor to capture images, helping to ensure the reliability and accuracy of the calibration results.

[0069] In one embodiment, Figure 4 As shown, the step of correcting the calibration point may specifically include the following steps S402 to S404.

[0070] S402: Correct the designated relative position corresponding to the point to be calibrated based on a preset offset ratio to obtain a corrected relative position corresponding to the point to be calibrated.

[0071] The preset offset ratio is used to characterize the correction amplitude of the specified relative position, and its value range is 0 to 1 (excluding 0 and 1). It can be understood that the larger the preset offset ratio, the smaller the correction amplitude of the specified relative position, and the smaller the correction amplitude of the calibration point is; the smaller the preset offset ratio, the larger the correction amplitude of the specified relative position, and the larger the correction amplitude of the calibration point is. Specifically, the calculation formula for correcting the relative position can be as follows: ΔP i ′=μΔP i , where ΔP i Indicates the specified relative position corresponding to the i-th point to be calibrated, μ indicates the preset offset ratio, ΔP i ′ represents the corrected relative position corresponding to the i-th point to be calibrated.

[0072] S404: Add the corrected relative positions corresponding to the designated center point and the point to be calibrated to obtain a corrected point to be calibrated.

[0073] Specifically, the calculation formula for the corrected point to be calibrated can be as follows: i ′=P0+ΔP i ′=P0+μΔP i It should be noted that the specific value of μ can be set based on actual needs and is not limited here. The larger the value of μ, the more iterations may be required to move points outside the visual sensor's field of view back into its field of view, resulting in slower calibration speed. The smaller the value of μ, the closer points outside the visual sensor's field of view may be to the center point, resulting in reduced calibration accuracy.

[0074] In this embodiment, for the points to be calibrated that are beyond the visual field of the visual sensor, the designated relative position corresponding to the points to be calibrated is reduced so that the points to be calibrated are closer to the designated center point, so that they can accurately enter the visual field of the visual sensor, thereby achieving real-time and accurate correction of the points to be calibrated during the calibration process. In addition, the correction method for the points to be calibrated in this embodiment can be applied to scenarios where the hardware structure changes. For example, when the relative position of the visual sensor and the robotic arm changes, the problematic points to be calibrated can be corrected during the calibration process through this embodiment, avoiding the situation where a fixed set of calibration points becomes invalid due to structural changes and cannot be used normally, thereby avoiding the reliance on professionals for frequent design of calibration points and reducing maintenance costs.

[0075] In one embodiment, the marker is a QR code. The marker is used to match the common point on the visual coordinate system and the robotic arm coordinate system. Traditional markers are generally graphics with specific colors, sharp ends, etc., but such markers require customized visual algorithms to complete the target recognition and positioning process. The accuracy and reliability of this recognition process inevitably rely on the personal ability of the algorithm developer, increasing the uncontrollable factors of the calibration process. At the same time, such markers themselves also have problems. Markers with fixed colors may be interfered with by similar colors in the light and background, and the sharp ends may be affected by the defocus of the visual sensor. Therefore, this embodiment uses a QR code as a marker. The QR code recognition algorithm is highly standardized, has strong environmental adaptability, and is stable, reliable, and accurate.

[0076] In one embodiment, AprilTag or ArUco QR codes are used as markers. These QR codes have been specifically optimized to significantly improve recognition reliability compared to ordinary QR codes. They can also provide depth information without relying on a depth camera, even with only a single 2D vision sensor. This allows the marker's spatial position in the visual coordinate system to be directly obtained, increasing the accuracy and reliability of the calibration process while reducing the complexity of code development.

[0077] Specifically, the process of visual sensors recognizing markers includes the following two schemes: First, for a visual system with both a depth camera and a two-dimensional camera, the two-dimensional camera is first used to identify the QR code marker to obtain the coordinates of the marker in the pixel coordinate system, and then the spatial position of the marker in the visual coordinate system of the depth camera is obtained through a matching algorithm; Second, for a visual system with only a two-dimensional camera, the internal parameters of the two-dimensional camera can be obtained in advance through the Zhang Zhengyou calibration method, and then based on the characteristics of the QR code marker itself, its spatial position in the two-dimensional camera coordinate system can be directly identified.

[0078] It should be noted that the AprilTag QR code or ArUco QR code used in the calibration process can be replaced by other markers when a depth camera or other visual sensor capable of directly feedback spatial coordinates is available. These markers include other QR codes, calibration checkerboards, and other special graphics with highly regular features and mature and universal recognition algorithms.

[0079] In one embodiment, Figure 5 As shown, a robot hand-eye calibration method is provided, including the following steps S501 to S506.

[0080] S501, obtaining a designated center point and a preset number of designated relative positions, where the designated center point satisfies a condition that when the marker moves to the designated center point, the marker is in the center area of ​​the visual sensor's field of view.

[0081] S502: Add the designated center point to each designated relative position to obtain a preset number of points to be calibrated.

[0082] S503, for each point to be calibrated, perform the following calibration operation: control the robotic arm to guide the marker to move toward the point to be calibrated, and when it is detected that the marker has moved into place, determine whether the marker appears in the field of view of the visual sensor. If the marker does not appear in the field of view of the visual sensor, correct the point to be calibrated, control the robotic arm to guide the marker to move toward the corrected point to be calibrated until the marker appears in the field of view of the visual sensor, control the visual sensor to capture an image, and obtain the position of the marker in the visual coordinate system based on the image as the first position corresponding to the point to be calibrated.

[0083] S504: Matching is performed based on each point to be calibrated and its corresponding first position to obtain a conversion relationship between the robot coordinate system and the visual coordinate system.

[0084] S505: Convert each first position to the robot arm coordinate system based on the conversion relationship to obtain an estimated position corresponding to each point to be calibrated.

[0085] S506: Obtain the matching error of each point to be calibrated based on the deviation between each point to be calibrated and the corresponding estimated position. When there is a matching error greater than a threshold among the matching errors of each point to be calibrated, select the maximum matching error from the matching errors greater than the threshold, and recalibrate the point to be calibrated corresponding to the maximum matching error until the matching errors of all points to be calibrated are less than or equal to the threshold. Output the final conversion relationship as the calibration result.

[0086] For the detailed description of steps S501 to S506, please refer to the above embodiment and will not be repeated here. In this embodiment, on the one hand, by evaluating the matching error of the calibrated points, possible error points or abnormal points in the calibration process can be found, and then by recalibrating until the matching error meets the preset accuracy requirements, the problems existing in the calibration process can be corrected, thereby improving the reliability and accuracy of the calibration results; on the other hand, by specifying the center point and the relative position to obtain the calibrated points, the calibrated points based on the specified center point can be distributed as evenly as possible within the visual sensor field of view, and at the same time, the calibrated points can cover as large a range as possible within the visual sensor field of view, which is conducive to improving the reliability and accuracy of the calibration results and is conducive to the subsequent adjustment of the calibrated points; on the other hand, during the calibration process, the calibrated points that exceed the visual sensor field of view are corrected in real time, avoiding the situation where a fixed set of calibration points becomes invalid due to structural changes and cannot be used normally, thereby avoiding the frequent design of calibration points and the reliance on professionals, reducing maintenance costs.

[0087] It should be noted that the above embodiments can be applied not only to the calibration of three-dimensional space to obtain the spatial transformation relationship between two three-dimensional coordinate systems, but also to the calibration of two-dimensional planes to obtain the perspective transformation relationship between two two-dimensional plane coordinate systems in space. For example, two-dimensional plane calibration can be achieved by restricting the movement of a robotic arm to a fixed plane. Specifically, in the visual coordinate system, a third dimension containing a fixed value can be added. In the robotic arm coordinate system, constraints can be added to the three-dimensional coordinates to ensure that they always lie on a specific plane, and then the perspective transformation relationship between the camera plane and the robotic arm plane can be solved.

[0088] It should be understood that, although the various steps in the various flow charts that the above-described embodiments relate to are shown in sequence according to the indications of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the various flow charts that the above-described embodiments relate to can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0089] In one embodiment, Figure 6 As shown, a robot hand-eye calibration device 600 is provided, comprising: an acquisition module 610, a calibration module 620, a matching module 630, a conversion module 640 and a correction module 650, wherein:

[0090] The acquisition module 610 is used to acquire a preset number of points to be calibrated, where the points to be calibrated represent designated positions of the marker in the robotic arm coordinate system.

[0091] The calibration module 620 is used to calibrate each point to be calibrated respectively to obtain a first position corresponding to each point to be calibrated. The first position represents the position of the marker in the visual coordinate system after the marker is guided to the corresponding point to be calibrated by the robotic arm.

[0092] The matching module 630 is used to perform matching based on each point to be calibrated and its corresponding first position to obtain a conversion relationship between the robot coordinate system and the visual coordinate system.

[0093] The conversion module 640 is used to convert each first position into the robot arm coordinate system based on the conversion relationship to obtain the estimated position corresponding to each point to be calibrated.

[0094] Correction module 650 is used to obtain the matching error of each point to be calibrated based on the deviation between each point to be calibrated and the corresponding estimated position. When there is a matching error that does not meet the preset accuracy requirement, the corresponding point to be calibrated is recalibrated until the matching errors of all points to be calibrated meet the preset accuracy requirement.

[0095] In one embodiment, the apparatus further includes a determination module configured to: determine that a matching error does not meet the preset accuracy requirement when a matching error greater than a threshold exists among the matching errors of the points to be calibrated; and determine that the matching errors of all the points to be calibrated meet the preset accuracy requirement when the matching errors of the points to be calibrated are less than or equal to the threshold. When recalibrating the corresponding points to be calibrated, the correction module 650 is specifically configured to: select the maximum matching error from the matching errors greater than the threshold, and recalibrate the point to be calibrated corresponding to the maximum matching error.

[0096] In one embodiment, when acquiring a preset number of points to be calibrated, the acquisition module 610 is specifically used to: acquire a designated center point and a preset number of designated relative positions, where the conditions satisfied by the designated center point include: when the marker moves to the designated center point, the marker is in the central area of ​​the visual sensor's field of view; and add the designated center point to each designated relative position to obtain a preset number of points to be calibrated.

[0097] In one embodiment, when the calibration module 620 calibrates each point to be calibrated respectively and obtains the first position corresponding to each point to be calibrated, it is specifically used to: for each point to be calibrated, perform the following calibration operation: control the robotic arm to guide the marker to move toward the point to be calibrated, and when it is detected that the marker moves into place, control the visual sensor to capture an image, and obtain the position of the marker in the visual coordinate system based on the image as the first position corresponding to the point to be calibrated.

[0098] In one embodiment, when the marker is detected to be moved into place, the calibration module 620 controls the visual sensor to capture an image. Specifically, when the marker is detected to be moved into place, the calibration module 620 is used to: determine whether the marker appears in the field of view of the visual sensor; when the marker appears in the field of view of the visual sensor, control the visual sensor to capture an image.

[0099] In one embodiment, the calibration module 620 is also used to: if the marker does not appear in the field of view of the visual sensor, correct the point to be calibrated, and control the robotic arm to guide the marker toward the corrected point to be calibrated until the marker appears in the field of view of the visual sensor.

[0100] In one embodiment, when correcting the point to be calibrated, the calibration module 620 is specifically used to: correct the specified relative position corresponding to the point to be calibrated based on a preset offset ratio to obtain the corrected relative position corresponding to the point to be calibrated; add the specified center point and the corrected relative position corresponding to the point to be calibrated to obtain the corrected point to be calibrated.

[0101] In one embodiment, the marker is a QR code.

[0102] The specific definition of the robot hand-eye calibration device can be found in the definition of the robot hand-eye calibration method above and will not be repeated here. The various modules in the above-mentioned robot hand-eye calibration device can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0103] In one embodiment, a robot is provided, comprising a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.

[0104] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.

[0105] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0106] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0107] It should be understood that the terms "first," "second," and the like in the above embodiments are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Furthermore, in the description of this application, unless otherwise specified, "a plurality of" means at least two.

[0108] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0109] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A robot hand-eye calibration method, characterized in that: The method comprises: Obtain a designated center point and a preset number of designated relative positions, wherein the conditions satisfied by the designated center point include: when the marker moves to the designated center point, the marker is in the central area of ​​the visual sensor's field of view; respectively add the designated center point to the coordinates of each dimension represented by the three-dimensional coordinate vector of each designated relative position to obtain the preset number of points to be calibrated, wherein the points to be calibrated represent the designated positions of the marker in the robot arm coordinate system; Calibrate each of the points to be calibrated respectively to obtain a first position corresponding to each of the points to be calibrated, wherein the first position represents a position of the marker in the visual coordinate system after the marker is guided to the corresponding point to be calibrated by the robotic arm; Matching is performed according to each of the to-be-calibrated points and its corresponding first position to obtain a conversion relationship between the robot arm coordinate system and the visual coordinate system; Based on the conversion relationship, each of the first positions is converted into the robot arm coordinate system to obtain an estimated position corresponding to each of the points to be calibrated; According to the deviation between each of the points to be calibrated and the corresponding estimated position, the matching error of each of the points to be calibrated is obtained; when there is a matching error greater than a threshold among the matching errors of the points to be calibrated, it is determined that there is a matching error that does not meet the preset accuracy requirement; when the matching errors of each of the points to be calibrated are less than or equal to the threshold, it is determined that the matching errors of all the points to be calibrated meet the preset accuracy requirement; when there is a matching error that does not meet the preset accuracy requirement, the maximum matching error is selected from the matching errors greater than the threshold, the point to be calibrated corresponding to the maximum matching error is recalibrated, and the conversion relationship between the robot coordinate system and the visual coordinate system, as well as the matching errors of each of the points to be calibrated, are re-obtained, until the matching errors of all the points to be calibrated meet the preset accuracy requirement.

2. The method according to claim 1, characterized in that Calibrate each of the points to be calibrated respectively to obtain a first position corresponding to each of the points to be calibrated, including: For each of the points to be calibrated, the following calibration operation is performed: the robotic arm is controlled to guide the marker to move toward the point to be calibrated. When it is detected that the marker has moved into place, the visual sensor is controlled to capture an image, and the position of the marker in the visual coordinate system is obtained based on the image as the first position corresponding to the point to be calibrated.

3. The method according to claim 2, characterized in that When it is detected that the marker moves into position, the visual sensor is controlled to collect images, including: When it is detected that the marker has moved into position, determining whether the marker appears within the field of view of the visual sensor; When the marker appears in the field of view of the visual sensor, the visual sensor is controlled to collect images.

4. The method according to claim 3, characterized in that Also includes: If the marker does not appear in the field of view of the visual sensor, the point to be calibrated is corrected, and the robotic arm is controlled to guide the marker to move toward the corrected point to be calibrated until the marker appears in the field of view of the visual sensor.

5. The method according to claim 4, characterized in that Get a preset number of points to be calibrated, including: Acquire a designated center point and a preset number of designated relative positions, wherein the designated center point satisfies a condition that: when the marker moves to the designated center point, the marker is in the center area of ​​the visual sensor's field of view; Adding the designated center point to each of the designated relative positions respectively to obtain the preset number of points to be calibrated; Correcting the point to be calibrated includes: Correcting the specified relative position corresponding to the point to be calibrated based on a preset offset ratio to obtain a corrected relative position corresponding to the point to be calibrated; The designated center point is added to the corrected relative position corresponding to the point to be calibrated to obtain a corrected point to be calibrated.

6. The method according to any one of claims 1 to 5, characterized in that The marker is a QR code.

7. A robot hand-eye calibration device, characterized in that: The device comprises: An acquisition module is used to acquire a designated center point and a preset number of designated relative positions, wherein the conditions satisfied by the designated center point include: when the marker moves to the designated center point, the marker is in the central area of ​​the visual sensor's field of view; the designated center point is respectively added to the coordinates of each dimension represented by the three-dimensional coordinate vector of each designated relative position to obtain the preset number of points to be calibrated, wherein the points to be calibrated represent the designated positions of the marker in the robot arm coordinate system; A calibration module, used to calibrate each of the points to be calibrated, and obtain a first position corresponding to each of the points to be calibrated, wherein the first position represents the position of the marker in the visual coordinate system after the marker is guided to the corresponding point to be calibrated by the robot arm; A matching module, used for matching the points to be calibrated and their corresponding first positions to obtain a conversion relationship between the robot coordinate system and the visual coordinate system; A conversion module, used for converting each of the first positions into the mechanical arm coordinate system based on the conversion relationship to obtain an estimated position corresponding to each of the points to be calibrated; The correction module is used to obtain the matching error of each of the points to be calibrated according to the deviation between each of the points to be calibrated and the corresponding estimated position; when there is a matching error greater than a threshold value among the matching errors of each of the points to be calibrated, it is determined that there is a matching error that does not meet the preset accuracy requirement; when the matching errors of each of the points to be calibrated are less than or equal to the threshold value, it is determined that the matching errors of all the points to be calibrated meet the preset accuracy requirement; when there is a matching error that does not meet the preset accuracy requirement, the maximum matching error is selected from the matching errors greater than the threshold value, the point to be calibrated corresponding to the maximum matching error is recalibrated, and the conversion relationship between the robot arm coordinate system and the visual coordinate system, as well as the matching errors of each of the points to be calibrated are re-acquired, until the matching errors of all the points to be calibrated meet the preset accuracy requirement.

8. The device according to claim 7, characterized in that When the calibration module calibrates each point to be calibrated respectively and obtains the first position corresponding to each point to be calibrated, it is specifically used to: for each point to be calibrated, perform the following calibration operation: control the robotic arm to guide the marker to move toward the point to be calibrated, and when it is detected that the marker has moved into place, control the visual sensor to capture an image, and obtain the position of the marker in the visual coordinate system based on the image as the first position corresponding to the point to be calibrated.

9. A robot comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images