A verification method, device, readable storage medium and program product

Abstract

Embodiments of the present application disclose a verification method and device, a readable storage medium and a program product. In the method, a robot determines a first visual estimated pose of a dexterous hand of the robot, the first visual estimated pose being an estimated pose of the dexterous hand at a verification point, the first visual estimated pose being obtained by three-dimensional reconstruction of feature information determined based on a second visual estimated pose, the second visual estimated pose being an estimated pose of the dexterous hand at a previous adjacent transition point of the verification point; the robot reads a robot kinematics nominal pose of the dexterous hand, the robot kinematics nominal pose being a theoretical pose of the dexterous hand at the verification point; and the robot determines whether the robot needs to re-perform hand-eye calibration based on the first visual estimated pose and the robot kinematics nominal pose. It can be seen that the robot takes the dexterous hand of the robot as a visual target, and the hand-eye calibration precision verification can be completed without additional installation of a target, thereby reducing the complexity of the verification.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a verification method, apparatus, readable storage medium, and program product. Background Technology

[0002] Hand-eye calibration refers to establishing the coordinate transformation relationship between the camera coordinate system and the robot coordinate system, enabling the robot to map visual perception information to the operating space, thereby achieving precise positioning and grasping of target objects. The accuracy of hand-eye calibration affects the robot's grasping performance; therefore, after completing hand-eye calibration, the robot needs to verify whether the calibration accuracy meets the preset indicators.

[0003] One verification method involves installing special targets such as calibration boards and concentric circle stickers on the dexterous hand or in the workspace, and then comparing the target posture detected by the camera with the theoretical target posture obtained by the robot to determine whether the calibration accuracy meets the requirements.

[0004] The aforementioned calibration process requires repeated installation, disassembly, and removal of the specialized target, resulting in cumbersome steps and a long processing time. Therefore, further research is needed on how to reduce the complexity of the hand-eye calibration accuracy calibration process. Summary of the Invention

[0005] This application provides a verification method, apparatus, readable storage medium, and program product that can reduce the complexity of the hand-eye calibration accuracy verification process.

[0006] Firstly, embodiments of this application provide a verification method. The method includes: The robot determines the first visual estimated pose of the robot's dexterous hand. The first visual estimated pose is the estimated pose of the dexterous hand at the check point. The first visual estimated pose is obtained by 3D reconstruction based on the feature information determined by the second visual estimated pose. The second visual estimated pose is the estimated pose of the dexterous hand at the adjacent transition point before the check point. The robot reads the nominal kinematic pose of the dexterous hand, which is the theoretical pose of the dexterous hand at the check point. Based on the robot's first visual pose estimation and the robot's kinematic nominal pose, it determines whether the robot needs to undergo hand-eye calibration again.

[0007] In this method, the robot can determine whether hand-eye calibration needs to be re-calibrated by estimating the pose through the first visual perception of the dexterous hand and the nominal kinematic pose of the robot. This eliminates the need for repeated installation, disassembly, and removal of the target, thereby reducing the complexity of the hand-eye calibration accuracy verification process.

[0008] Furthermore, since this method eliminates the need for specialized targets such as calibration boards and concentric circle stickers, it reduces the ongoing additional costs associated with purchasing, replacing, and maintaining specialized targets. It also minimizes damage to precision end-effectors such as dexterous hands caused by repeated installation and disassembly. Because installing specialized targets requires a high level of operator skill, this method also lowers the operational barrier to hand-eye calibration accuracy verification, facilitating its wider application.

[0009] Furthermore, in this method, the robot can complete the calibration accuracy verification during normal operation without the need for periodic shutdowns for calibration verification, and without the need to check whether recalibration is needed after a grasping failure, thereby saving downtime and improving the accuracy of the robot's grasping operation.

[0010] In one alternative implementation, the robot determines the first visually estimated pose of its dexterous hand, specifically as follows: The robot estimates the pose based on its second vision and projects the three-dimensional (3D) feature points on the computer-aided design (CAD) model of the dexterous hand into the left and right eye images respectively, thus obtaining the two-dimensional (2D) feature points of the left eye image and the two-dimensional feature points of the right eye image. The three-dimensional feature points are obtained through structural downsampling. The robot triangulates the cross-view matching pairs consisting of two-dimensional feature points from the left and right images to obtain the reconstructed three-dimensional feature points. The robot uses the iterative closest point (ICP) algorithm to register the reconstructed 3D feature points with each other to obtain the first visual estimate of the pose.

[0011] In this embodiment, the robot reconstructs three-dimensional feature points through binocular vision triangulation, and then performs iterative nearest-point registration between the reconstructed three-dimensional feature points and the three-dimensional feature points on the computer-aided design model to obtain the first visual estimated pose, thereby improving the estimation accuracy of the first visual estimated pose.

[0012] In one optional implementation, the second visually estimated pose is determined based on the visually estimated pose of the dexterous hand at the starting point. The robot determines the visually estimated pose of the dexterous hand at the starting point as follows: The robot uses the three-dimensional feature points on the computer-aided design model of the dexterous hand to determine the two-dimensional feature points of the dexterous hand in the image at the starting point; The robot determines the visually estimated pose of its dexterous hand at the starting point based on matching pairs consisting of two-dimensional and three-dimensional feature points.

[0013] In this embodiment, the robot determines the visually estimated pose of the dexterous hand at the starting point by matching pairs composed of two-dimensional feature points and three-dimensional feature points, which helps to provide a reliable initial visually estimated pose for subsequent operations.

[0014] In one alternative implementation, the robot determines the second visual estimated pose by: The robot projects the contour points of the computer-aided design model of the dexterous hand into a template contour based on the initial pose. The initial pose is the visually estimated pose of the dexterous hand of the path node preceding the adjacent transition point of the verification point. The robot updates its initial pose based on the difference between the template contour and the edge contour, which is extracted from the image of the dexterous hand at the previous adjacent transition point of the verification point. The robot determines the updated initial pose as the second visual estimate pose.

[0015] In this implementation, the robot iteratively adjusts its initial pose by comparing the difference between the template contour and the edge contour to determine the second visual estimated pose, without the need to detect and match feature points, thereby improving computational efficiency.

[0016] In one optional implementation, the robot determines whether it needs to re-calibrate its hand-eye relationship based on the first visual estimation of its pose and the robot's nominal kinematic pose. Specifically: Based on the robot's first visual estimated pose and the robot's nominal kinematic pose, the robot determines the displacement difference and angle difference between the first visual estimated pose and the robot's nominal kinematic pose; If the robot's displacement difference or angle difference exceeds its corresponding first threshold, it is determined that the robot needs to recalibrate its hand and eye. If both the displacement difference and the angle difference are less than their corresponding second thresholds, it is determined that the robot does not need to undergo hand-eye calibration again.

[0017] In this implementation, the robot can quickly determine whether hand-eye calibration needs to be re-performed by comparing the first visual estimated pose of the dexterous hand at a single calibration point with the robot's kinematic nominal pose, thereby improving calibration efficiency.

[0018] In one optional implementation, when there are multiple calibration points, the robot determines whether it needs to re-calibrate its hand-eye relationship based on the first visual pose estimate and the robot's nominal kinematic pose. Specifically: Based on the first visual estimation of the pose and the nominal kinematic pose of the robot at multiple verification points, the robot determines the first difference between the estimated displacement difference and the theoretical displacement difference between each two adjacent verification points, and the second difference between the estimated angle difference and the theoretical angle difference between each two adjacent verification points. If the robot's first or second difference is greater than its corresponding third threshold, it is determined that the robot needs to recalibrate its hand and eye. If both the first and second differences are less than their corresponding fourth thresholds, it is determined that the robot does not need to recalibrate its hand and eye.

[0019] In this implementation, the robot can make a comprehensive judgment based on the first visual estimated pose and the robot's kinematic nominal pose at multiple verification points to determine whether hand-eye calibration needs to be re-performed, thereby improving the reliability of the verification results.

[0020] In one alternative implementation, the check point is a path node where the dexterous hand needs to stop on a continuous movement path and where the visually estimated posture of the dexterous hand at the check point is determined. The continuous motion path includes a starting point, a check point, and a transition point. The starting point is the initial path node on the continuous motion path, the check point is received through the robot's control interface, and the transition point is obtained through interpolation.

[0021] In this embodiment, the robot's dexterous hand can autonomously complete the movement and verification along a continuous movement path, thereby reducing the human cost of verification.

[0022] Secondly, embodiments of this application provide a robotic device that includes a unit for performing the method as described in the first aspect.

[0023] In one optional implementation, the processing unit is used to determine the first visual estimated pose of the robot's dexterous hand. The first visual estimated pose is the estimated pose of the dexterous hand at the verification point. The first visual estimated pose is obtained by three-dimensional reconstruction of the feature information determined based on the second visual estimated pose. The second visual estimated pose is the estimated pose of the dexterous hand at the previous adjacent transition point of the verification point. The reading unit is used to read the robot's nominal kinematic pose of the dexterous hand, which is the theoretical pose of the dexterous hand at the verification point. The processing unit is also used to determine whether the robot needs to recalibrate its hand and eye based on the first visual estimation of the pose and the robot's kinematic nominal pose.

[0024] Other alternative implementations of the robotic device and their beneficial effects can be found in the relevant content of the first aspect above, and will not be repeated here.

[0025] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-readable instructions that, when executed on a robotic device, cause the robotic device to perform the method described in the first aspect.

[0026] Fourthly, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the method described in the first aspect. Attached Figure Description

[0027] Figure 1 A flowchart illustrating a verification method provided in an embodiment of this application; Figure 2 A flowchart illustrating another verification method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a robot device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0028] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] It should be understood that the terms "comprising" or "including" indicate the presence of the stated features, steps, operations, elements, components, items, types, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, types, and / or groups. The terms "or," "and / or," and "comprising at least one of the following," as used in this application, can be interpreted as inclusive, or mean any one or any combination thereof. For example, "comprising at least one of the following: A, B, C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C," and similarly, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C." Exceptions to this definition only occur when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0030] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0031] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0032] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0033] The following describes the terms related to the embodiments of this application.

[0034] 1. Object Detection Object detection refers to the detection of regions of interest in an image. Typically, labeled data is used as training samples, and a neural network is trained using deep learning. Once trained, the neural network is able to detect regions of interest in an image.

[0035] 2. Six-degrees-of-freedom (6D) attitude estimation 6D pose estimation refers to calculating the 6D pose of a target object in the camera coordinate system using data such as images, depth maps, and templates. The 6D pose can be represented as (x, y, z, rx, ry, rz), where (x, y, z) can be referred to as or understood as translation components or translation vectors, and (rx, ry, rz) can be referred to as or understood as rotation components or rotation vectors. The translation and rotation vectors can be used to represent the position and orientation of the target object in three-dimensional space, respectively.

[0036] 3. Object Tracking Object tracking refers to using the target detection results of the initial frame as prior information to track the position or pose changes of the same target object in consecutive image frames.

[0037] 4. Contour Feature Extraction Contour feature extraction refers to the process of extracting the edge contour information of a target object from an image by performing grayscale conversion, binarization, edge detection, and other processing on the original image, in order to obtain a geometric description of the target object.

[0038] 5. Binocular imaging Binocular imaging refers to the process of simultaneously acquiring left and right images using two cameras that have been calibrated for intrinsic and extrinsic parameters, and calculating the depth information of target objects in the scene based on the disparity map between the left and right images.

[0039] 6. Triangulation Triangulation is a 3D reconstruction technique. Based on the intrinsic and extrinsic parameters of a binocular camera, this technique finds matching pixels in the left and right eye images and calculates the coordinates of these pixels in 3D space.

[0040] 7. Calibration and Verification Calibration and verification are generally divided into camera calibration and accuracy verification. Camera calibration is used to solve for distortion coefficients, intrinsic and extrinsic parameters, etc.; accuracy verification refers to evaluating the calibration accuracy by additionally installing special targets such as calibration boards and concentric circle stickers on the robot's dexterous hand or in the workspace, and calculating indicators such as reprojection error.

[0041] However, the need for repeated disassembly and assembly of the dedicated target makes the calibration process cumbersome and time-consuming; the procurement, replacement, and maintenance of the calibration plate and concentric circle stickers also increase ongoing additional costs; frequent disassembly and assembly can also easily damage precision end-effectors such as dexterous hands. In addition, this calibration method requires a high level of operator proficiency, and beginners need extensive guidance to perform it correctly, making it difficult to adopt and popularize.

[0042] To address the aforementioned issues, this application provides a verification method in which a robot can verify the accuracy of hand-eye calibration using its own dexterous hand, without the need for additional target installation, thereby reducing the complexity of the verification.

[0043] Figure 1 This is a flowchart illustrating a verification method provided in an embodiment of this application, using a robot as an example. The method includes, but is not limited to, the following steps: S101. The robot determines the first visual estimated pose of the robot's dexterous hand. The first visual estimated pose is the estimated pose of the dexterous hand at the verification point. The first visual estimated pose is obtained by three-dimensional reconstruction based on the feature information determined by the second visual estimated pose. The second visual estimated pose is the estimated pose of the dexterous hand at the previous adjacent transition point of the verification point.

[0044] A checkpoint is a path node (or trajectory point) on a continuous motion path where the dexterous hand must stop and where its visually estimated posture is determined. This continuous motion path can be used to define the 6D posture changes of the dexterous hand (or, for example, an end effector flange) in space. Specifically, as the dexterous hand moves along this path, its translation and rotation vectors change with the path. This continuous motion path includes a starting point, a checkpoint, and transition points. The starting point is the initial path node of the continuous motion path, and the transition points are used to smooth the posture changes of the dexterous hand and maintain its posture tracking.

[0045] Optionally, the verification points are received through the robot's control interface. Specifically, the robot receives one or more verification points from the operator via the control interface, obtained by adjusting the translation vector (x, y, z) and rotation vector (rx, ry, rz) based on the starting point. The starting point can also be received through the robot's control interface.

[0046] Optionally, transition points are obtained through interpolation. Specifically, the robot obtains one or more transition points between two adjacent path nodes through interpolation.

[0047] In one optional implementation, before the robot executes S101, the following steps are also performed: the robot receives a starting point and one or more check points through the control interface; the robot sorts the starting point and the one or more check points to obtain a path node sequence; the robot determines multiple transition points between every two adjacent nodes in the path node sequence by interpolation; the robot generates a continuous motion path based on the path node sequence and the multiple transition points.

[0048] Optionally, before the robot executes S101, a continuous motion path is determined through the above implementation method only if there is no continuous motion path.

[0049] In one optional implementation, S101 includes: the robot projects three-dimensional feature points (feature points can also be called key points) on the computer-aided design model of the dexterous hand onto the left and right eye images respectively based on the second visual estimated pose, to obtain two-dimensional feature points in the left eye image and two-dimensional feature points in the right eye image, and the three-dimensional feature points are obtained by structural downsampling; the robot triangulates the cross-view matching pair composed of two-dimensional feature points in the left eye image and two-dimensional feature points in the right eye image to obtain the reconstructed three-dimensional feature points; the robot registers the reconstructed three-dimensional feature points with the three-dimensional feature points using an iterative nearest-point algorithm to obtain the first visual estimated pose.

[0050] In this method, the robot reconstructs three-dimensional feature points through binocular vision triangulation, and then performs iterative nearest-point registration between the reconstructed three-dimensional feature points and the three-dimensional feature points on the computer-aided design model to obtain the first visual estimated pose, thereby improving the estimation accuracy of the first visual estimated pose.

[0051] Because the fingers of a dexterous hand are highly similar, direct binocular matching can lead to mismatches, causing subsequent registration errors. Therefore, in this approach, the robot uses the second vision's estimated pose as a priori spatial constraint, projecting the 3D feature points of the dexterous hand's computer-aided design model onto the left and right eye images. This compresses the matching range to a small confidence interval, reducing mismatches caused by the high similarity of the fingers. Thus, the improvement in the estimation accuracy of the first vision's estimated pose does not stem from the projection, triangulation, or iterative nearest-point algorithm itself, but rather relies on the prior information provided by the second vision's estimated pose.

[0052] Optionally, the two-dimensional feature points of the left and right images obtained by projecting the three-dimensional feature points on the computer-aided design model of the dexterous hand based on the robot's pose estimation using the second vision can both be understood as feature information determined based on the pose estimation using the second vision.

[0053] Optionally, in addition to improving the estimation accuracy of the first visual pose estimation through the above methods, the robot can also perform pose optimization of dense point clouds based on binocular imaging to improve the estimation accuracy of the first visual pose estimation.

[0054] The following describes possible implementation methods for robots to determine the estimated pose using a second vision.

[0055] Implementation method 1.1 The robot determines the second visual estimated pose based on the visual estimated pose of the path node of the adjacent transition point before the verification point of the dexterous hand.

[0056] Taking a continuous motion path consisting of a starting point, transition point 1, transition point 2, and verification point 1 in sequence, and the visually estimated posture of the dexterous hand at transition point 2 as the second visually estimated posture, the robot determines the second visually estimated posture based on the visually estimated posture of the dexterous hand at transition point 1.

[0057] Taking a continuous motion path consisting of a starting point, transition point 1, and verification point 1 in sequence, and the visually estimated posture of the dexterous hand at transition point 1 as the second visually estimated posture, the robot determines the second visually estimated posture based on the visually estimated posture of the dexterous hand at the starting point.

[0058] Taking a continuous motion path consisting of a starting point, transition point 1, verification point 1, transition point 2, and verification point 3 in sequence, and the visually estimated posture of the dexterous hand at transition point 2 as the second visually estimated posture, the robot determines the second visually estimated posture based on the visually estimated posture of the dexterous hand at verification point 1.

[0059] As can be seen, the robot can determine the second visual estimated pose by using the starting point, verification point, or transition point adjacent to the transition point corresponding to the second visual estimated pose.

[0060] Optionally, the robot determines a second visually estimated pose based on the visually estimated pose of the path node preceding the adjacent transition point of the dexterous hand at the verification point. Specifically, the robot projects the contour points of the computer-aided design model of the dexterous hand into a template contour based on the initial pose, where the initial pose is the visually estimated pose of the path node preceding the adjacent transition point of the dexterous hand at the verification point. The robot updates the initial pose based on the difference between the template contour and the edge contour, where the edge contour is extracted from the image of the adjacent transition point of the dexterous hand at the verification point. The robot determines the updated initial pose as the second visually estimated pose.

[0061] In this approach, since the relative pose changes of adjacent path nodes in a continuous motion path are small, the robot uses the visually estimated pose of the path node preceding the adjacent transition point of the dexterous hand at the verification point as the initial pose, and iteratively adjusts it to obtain the second visually estimated pose, which can reduce the adjustment space and improve the estimation efficiency.

[0062] Implementation method 1.2 The robot determines the second visual estimated pose based on the visual estimated pose of the dexterous hand at the starting point.

[0063] Optionally, the robot determines the second visual estimated pose based on the visual estimated pose of the dexterous hand at the starting point. Specifically, the robot projects the contour points of the computer-aided design model of the dexterous hand as a template contour based on the initial pose, and the initial pose is the visual estimated pose of the dexterous hand at the starting point. The robot updates the initial pose based on the difference between the template contour and the edge contour, where the edge contour is extracted from the image of the dexterous hand at the previous adjacent transition point of the verification point. The robot determines the updated initial pose as the second visual estimated pose.

[0064] The robot's initial pose update is an iterative adjustment process based on differences. Specifically, the robot determines the difference between the template contour and the edge contour, and updates the initial pose in the direction that reduces this difference until the template contour and the edge contour match.

[0065] In this approach, the robot determines the second visual estimation pose by using contour differences, thereby improving the estimation efficiency of pose while ensuring estimation accuracy, thus balancing efficiency and robustness.

[0066] Implementation method 1.3 The robot determines the second visual estimated pose based on the visual estimated pose of the dexterous hand at the previous check point of the adjacent transition point before the check point.

[0067] Taking a continuous motion path consisting of a starting point, transition point 1, check point 1, transition point 2, transition point 3, and check point 2 in sequence, and assuming the dexterous hand's visually estimated posture at transition point 3 is the second visually estimated posture, the robot determines the second visually estimated posture based on the dexterous hand's visually estimated posture at check point 1. If the dexterous hand's visually estimated posture at transition point 1 is the second visually estimated posture, and there is no check point preceding transition point 1, the robot determines the second visually estimated posture based on the dexterous hand's visually estimated posture at the starting point.

[0068] Optionally, the robot determines a second visually estimated pose based on the visually estimated pose of the dexterous hand at the previous adjacent transition point of the verification point, including: the robot projects the contour points of the computer-aided design model of the dexterous hand as a template contour based on the initial pose, the initial pose being the visually estimated pose of the dexterous hand at the previous adjacent transition point of the verification point; the robot updates the initial pose based on the difference between the template contour and the edge contour, the edge contour being extracted from the image of the dexterous hand at the previous adjacent transition point of the verification point; the robot determines the updated initial pose as the second visually estimated pose.

[0069] In this method, the visually estimated pose of the dexterous hand at the previous verification point is obtained by the robot reconstructing 3D feature points through binocular vision triangulation, and then iteratively registering the reconstructed 3D feature points with the 3D feature points on the dexterous hand's computer-aided design model, resulting in high estimation accuracy. Therefore, the robot uses the visually estimated pose of the previous verification point as the initial pose, iteratively adjusting the initial pose to obtain the second visually estimated pose, thereby improving the estimation accuracy of the second visually estimated pose.

[0070] In one optional implementation, the robot may also refer to the above implementation 1.1, implementation 1.2 or implementation 1.3 to determine the visually estimated pose of the dexterous hand at transition points other than the transition point corresponding to the second visually estimated pose on the continuous motion path, so as to maintain dexterous hand pose tracking.

[0071] For example, the robot determines the visually estimated pose of the dexterous hand at the current transition point based on the visually estimated pose of the dexterous hand at the path node preceding the current transition point. Therefore, for a continuous motion path consisting of the starting point, transition point 1, transition point 2, check point 1, and transition point 3 in sequence, the robot determines the visually estimated pose of the dexterous hand at transition point 1 based on the visually estimated pose of the dexterous hand at the starting point; the robot determines the visually estimated pose of the dexterous hand at transition point 2 based on the visually estimated pose of the dexterous hand at transition point 1; and the robot determines the visually estimated pose of the dexterous hand at transition point 3 based on the visually estimated pose of the dexterous hand at check point 1.

[0072] It is evident that the visually estimated pose of the dexterous hand at the previous path node at the current transition point serves as prior information, providing a reliable initial value for the pose change of the dexterous hand at the current transition point, thereby reducing the adjustment space and improving estimation efficiency.

[0073] For example, the robot determines the visually estimated posture of the dexterous hand at the current transition point based on the visually estimated posture of the dexterous hand at the starting point. Therefore, for a continuous motion path consisting of the starting point, transition point 1, transition point 2, and check point 3, the robot determines the visually estimated posture of the dexterous hand at transition point 1 based on the visually estimated posture of the dexterous hand at the starting point; the robot determines the visually estimated posture of the dexterous hand at transition point 2 based on the visually estimated posture of the dexterous hand at the starting point.

[0074] It is evident that the visually estimated pose of the dexterous hand at the starting point serves as prior information, providing a reliable initial value for subsequent tracking of the dexterous hand's pose changes. This allows the robot to quickly determine the visually estimated pose of the dexterous hand at the transition point, thus maintaining dexterous hand pose tracking.

[0075] For example, the robot determines the visually estimated pose of the dexterous hand at the current transition point based on the visually estimated pose of the previous checkpoint. Therefore, for a continuous motion path consisting of the starting point, transition point 1, checkpoint 1, transition point 2, transition point 3, and checkpoint 2 in sequence, the robot determines the visually estimated pose of the dexterous hand at transition point 1 based on the visually estimated pose of the dexterous hand at the starting point; the robot determines the visually estimated pose of the dexterous hand at transition point 1 based on the visually estimated pose of the dexterous hand at checkpoint 1; and the robot determines the visually estimated pose of the dexterous hand at transition point 3 based on the visually estimated pose of the dexterous hand at checkpoint 1.

[0076] It is evident that when the robot determines the visual estimated pose of the dexterous hand at the transition point on the continuous motion path, it updates the visual estimated pose of the dexterous hand at the previous verification point as the initial pose, thereby improving the estimation accuracy of the visual estimated pose of the dexterous hand at the transition point.

[0077] Optionally, the stability of robot tracking can be improved by methods such as Kalman filtering during the tracking of dexterous hand posture changes.

[0078] The following describes a possible implementation method for a robot to determine the visually estimated pose of its dexterous hand at the initial point.

[0079] In one alternative implementation, the robot determines the visually estimated pose of the dexterous hand at the starting point, including: the robot determining two-dimensional feature points in an image of the dexterous hand at the starting point using three-dimensional feature points on a computer-aided design model of the dexterous hand; and the robot determining the visually estimated pose of the dexterous hand at the starting point based on matching pairs composed of two-dimensional and three-dimensional feature points.

[0080] In one optional implementation, the robot determines the two-dimensional feature points of the dexterous hand in the image at the starting point by using the three-dimensional feature points on the computer-aided design model of the dexterous hand. Specifically, the robot projects the three-dimensional feature points on the computer-aided design model of the dexterous hand onto the image to obtain training data, which includes bounding boxes, masks, and two-dimensional feature points. The robot trains an object detection model based on the training data. The object detection model is used to learn to detect bounding boxes, masks, and two-dimensional feature points associated with the target in the image. The robot outputs the two-dimensional feature points of the dexterous hand in the image at the starting point through the object detection model.

[0081] In this case, the two-dimensional feature points in the image of the dexterous hand at the starting point output by the robot through the target detection model have a one-to-one correspondence with the three-dimensional feature points on the computer-aided design model of the dexterous hand. Thus, the robot can obtain a matching pair composed of two-dimensional feature points and three-dimensional feature points.

[0082] In one alternative implementation, the robot determines the visually estimated pose of the dexterous hand at the starting point based on a pair of matching two-dimensional and three-dimensional feature points. Specifically, the robot calculates the visually estimated pose of the dexterous hand at the starting point using the random sample consensus (RANSAC) perspective-n-point (PnP) algorithm based on the pair of matching two-dimensional and three-dimensional feature points.

[0083] In another alternative implementation, the robot determines the visually estimated pose of the dexterous hand at the starting point by employing end-to-end 6D pose estimation. For example, the end-to-end 6D pose estimation could be a FoundationPose model.

[0084] S102. The robot reads the nominal kinematic pose of the dexterous hand, which is the theoretical pose of the dexterous hand at the check point.

[0085] In one optional implementation, the nominal kinematic pose of the robot is determined as follows: at a check point, the current joint angle values ​​of each joint of the dexterous hand are read from the robot's controller; based on the robot's kinematic model, forward kinematic calculations are performed on the current joint angle values ​​to obtain the six-dimensional pose of the dexterous hand end in the robot's base coordinate system, which is taken as the nominal kinematic pose of the robot.

[0086] In one alternative implementation, the robot's kinematic nominal pose is an end-effector six-dimensional pose estimate directly read from the robot's controller. This estimate is generated by the controller based on encoder readings from each joint and combined with internal filtering or state estimation algorithms.

[0087] In one optional implementation, the robot's nominal kinematic pose is adopted using the six-dimensional pose command value corresponding to the check point.

[0088] S103. Based on the robot's first visual pose estimation and the robot's kinematic nominal pose, determine whether the robot needs to recalibrate its hand and eye.

[0089] In one optional implementation, S103 includes: the robot determining the displacement difference and angle difference between the first visually estimated pose and the robot's nominal kinematic pose based on the first visually estimated pose and the robot's nominal kinematic pose; if the displacement difference or angle difference is greater than its corresponding first threshold, the robot determines that it needs to re-calibrate its hand-eye system; if both the displacement difference and angle difference are less than their corresponding second thresholds, the robot determines that it does not need to re-calibrate its hand-eye system.

[0090] For example, if the translation vector of the first visually estimated pose is (x1, y1, z1) and the rotation vector is (rx1, ry1, rz1), and the translation vector of the robot's nominal kinematic pose is (x2, y2, z2) and the rotation vector is (rx2, ry2, rz2), then the displacement difference between the first visually estimated pose and the robot's nominal kinematic pose refers to x1-x2, y1-y2, and z1-z2; the angle difference between the first visually estimated pose and the robot's nominal kinematic pose refers to rx1-rx2, ry1-ry2, and rz1-rz2. If any of x1-x2, y1-y2, z1-z2, rx1-rx2, ry1-ry2, or rz1-rz2 is greater than its corresponding first threshold, then the robot needs to recalibrate its hand-eye system. If all of x1-x2, y1-y2, z1-z2, rx1-rx2, ry1-ry2, and rz1-rz2 are less than their corresponding second thresholds, then the robot needs to recalibrate its hand-eye system.

[0091] Optionally, the first threshold and the second threshold may be the same or different, and this application embodiment does not limit this.

[0092] Optionally, the first threshold and the second threshold can be determined based on the inherent error of the visually estimated pose, the accuracy of the robot's nominal kinematic pose, and / or the tolerance requirements of the specific application scenario.

[0093] In another optional implementation, when there are multiple verification points, S103 includes: the robot, based on the first visual estimated pose and the robot's nominal kinematic pose of the multiple verification points, determines a first difference between the estimated displacement difference and the theoretical displacement difference between each pair of adjacent verification points, and a second difference between the estimated angle difference and the theoretical angle difference between each pair of adjacent verification points; if the robot's first difference or second difference is greater than its corresponding third threshold, it is determined that the robot needs to re-perform hand-eye calibration; if the robot's first difference and second difference are both less than their corresponding fourth thresholds, it is determined that the robot does not need to re-perform hand-eye calibration.

[0094] Taking a continuous motion path consisting of a starting point, transition point 1, check point 1, transition point 2, and check point 2 in sequence, where check point 1 and check point 2 are two adjacent check points among multiple check points, the translation vector of the dexterous hand's first visual estimated posture at check point 1 is (x1, y1, z1), and the rotation vector is (rx1, ry1, rz1); the translation vector of the dexterous hand's nominal robot kinematic posture at check point 1 is (x2, y2, z2), and the rotation vector is (rx2, ry2, rz2); the translation vector of the dexterous hand's first visual estimated posture at check point 2 is (x3, y3, z3), and the rotation vector is (rx3, ry3, rz3); the translation vector of the dexterous hand's nominal robot kinematic posture at check point 2 is (x4, y4, z4), and the rotation vector is (rx4, ry4, rz4). Therefore, the first difference between the estimated displacement difference and the theoretical displacement difference between verification point 1 and verification point 2 is (x1-x2)-(x3-x4), (y1-y2)-(y3-y4) and (z1-z2)-(z3-z4); the second difference between the estimated angle difference and the theoretical angle difference between verification point 1 and verification point 2 is (rx1-rx2)-(rx3-rx4), (ry1-y2)-(ry3-ry4) and (rz1-rz2)-(rz3-rz4). Thus, if any of the following values ​​is greater than the corresponding third threshold, the robot needs to recalibrate its hand-eye alignment; if the following values ​​are less than the corresponding fourth threshold, the robot needs to recalibrate its hand-eye alignment.

[0095] Optionally, the third threshold and the fourth threshold may be the same or different, and this application embodiment does not limit this.

[0096] Optionally, the third and fourth thresholds can be determined based on the inherent error of the visually estimated pose, the accuracy of the robot's nominal kinematic pose, and / or the tolerance requirements of the specific application scenario.

[0097] As can be seen, the embodiments of this application construct a closed-loop logic for dexterous hand tracking, calibration verification, and verification decision-making to reduce the complexity of hand-eye calibration accuracy verification. Specifically: the robot locates the position of the dexterous hand in the image at the starting point through target detection, and then determines the visually estimated pose of the dexterous hand at the starting point, using it as the initial value for dexterous hand tracking; the robot continuously tracks the pose changes of the dexterous hand in time based on contour features to maintain the pose estimation; at the verification point, the robot uses binocular vision triangulation to obtain a higher-precision visually estimated pose, eliminating the accumulated error of time-series tracking; the robot compares the visually estimated pose of the dexterous hand at the verification point with the robot's kinematic nominal pose to determine whether hand-eye calibration needs to be recalibrated. In this way, there is no need for special targets such as calibration boards and concentric circle stickers; the robot can complete the verification of hand-eye calibration accuracy during normal operation, thereby reducing the complexity of verification and saving downtime and labor costs.

[0098] Furthermore, since this embodiment eliminates the need for dedicated targets such as calibration plates and concentric circle stickers, it reduces the ongoing additional costs associated with purchasing, replacing, and maintaining dedicated targets. It also minimizes damage to precision end-effectors such as dexterous hands caused by repeated installation and disassembly. Because installing dedicated targets requires a high level of operator skill, this embodiment also lowers the operational threshold for hand-eye calibration accuracy verification, facilitating wider application.

[0099] Furthermore, the embodiments of this application are applicable not only to calibration and verification scenarios but also to robot teaching and motion compensation scenarios. In other words, the robot can use the implementation methods described in these embodiments to calculate the difference between the estimated posture and the theoretical posture of the dexterous hand in different regions, and incorporate this difference as a compensation amount into the teaching trajectory, thereby improving absolute positioning accuracy.

[0100] The formulas used in the embodiments of this application are described below.

[0101] 1. Projection of 3D feature points The robot converts the three-dimensional feature points in the world coordinate system into three-dimensional feature points in the camera coordinate system using the following formula (1).

[0102] (1).

[0103] in, Represents three-dimensional feature points in the world coordinate system; Represents three-dimensional feature points in the camera coordinate system; Represents the rotation matrix; This represents the translation vector.

[0104] The robot projects the three-dimensional feature points in the camera coordinate system onto the image using the following formula (2) to obtain the corresponding two-dimensional pixel coordinates.

[0105] (2).

[0106] in, Represents the coordinate components of a 3D feature point along the x-axis in the camera coordinate system; Represents the coordinate components of a 3D feature point along the y-axis in the camera coordinate system; Represents the coordinate components of a 3D feature point along the z-axis in the camera coordinate system; Represents the camera intrinsic parameter matrix; and represents the two-dimensional pixel coordinates of the three-dimensional feature points on the image; 1 represents the normalization term for the homogeneous coordinates; It represents depth information.

[0107] 2. Triangulation 2.1 Basic Projection Formulas The robot determines the three-dimensional homogeneous space point in the world coordinate system using the following formula (3).

[0108] (3).

[0109] in, Represents a point in three-dimensional homogeneous space in the world coordinate system; Represents the coordinate components of a point in three-dimensional homogeneous space along the x-axis in the world coordinate system; Represents the coordinate components of a point in three-dimensional homogeneous space along the y-axis in the world coordinate system; Represents the coordinate components of a point in three-dimensional homogeneous space along the z-axis in the world coordinate system; This represents the homogeneous coordinate components.

[0110] The robot determines the projection matrix using the following formula (4).

[0111] (4).

[0112] in, Represents the projection matrix; Represents the camera intrinsic parameter matrix; It is a rotation matrix; It is a translation vector.

[0113] The robot expresses the relationship between the projection matrix and the rows of the projection matrix using the following formula (5).

[0114] (5).

[0115] in, Represents the projection matrix of the i-th camera (or the i-th view); express The first row vector; express The second row vector; express The third row vector.

[0116] 2.2 Formula for Constructing a Single Matching Point The robot is determining the pixel points points in three-dimensional homogeneous space In the case of matching, the geometric relationship between the two is constrained by the following formula (6), and formula (7) is obtained by rearranging formula (6).

[0117] (6).

[0118] (7).

[0119] in, and Represents two-dimensional pixel coordinates; Represents a point in three-dimensional homogeneous space in the world coordinate system; This represents the row vector of the first row of the projection matrix; This represents the row vector of the second row of the projection matrix; This represents the row vector of the 3rd row of the projection matrix.

[0120] 2.3 Two Views Figure 3 Angular construction matrix Assume the projection matrix of camera 1 is Pixel 1 is The projection matrix of camera 2 is Pixel 2 is The robot constructs the matrix using the following formula (8). .

[0121] (8).

[0122] in, Represents the coefficient matrix; express It is a 4×4 matrix; and Represents the pixel coordinates on the image of camera 1; and Indicates the pixel coordinates on the image from camera 2; Represents the projection matrix The first row vector; Represents the projection matrix The second row vector; Represents the projection matrix The third row vector; Represents the projection matrix The first row vector; Represents the projection matrix The second row vector; Represents the projection matrix The third row vector.

[0123] The robot constructs equations using formula (9).

[0124] (9).

[0125] in, Represents the coefficient matrix; It represents a point in three-dimensional homogeneous space in the world coordinate system.

[0126] 2.4 General for n views Matrix (Multi-point matching triangulation) Assume the projection matrix of camera 1 is Pixel 1 is The projection matrix of camera 2 is Pixel 2 is The projection matrix of camera n is , pixel n is The robot constructs the matrix using the following formula (10). .

[0127] (10).

[0128] in, Represents the coefficient matrix; express It is a 2n×4 matrix; and Represents the pixel coordinates on the image of camera 1; and Indicates the pixel coordinates on the image from camera 2; and Represents the pixel coordinates on the image of camera n; Represents the projection matrix The first row vector; Represents the projection matrix The second row vector; Represents the projection matrix The third row vector; Represents the projection matrix The first row vector; Represents the projection matrix The second row vector; Represents the projection matrix The third row vector; Represents the projection matrix The first row vector; Represents the projection matrix The second row vector; Represents the projection matrix The third row vector.

[0129] 2.5 Singular Value Decomposition (SVD) Solution Formula The robot uses the following formula (11) to... Perform singular value decomposition.

[0130] (11).

[0131] in, Represents the coefficient matrix; Represents a left singular matrix; Represents a singular value diagonal matrix; Represents a right singular matrix; This represents the transpose of a right singular matrix.

[0132] The robot will use the following formula (12) to... The last column is a homogeneous solution.

[0133] (12).

[0134] in, Represents a point in three-dimensional homogeneous space in the world coordinate system; express The last column; Represents the coordinate components of a point in three-dimensional homogeneous space along the x-axis in the world coordinate system; Represents the coordinate components of a point in three-dimensional homogeneous space along the y-axis in the world coordinate system; This represents the coordinate components of a point in three-dimensional homogeneous space along the z-axis in the world coordinate system.

[0135] The robot obtains three-dimensional Euclidean coordinates using the following formula (13).

[0136] (13).

[0137] in, Represents three-dimensional Euclidean coordinates; Represents the coordinate components of a point in three-dimensional homogeneous space along the x-axis in the real world; Represents the coordinate components of a point in three-dimensional homogeneous space along the y-axis in the real world; This represents the coordinate components of a point in three-dimensional homogeneous space along the z-axis in the real world.

[0138] 3. ICP 3.1 Basic Definitions Define the source point set (to be aligned) as follows: .in, This represents the set of three-dimensional feature point clouds currently observed by the robot; Represents the three-dimensional coordinate vector of the i-th point in the source point cloud; express It is a 3×1 matrix.

[0139] Define the target point set (reference) as .in, Represents the target reference point cloud set; This represents the three-dimensional coordinate vector of the j-th point in the target point cloud; express It is a 3×1 matrix.

[0140] ICP process requires solution transformation: rotation matrix Translation vector .

[0141] The function to minimize the matching point error is .in, This represents a function that minimizes the matching point error. Indicates the source point The new position after rotation and translation; This represents the target point that matches the transformed source point. This represents the square of the Euclidean distance.

[0142] 3.2 ICP Process 3.2.1 Corresponding point matching The robot applies the following formula (14) to each transformed source point Find the point with the closest Euclidean distance in the target point set as the match. .

[0143] (14).

[0144] in, Indicates the source point after transformation; Represents the target point set Any point in the array; arg min represents any point in the array. In the process, select the point with the closest Euclidean distance and determine it as the current matching pair. .

[0145] 3.2.2 Decentralization The robot determines the centroid of the source point set using the following formula (15).

[0146] (15).

[0147] in, Indicates the centroid of the source set; Represents the three-dimensional coordinate vector of the i-th point in the source point cloud; Indicates the total number.

[0148] The robot determines the centroid of the target center point using the following formula (16).

[0149] (16).

[0150] in, Indicate the centroid of the target point set; Represents the three-dimensional coordinate vector of the i-th point in the target point cloud; Indicates the total number.

[0151] The robot obtains the decentered coordinates using the following formulas (17) and (18).

[0152] (17).

[0153] (18).

[0154] in, express The three-dimensional coordinate vector after centroid removal; express The three-dimensional coordinate vector after centroid removal.

[0155] The robot simplifies the function of minimizing the matching point error to obtain the following formula (19).

[0156] (19).

[0157] in, This represents a function that minimizes the matching point error. express The three-dimensional coordinate vector after centroid removal; express The three-dimensional coordinate vector after centroid removal; This represents the rotation matrix.

[0158] 3.2.3 Construct the covariance matrix and use SVD to solve for the optimal rotation matrix. The robot solves the covariance matrix using the following formula (20).

[0159] (20).

[0160] in, Represent the covariance matrix; express The three-dimensional coordinate vector after centroid removal; express The three-dimensional coordinate vector after centroid removal; express Transpose of the three-dimensional coordinate vector after centroid removal.

[0161] The robot is based on the following formula (21) Perform SVD decomposition.

[0162] (twenty one).

[0163] in, Represent the covariance matrix; Represents a left singular matrix; Represents a singular value diagonal matrix; Represents a right singular matrix; This represents the transpose of a right singular matrix.

[0164] The robot determines the optimal rotation matrix using the following formula (22). .

[0165] (twenty two).

[0166] in, Represents the optimal rotation matrix; This represents the transpose of the left singular matrix; Represents a right singular matrix; express The determinant value.

[0167] 3.2.4 Solving for the optimal translation vector The robot is determined by the following formula (23). .

[0168] (twenty three).

[0169] in, Represents the optimal translation vector; Indicate the centroid of the target point set; Indicates the centroid of the source set; This represents the optimal rotation matrix.

[0170] 3.2.5 Iterative Updates and Convergence Judgment The robot updates the source point set using the following formula (24).

[0171] (twenty four).

[0172] in, Represents the optimal rotation matrix; Represents the optimal translation vector; This represents the three-dimensional coordinate vector of the i-th point in the source point cloud.

[0173] After updating the origin set, the robot can repeat the above steps and iterate until the convergence condition or the preset maximum number of iterations is met, and then proceed based on the final result. , Determine the rotation and translation vectors to determine the 6D pose, which can be... Figure 1 The first visually estimated pose in the embodiment.

[0174] Figure 2 In another verification method provided in this application, the robot first needs to determine whether the continuous motion path has been saved. If the continuous motion path has been saved and imported, the robot determines the visual estimated posture of the dexterous hand at the starting point and tracks the posture of the dexterous hand. Thus, the robot can verify the hand-eye calibration accuracy at the verification point based on the difference between the visual estimated posture of the dexterous hand at the verification point and the nominal kinematic posture of the robot, so as to determine whether the hand-eye calibration needs to be re-performed.

[0175] S201. The robot determines whether the continuous motion path has been saved.

[0176] Optionally, if the robot has saved the continuous motion path, it can directly execute S203; if the robot has not saved the continuous motion path, it can execute S202 first and then S203.

[0177] Optionally, an exemplary description of a continuous motion path can be found in [reference needed]. Figure 1 The relevant content in S101 of the above embodiment will not be repeated here.

[0178] S202. The robot generates a continuous motion path based on the starting point received from the control interface and one or more verification points.

[0179] In one optional implementation, S202 includes: the robot receiving a starting point and one or more verification points input by the operator through a control interface; the robot sorting the starting point and the one or more verification points to obtain a path node sequence; the robot determining multiple transition points between every two adjacent nodes in the path node sequence by interpolation; and the robot generating a continuous motion path based on the path node sequence and the multiple transition points.

[0180] Optionally, exemplary descriptions of starting points, checkpoints, and transition points can be found in [reference needed]. Figure 1 The relevant content in S101 of the above embodiment will not be repeated here.

[0181] S203. The robot imports a continuous motion path.

[0182] S204. The robot determines the visually estimated pose of the dexterous hand at the starting point.

[0183] Optionally, possible implementations for the robot to determine the visually estimated pose of the dexterous hand at the starting point can be found in [reference needed]. Figure 1 The relevant content in S101 of the above embodiment will not be repeated here.

[0184] S205. The robot performs posture tracking on the dexterous hand.

[0185] Optionally, possible implementations of robot posture tracking of a dexterous hand can be found in [reference needed]. Figure 1 The relevant content in S101 of the above embodiment will not be repeated here.

[0186] S206. The robot determines the visually estimated pose of the dexterous hand at the check point.

[0187] Optionally, possible implementations for the robot to determine the dexterous hand at the verification point can be found in [reference needed]. Figure 1 The relevant content in S101 of the above embodiment will not be repeated here.

[0188] S207. The robot determines whether the hand-eye calibration accuracy meets the conditions.

[0189] Optionally, the robot executes S208 if the hand-eye calibration accuracy does not meet the requirements; and executes S209 if the hand-eye calibration accuracy meets the requirements.

[0190] In one optional implementation, S207 includes: the robot reading the robot's nominal kinematic pose of the dexterous hand at the calibration point; and the robot determining whether the hand-eye calibration accuracy meets the conditions based on the visually estimated pose of the dexterous hand at the calibration point and the robot's nominal kinematic pose.

[0191] Optionally, the hand-eye calibration accuracy not meeting the condition can be understood as: the difference between the visually estimated pose and the robot's nominal kinematic pose is greater than a threshold; the hand-eye calibration accuracy meeting the condition can be understood as: the difference between the visually estimated pose and the robot's nominal kinematic pose is less than a threshold. For a single calibration point, this difference can be... Figure 1 In the embodiment described, the displacement difference or angle difference in S103; for multiple verification points, this difference can be... Figure 1 The first difference or the second difference in S103 in the embodiment.

[0192] Optionally, possible implementations for the robot to determine whether the hand-eye calibration accuracy meets the conditions can be found in [reference needed]. Figure 1 The relevant content in S103 of the above embodiment will not be repeated here.

[0193] S208. The robot recalibrates its hand and eye.

[0194] S209. The robot completes the verification of the accuracy of the hand-eye calibration.

[0195] Figure 3 This is a schematic diagram of a robot device provided in an embodiment of this application. The robot device includes at least a processing unit 310 and a reading unit 320.

[0196] In one alternative approach, the processing unit 310 is used to determine the first visual estimated pose of the robot's dexterous hand. The first visual estimated pose is the estimated pose of the dexterous hand at the verification point. The first visual estimated pose is obtained by three-dimensional reconstruction based on the feature information determined by the second visual estimated pose. The second visual estimated pose is the estimated pose of the dexterous hand at the previous adjacent transition point of the verification point. The reading unit 320 is used to read the robot kinematic nominal pose of the dexterous hand, which is the theoretical pose of the dexterous hand at the verification point. The processing unit 310 is also used to determine whether the robot needs to recalibrate its hand and eye based on the first visual estimation of the pose and the robot's kinematic nominal pose.

[0197] Optionally, the processing unit 310 determines the first visual estimation pose of the robot's dexterous hand, specifically: The processing unit 310 estimates the pose based on the second vision and projects the three-dimensional feature points on the computer-aided design model of the dexterous hand into the left and right eye images respectively, to obtain the two-dimensional feature points of the left eye image and the two-dimensional feature points of the right eye image. The three-dimensional feature points are obtained through structural downsampling. The processing unit 310 triangulates the cross-view matching pairs composed of two-dimensional feature points from the left eye image and two-dimensional feature points from the right eye image to obtain the reconstructed three-dimensional feature points. The processing unit 310 registers the reconstructed 3D feature points with each other using an iterative nearest-point algorithm to obtain the first visual estimated pose.

[0198] Optionally, the second visually estimated pose is determined based on the visually estimated pose of the dexterous hand at the starting point. The processing unit 310 is further configured to perform the following steps to determine the visually estimated pose of the dexterous hand at the starting point: Processing unit 310 determines the two-dimensional feature points of the dexterous hand in the image of the starting point by using the three-dimensional feature points on the computer-aided design model of the dexterous hand; The processing unit 310 determines the visually estimated pose of the dexterous hand at the starting point based on matching pairs composed of two-dimensional feature points and three-dimensional feature points.

[0199] Optionally, the processing unit 310 is also configured to perform the following steps to determine the second visual estimated pose: The processing unit 310 projects the contour points of the computer-aided design model of the dexterous hand into a template contour based on the initial posture. The initial posture is the visually estimated posture of the path node of the adjacent transition point before the verification point of the dexterous hand. Processing unit 310 updates the initial pose based on the difference between the template contour and the edge contour, whereby the edge contour is extracted from the image of the dexterous hand at the previous adjacent transition point of the verification point. The processing unit 310 determines the updated initial pose as the second visual estimated pose.

[0200] Optionally, the processing unit 310 determines whether the robot needs to re-calibrate its hand-eye relationship based on the first visual estimation of the pose and the robot's nominal kinematic pose. Specifically: The processing unit 310 determines the displacement difference and angle difference between the first visual estimated pose and the robot's kinematic nominal pose based on the first visual estimated pose and the robot's kinematic nominal pose. If the displacement difference or angle difference is greater than its corresponding first threshold, the processing unit 310 determines that the robot needs to recalibrate its hand and eye. When the displacement difference and angle difference are both less than their corresponding second thresholds, the processing unit 310 determines that the robot does not need to recalibrate its hand and eye.

[0201] Optionally, when there are multiple verification points, the processing unit 310 determines whether the robot needs to re-calibrate its hand-eye relationship based on the first visual estimation of the pose and the robot's nominal kinematic pose. Specifically: Based on the first visual estimated pose and the robot's kinematic nominal pose of multiple verification points, the processing unit 310 determines the first difference between the estimated displacement difference and the theoretical displacement difference between each two adjacent verification points, and the second difference between the estimated angle difference and the theoretical angle difference between each two adjacent verification points. If the first difference or the second difference is greater than its corresponding third threshold, the processing unit 310 determines that the robot needs to re-calibrate its hand and eye. If both the first difference and the second difference are less than their corresponding fourth threshold, the processing unit 310 determines that the robot does not need to recalibrate its hand and eye.

[0202] Optionally, a check point is a path node on a continuous motion path that the dexterous hand needs to stop at and to determine the visually estimated posture of the dexterous hand at that check point. The continuous motion path includes a starting point, a check point, and a transition point. The starting point is the initial path node on the continuous motion path, the check point is received through the robot's control interface, and the transition point is obtained through interpolation.

[0203] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device includes at least a processor 410, a user interface 420, and a memory 430, which are interconnected. The memory 430 is used to store computer programs, which include program instructions, and the processor 410 is used to execute the program instructions.

[0204] The memory 430 may include volatile memory, such as random-access memory (RAM); the memory 430 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; the memory 430 may also include a combination of the above types of memory.

[0205] Processor 410 may be a central processing unit (CPU). Processor 410 may further include hardware chips. The aforementioned hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc. The aforementioned PLDs may be field-programmable gate arrays (FPGAs), generic array logic (GALs), etc.

[0206] In an optional implementation, memory 430 is also used to store program instructions. Processor 410 can invoke program instructions: processor 410 is configured to invoke program instructions to execute the relevant content of the above method embodiments, which will not be repeated here.

[0207] The embodiments of this application and the method embodiments shown above are based on the same concept and have the same technical effects. For the specific principles, please refer to the description of the embodiments shown above, which will not be repeated here.

[0208] This application also provides a computer-readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed on a robot device, the robot device performs the steps of a verification method as described in the above embodiments.

[0209] The embodiments of the computer-readable storage medium provided in this application include all the technical features of the above-described verification method embodiments. The extended and explained contents of the specification are basically the same as the embodiments of the above method, and will not be repeated here.

[0210] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the methods described in the various possible implementations above.

[0211] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0212] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0213] The units in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0214] In this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions are generally described in detail only when they appear for the first time. When they appear again, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions that are not described in detail later can be referred to their previous relevant detailed descriptions.

[0215] In this application, the descriptions of the various embodiments have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0216] The technical features of the present application can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0217] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause the robot device to execute the methods of each embodiment of this application.

[0218] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, storage disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., SSD), etc.

[0219] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A verification method, characterized in that, The method includes: Determine the first visual estimated pose of the robot's dexterous hand. The first visual estimated pose is the estimated pose of the dexterous hand at the check point. The first visual estimated pose is obtained by three-dimensional reconstruction based on the feature information determined by the second visual estimated pose. The second visual estimated pose is the estimated pose of the dexterous hand at the previous adjacent transition point of the check point. Read the robot kinematic nominal pose of the dexterous hand, which is the theoretical pose of the dexterous hand at the verification point; Based on the first visually estimated pose and the robot's nominal kinematic pose, determine whether the robot needs to undergo re-calibration of its hand and eye. The determination of the robot's dexterous hand's first visual estimation pose includes: Based on the second visual estimation pose, the three-dimensional feature points on the computer-aided design model of the dexterous hand are projected into the left and right eye images respectively to obtain the two-dimensional feature points of the left eye image and the two-dimensional feature points of the right eye image. The three-dimensional feature points on the computer-aided design model of the dexterous hand are obtained through structural downsampling. The cross-view matching pairs composed of two-dimensional feature points from the left eye image and two-dimensional feature points from the right eye image are triangulated to obtain the reconstructed three-dimensional feature points. By using an iterative nearest-point algorithm, the reconstructed 3D feature points are registered with the 3D feature points on the computer-aided design model of the dexterous hand to obtain the first visually estimated pose. And wherein, the second visually estimated pose is determined based on the visually estimated pose of the dexterous hand at the starting point, and determining the visually estimated pose of the dexterous hand at the starting point includes: Using the three-dimensional feature points on the computer-aided design model of the dexterous hand, determine the two-dimensional feature points of the dexterous hand in the image at the starting point; Based on the matching pairs consisting of the two-dimensional feature points and the three-dimensional feature points on the computer-aided design model of the dexterous hand, the visually estimated pose of the dexterous hand at the starting point is determined.

2. The method according to claim 1, characterized in that, Determining the second visually estimated pose includes: Based on the initial pose, the contour points of the computer-aided design model of the dexterous hand are projected as the template contour. The initial pose is the visually estimated pose of the dexterous hand at the path node preceding the adjacent transition point of the verification point. Based on the difference between the template contour and the edge contour, the initial pose is updated, wherein the edge contour is extracted from the image of the dexterous hand at the previous adjacent transition point of the verification point; The updated initial pose is determined as the second visually estimated pose.

3. The method according to claim 1 or 2, characterized in that, The step of determining whether the robot needs to undergo re-calibration of its hands and eyes based on the first visually estimated pose and the robot's nominal kinematic pose includes: Based on the first visually estimated pose and the robot's kinematic nominal pose, determine the displacement difference and angle difference between the first visually estimated pose and the robot's kinematic nominal pose; If the displacement difference or the angle difference is greater than its corresponding first threshold, it is determined that the robot needs to recalibrate its hand and eye. If both the displacement difference and the angle difference are less than their corresponding second thresholds, it is determined that the robot does not need to be recalibrated.

4. The method according to claim 1 or 2, characterized in that, When there are multiple verification points, determining whether the robot needs to undergo hand-eye calibration again based on the first visually estimated pose and the robot's nominal kinematic pose includes: Based on the first visual estimated pose and the robot's kinematic nominal pose of the multiple verification points, a first difference between the estimated displacement difference and the theoretical displacement difference between each pair of adjacent verification points is determined, as well as a second difference between the estimated angle difference and the theoretical angle difference between each pair of adjacent verification points. If either the first difference or the second difference is greater than its corresponding third threshold, it is determined that the robot needs to recalibrate its hand-eye relationship. If both the first difference and the second difference are less than their corresponding fourth threshold, it is determined that the robot does not need to recalibrate its hand and eye.

5. The method according to claim 1, characterized in that, The verification point is a path node where the dexterous hand needs to stop on the continuous movement path and where the visual estimated posture of the dexterous hand at the verification point is determined; The continuous motion path includes a starting point, a check point, and a transition point. The starting point is the initial path node on the continuous motion path. The check point is received through the robot's control interface, and the transition point is obtained through interpolation.

6. A robotic device, characterized in that, The robotic device includes a unit for performing the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed on the robot device, cause the robot device to perform the method as described in any one of claims 1 to 5.

8. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method as described in any one of claims 1 to 5.

Images