Robotic hand-eye calibration method, system, device and medium for multiplexed vision pipeline

Abstract

The present application relates to the technical field of robot hand-eye calibration, and particularly relates to a mechanical arm hand-eye calibration method, system, device and medium of multiplexing vision pipeline, the method comprising: presetting calibration point positions, recording mechanical arm coordinates of each calibration point position under a mechanical arm base coordinate system and corresponding text prompts; placing objects in each calibration point position in turn and triggering confirmation, collecting time-synchronized color images and depth images; calling a vision processing pipeline to process the objects at each calibration point position, obtaining top surface thin layer point clouds of the objects; calculating camera coordinates of calibration reference points according to calibration special aggregation rules, so that the calibration reference points are located at the center of the top surface of the objects; adding the camera coordinates of the calibration reference points and the mechanical arm coordinates of the corresponding calibration point positions as a pair into a point pair set; after traversing all the calibration point positions, performing rigid body registration on the point pair set, and solving a homogeneous transformation matrix from the camera coordinate system to the mechanical arm base coordinate system.

Description

Technical Field

[0001] This invention relates to the field of robot hand-eye calibration technology, and in particular to a method, system, device and medium for robotic arm hand-eye calibration that reuses vision pipelines. Background Technology

[0002] Hand-eye calibration is a necessary calibration step before a robot vision grasping system is put into production. Its purpose is to determine the rigid body transformation relationship (rotation matrix R and translation vector t) between the camera coordinate system and the robot arm base coordinate system, converting visual perception results into executable motion coordinates for the robot arm. In the eye-to-hand configuration, the camera is fixedly mounted on a bracket above the working area and does not move with the end effector. The homogeneous transformation matrix obtained from calibration remains unchanged during system operation and is a crucial factor in determining the accuracy of the entire vision grasping chain.

[0003] Existing hand-eye calibration methods are mainly classified into three categories based on reference standards: (i) based on calibration targets (checkerboard, ArUco, AprilTag, etc.) + corner detection / perspective-n-point (PnP) calculation (OpenCVcalibrateHandEye, etc.); (ii) known 3D points + singular value decomposition (SVD) or quaternion rigid body registration (Arun 1987, Horn 1987); (iii) manual teaching method, in which the operator manually guides the end effector of the robotic arm to touch several reference positions on the worktable and records the coordinate correspondence. The accuracy is highly dependent on the operator's skill and the consistency is poor. All of the above methods rely on special tools or additional equipment, and the vision processing flow used for calibration (corner detection, center fitting) is fundamentally different from the vision flow (segmentation, point cloud processing) during production grasping.

[0004] Relying on calibration targets has problems such as manufacturing precision, fragility and loss, and sensitivity to occlusion; moreover, the visual processing algorithm of the calibration process is inconsistent with the production grasping task, and the accuracy measured by calibration does not directly represent the accuracy of production grasping.

[0005] When a regular object is used instead of a calibration target, if the aggregation rule of "taking the arithmetic mean of XYZ" in the grabbing scene is followed, the reference point will fall in the middle of the upper half of the object instead of the center of the top surface. It is difficult for the operator to verify with the naked eye, and the height is affected by the selection of the thickness of the top surface slice.

[0006] The information disclosed in this background section is intended only to enhance the understanding of the general background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0007] This invention provides a robotic arm hand-eye calibration method, system, device, and medium that reuses vision pipelines, thereby effectively solving the problems in the prior art.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is: a robotic arm hand-eye calibration method that reuses vision pipelines, comprising the following steps:

[0009] Preset at least 4 calibration points that are not all located on the same plane, and record the robot arm coordinates and corresponding text prompts for each calibration point in the robot arm base coordinate system;

[0010] The object is placed sequentially at each of the specified calibration points and confirmation is triggered to collect time-synchronized color and depth maps.

[0011] The same vision processing pipeline as the production capture task is invoked to process the object at each calibration point to obtain a thin point cloud of the top surface of the object.

[0012] The camera coordinates of the calibration reference point are calculated for the thin point cloud on the top surface according to the calibration-specific aggregation rules: the arithmetic mean of the two horizontal coordinate components is taken, and the minimum value of the vertical coordinate component is taken, so that the calibration reference point is located at the center of the top surface of the object.

[0013] The camera coordinates of the calibration reference point and the robot coordinates of the corresponding calibration point are added as a pair to the point pair set; after traversing all calibration points, rigid body registration is performed on the point pair set, and the homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system is solved.

[0014] Furthermore, the at least four calibration points, which are not all located on the same plane, are distributed at at least two different height levels and are evenly distributed within the workspace accessible to the robotic arm.

[0015] The non-coplanarity of calibration points is determined by calculating the volume of the convex hull formed by all calibration points. If the volume of the convex hull is greater than a preset threshold, it is determined to be non-coplanar.

[0016] Furthermore, the object is a common rigid object with divisible appearance features and an approximately continuous curved surface on its top surface, requiring no special calibration target or marker.

[0017] Furthermore, the vision processing pipeline includes:

[0018] Based on the text prompt, a binary mask is generated from the color image using a multimodal segmentation model. The target point cloud is obtained by back-projecting the depth image using the binary mask and depth validity as joint filtering conditions. The top surface thin layer point cloud is extracted along the optical axis using the nearest point depth plus a fixed layer thickness as the slicing threshold.

[0019] Furthermore, the vision processing pipeline is the same as the vision processing pipeline used to locate and grasp targets in the production grasping task, and they share the multimodal segmentation model parameters, back projection parameters, and top surface thin layer slicing parameters.

[0020] Furthermore, the calibration-specific aggregation rule and the crawling aggregation rule used in the production crawling task are consistent in the horizontal direction, both taking an arithmetic mean, but are distinguished in the vertical direction:

[0021] The calibration-specific aggregation rule takes the minimum value in the vertical direction to ensure that the calibration reference point is strictly located at the highest point of the object.

[0022] The gripping aggregation rule takes the arithmetic mean in the vertical direction to place the gripping reference point in the middle of the upper half of the object so that the gripper can hold it stably.

[0023] Both aggregation rules share the same top surface thin-layer point cloud as input.

[0024] Furthermore, the rigid body registration employs a singular value decomposition algorithm, including:

[0025] Remove the centroids of the set of point pairs to obtain zero-mean point pairs, construct the cross-covariance matrix, and perform singular value decomposition on the cross-covariance matrix to obtain the rotation matrix. ,when The last column of the right singular vector matrix is ​​inverted and R is recalculated to eliminate the reflection solution. Then, the translation vector t is calculated from the centroid and combined into a 4×4 homogeneous transformation matrix.

[0026] Furthermore, the method also includes:

[0027] The camera coordinates of all calibration points are transformed to the robotic arm coordinate system using the homogeneous transformation matrix, and the root mean square error between the transformation result and the corresponding robotic arm coordinates is calculated as the registration residual.

[0028] When the registration residual is greater than a preset threshold, the operator is prompted to check the acquisition quality of the calibration points and re-execute the calibration process;

[0029] When the registration residual is not greater than the preset threshold, the homogeneous transformation matrix and the registration residual are persistently saved.

[0030] Furthermore, the preset threshold is 3 to 5 millimeters; the method also includes a real-machine verification step:

[0031] The verification object is placed at a non-calibrated location. The camera coordinates of the verification object are calculated through the vision processing pipeline and transformed into the robot arm coordinate system through the homogeneous transformation matrix. The robot arm then moves to the actual coordinates and the deviation between the end of the robot arm and the center of the top surface of the verification object is visually observed to intuitively confirm the calibration quality.

[0032] The present invention also includes a robotic arm hand-eye calibration system that reuses vision pipelines, using the method described above, the system comprising:

[0033] The camera is fixedly mounted above the working area of ​​the robotic arm to collect time-synchronized color and depth maps.

[0034] A robotic arm is used to perform movements based on the camera coordinates and the robotic arm coordinates of a calibration reference point.

[0035] The main control computer is used to preset calibration points and record the coordinates of the robotic arm and text prompts. It calls the same vision processing pipeline as the production grasping task to process the object image to obtain a thin point cloud on the top surface. It calculates the camera coordinates of the calibration reference point using a calibration-specific aggregation rule and performs rigid body registration on the point pair set to solve the homogeneous transformation matrix. The calibration-specific aggregation rule is to take the arithmetic mean of the horizontal coordinate components and the minimum value of the vertical coordinate components.

[0036] The present invention also includes a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described above.

[0037] The present invention also includes a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described above.

[0038] The beneficial effects of this invention are as follows: Ordinary objects replace calibration targets; object recognition is achieved through multimodal segmentation driven by text prompts; calibration and production grasping share the same vision pipeline, eliminating the need for dedicated calibration targets and reducing systematic deviations between calibration and production; the aggregation rules for calibration reference points and grasping reference points are intentionally differentiated: calibration uses "average horizontal value + highest vertical point" to make the reference point closer to the center of the object's top surface; grasping uses "average XYZ values" to place the reference point in the center of the object for easier gripping; both share the same thin-layer point cloud of the top surface as input. When ordinary objects are used as calibration references, operators can visually verify the consistency between the reference point and the center of the object's top surface; the tolerance for occlusion and lighting changes is better than corner detection. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1This is a flowchart of the method in Embodiment 1 of the present invention;

[0041] Figure 2 This is a flowchart of the method in Embodiment 2 of the present invention;

[0042] Figure 3 This is a schematic diagram comparing the aggregation rules for calibration and capture in Embodiment 2 of the present invention;

[0043] Figure 4 This is a schematic diagram of the computer device in Embodiment 3 of the present invention. Detailed Implementation

[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0045] Example 1:

[0046] like Figure 1 As shown: A method for hand-eye calibration of a robotic arm that reuses a vision pipeline, comprising the following steps:

[0047] Preset at least 4 calibration points that are not all located on the same plane, and record the robot arm coordinates and corresponding text prompts for each calibration point in the robot arm base coordinate system;

[0048] The object is placed sequentially at each calibration point and confirmation is triggered to collect time-synchronized color and depth maps;

[0049] The same vision processing pipeline as the production grabbing task is invoked to process the objects at each calibration point to obtain a thin point cloud of the top surface of the object.

[0050] The camera coordinates of the calibration reference point are calculated for the thin point cloud on the top surface according to the calibration-specific aggregation rules: the arithmetic mean of the two horizontal coordinate components is taken, and the minimum value of the vertical coordinate component is taken, so that the calibration reference point is located at the center of the top surface of the object.

[0051] The camera coordinates of the calibration reference point and the robot coordinates of the corresponding calibration point are added as a pair to the point pair set; after traversing all calibration points, rigid body registration is performed on the point pair set, and the homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system is solved.

[0052] Using ordinary objects instead of calibration targets, object recognition is achieved through multimodal segmentation driven by text prompts. By sharing the same vision pipeline between calibration and production grasping, a dedicated calibration target is unnecessary, reducing systematic deviations between calibration and production. The aggregation rules for calibration reference points and grasping reference points are intentionally differentiated: calibration uses "average horizontal value + highest vertical point" to make the reference point closer to the center of the object's top surface; grasping uses "average XYZ values" to place the reference point in the center of the object for easier gripping. Both use the same thin-layer point cloud of the top surface as input. When ordinary objects are used as calibration references, operators can visually verify the consistency between the reference point and the object's top surface center. The system exhibits better tolerance to occlusion and lighting variations than corner detection.

[0053] Among them, at least four calibration points, not all located on the same plane, are distributed at at least two different height levels and are evenly distributed within the workspace accessible to the robotic arm;

[0054] The non-coplanarity of calibration points is determined by calculating the volume of the convex hull formed by all calibration points. If the volume of the convex hull is greater than a preset threshold, it is determined to be non-coplanar.

[0055] As a preferred embodiment of the above, the object is a common rigid object with divisible appearance features and an approximately continuous curved surface on the top surface, without the need for a specially made calibration target or marker.

[0056] In this embodiment, the vision processing pipeline includes:

[0057] Based on the text prompts, a binary mask is generated from the color image using a multimodal segmentation model. The target point cloud is obtained by back-projecting the depth map using the binary mask and depth validity as joint filtering conditions. The top surface thin layer point cloud is extracted along the optical axis using the nearest point depth plus a fixed layer thickness as the slicing threshold.

[0058] The vision processing pipeline is the same pipeline used for locating and grasping targets in the production grasping task, and it shares the multimodal segmentation model parameters, back projection parameters, and top surface thin layer slicing parameters.

[0059] As a preferred embodiment of the above, the calibration-specific aggregation rules and the crawling aggregation rules used in the production crawling task are kept consistent in the horizontal direction by taking the arithmetic mean, while they are distinguished in the vertical direction:

[0060] The calibration-specific aggregation rule takes the minimum value in the vertical direction to ensure that the calibration reference point is strictly located at the highest point of the object;

[0061] The arithmetic mean of the vertical direction of the grasping aggregation rule is taken to place the grasping reference point in the middle of the upper half of the object so that the gripper can hold it stably;

[0062] Both aggregation rules share the same top surface thin-layer point cloud as input.

[0063] In this embodiment, rigid body registration employs a singular value decomposition algorithm, including:

[0064] Removing the centroids of the set of point pairs yields zero-mean point pairs. A cross-covariance matrix is ​​constructed, and singular value decomposition is performed on this matrix to obtain the rotation matrix. ,when The last column of the right singular vector matrix is ​​inverted and R is recalculated to eliminate the reflection solution. Then, the translation vector t is calculated from the centroid and combined into a 4×4 homogeneous transformation matrix.

[0065] As a preferred embodiment of the above, the method further includes:

[0066] The camera coordinates of all calibration points are transformed to the robotic arm coordinate system using a homogeneous transformation matrix. The root mean square error between the transformation result and the corresponding robotic arm coordinates is calculated as the registration residual.

[0067] When the registration residual is greater than the preset threshold, the operator is prompted to check the acquisition quality of the calibration points and re-execute the calibration process;

[0068] The homogeneous transformation matrix and the registration residual are persistently saved when the registration residual is not greater than a preset threshold.

[0069] The preset threshold is 3 to 5 millimeters; the method also includes a real-machine verification step:

[0070] The verification object is placed at a non-calibrated location. The camera coordinates of the verification object are calculated through the vision processing pipeline and transformed into the robot arm coordinate system through a homogeneous transformation matrix. The robot arm then moves to the actual coordinates and the deviation between the end of the robot arm and the center of the top surface of the verification object is visually observed to intuitively confirm the calibration quality.

[0071] This embodiment also includes a robotic arm hand-eye calibration system that reuses a vision pipeline, using the method described above. The system includes:

[0072] The camera is fixedly mounted above the working area of ​​the robotic arm to collect time-synchronized color and depth maps.

[0073] A robotic arm is used to perform movements based on the camera coordinates and the robotic arm coordinates of a calibration reference point.

[0074] The main control computer is used to preset calibration points and record the coordinates of the robotic arm and text prompts. It calls the same vision processing pipeline as the production grasping task to process the object image to obtain a thin point cloud on the top surface. It calculates the camera coordinates of the calibration reference point using the calibration-specific aggregation rule, and performs rigid body registration on the point pair set to solve the homogeneous transformation matrix. The calibration-specific aggregation rule is to take the arithmetic mean of the horizontal coordinate components and the minimum value of the vertical coordinate components.

[0075] Example 2:

[0076] Figure 2 The diagram shows the overall flowchart of the calibration method of this invention, which is divided into three stages: offline configuration, online acquisition loop, and solution saving. In the offline configuration stage, the operator presets no fewer than four non-coplanar calibration points. Each point records the coordinates of the robotic arm base coordinate system and the corresponding text prompt, used for object positioning during subsequent online acquisition. In the online acquisition loop, the operator sequentially places the object at each calibration point and triggers confirmation; the system acquires time-synchronized color and depth maps, calling the same vision pipeline as the production capture (multimodal segmentation → masking → gated back projection → top surface thin-layer slicing), and extracts the top surface thin-layer point cloud; the calibration-specific aggregation rule calculates the camera coordinates of the calibration reference point by taking the arithmetic mean of the horizontal coordinates (X, Y) and the minimum value of the vertical coordinate (Z), and pairs them with the corresponding robotic arm coordinates of the calibration point, adding them to the point pair set; the loop exits after all points are completed. During the solution phase, SVD rigid body registration (including reflection correction) is performed on the point-to-set set to obtain a 4×4 homogeneous transformation matrix; then the registration residual RMSE is calculated: if it exceeds the preset threshold, the operator is prompted to recalibrate; otherwise, the transformation matrix and registration residual are persistently saved for subsequent crawling tasks to load.

[0077] The specific steps include:

[0078] At least four calibration points, not all located on the same plane, are preset. Each point records its coordinates in the robot arm base coordinate system and a text prompt. It is recommended that the at least four calibration points be distributed across at least two different height layers. The number of points and height layers are determined based on the size of the robot arm's workspace and accuracy requirements. In practice, it is recommended to deploy 6 to 10 points evenly distributed within the reachable space. Non-coplanarity is determined by calculating the volume of the convex hull formed by all calibration points. If the volume is greater than a preset threshold, it is determined to be non-coplanar.

[0079] The operator places the object at the current calibration point and triggers confirmation; wherein the object is a common rigid object with divisible appearance features and an approximately continuous curved surface on the top surface.

[0080] The camera acquires time-synchronized color and depth maps and reads camera intrinsic parameters.

[0081] The visual processing pipeline is invoked to obtain a thin point cloud of the target top surface; the visual processing pipeline includes: sending a color image and text prompts into multimodal segmentation to obtain a binary mask; using the logical AND of the two conditions "mask value is true" and "depth value is valid" as back projection filtering; and extracting the thin point cloud of the top surface using "nearest depth + fixed layer thickness".

[0082] The calibration reference point is calculated for the thin point cloud on the top surface according to the calibration-specific aggregation rule: the arithmetic mean of the two horizontal coordinate components is taken, and the minimum value in the vertical direction is taken; the horizontal average is taken so that the reference point is located near the center of the top surface, and the minimum value in the vertical direction is taken so that the reference point is strictly located at the highest point of the object, thereby eliminating the influence of the top surface slice thickness Δh on the height estimation, and making the calibration result robust to the slice parameters;

[0083] The camera coordinates of the calibration reference point and the robot coordinates of the corresponding calibration point are taken as a set of added points.

[0084] The classic SVD algorithm of Arun 1987 is used to perform rigid body registration on the set of point pairs—centroid removal, construction of cross-covariance matrix, SVD decomposition, and rigid body rotation obtained from the rotation matrix R = V·Uᵀ; when det(R) = −1, the last column of the right singular vector matrix is ​​inverted and R is recalculated to eliminate reflection solutions and ensure true rigid body transformation. Then, the translation vector is calculated from the centroid to obtain a 4×4 homogeneous transformation matrix.

[0085] Accuracy evaluation of the transformation matrix: Transform the camera coordinates of all calibration points to the robotic arm coordinate system using the transformation matrix obtained in step 7, and calculate the root mean square error (RMSE) between the transformation result and the corresponding robotic arm coordinates as the registration residual; if the residual is greater than the preset threshold (usually 3-5 mm), prompt the operator to check the acquisition quality of each calibration point and repeat steps 1 to 7; if the residual is within an acceptable range, persistently save the homogeneous transformation matrix, rotation matrix, translation vector, and registration residual values ​​for subsequent grasping tasks; the operator can also choose to perform a live verification step: place an additional verification object at a non-calibration point, calculate its camera coordinates through the vision pipeline and transform them to the robotic arm coordinate system using the calibration matrix, have the robotic arm actually move to that coordinate and hover, and visually observe the deviation between the end effector and the center of the top surface of the object to further confirm the calibration quality.

[0086] The industrial gripping station adopts a multi-point, multi-height layer configuration, using ordinary hardware as calibration objects, and the text prompt is uniformly "object". The number of points and height layers can be expanded according to the accuracy requirements and the reach of the robotic arm. The more points and the more uniform the height layer distribution, the stronger the noise suppression capability of SVD registration.

[0087] Figure 3The diagram shows a comparison of point cloud aggregation rules between calibration mode and gripping mode. Both modes share the same top surface thin-layer point cloud as input, generated by the same vision pipeline. In calibration mode, the horizontal coordinates (X, Y) are taken as an arithmetic mean, and the vertical coordinate (Z) is taken as the minimum value (i.e., the highest point of the object), ensuring that the calibration reference point falls precisely at the center of the object's top surface, facilitating visual verification by the operator to confirm the consistency between the reference point and the object's top surface center. In gripping mode, the arithmetic mean of all three coordinate directions (X, Y, Z) is taken, placing the reference point in the middle of the upper half of the object, facilitating stable gripping by the gripper. The two modes maintain consistency in horizontal aggregation rules, but intentionally differentiate vertical aggregation rules to meet the different needs of calibration accuracy verification and gripping stability; this differentiation also eliminates the influence of the top surface slice thickness Δh on the calibration height estimation.

[0088] This embodiment has the following beneficial effects:

[0089] By replacing the calibration target with a regular object, object recognition is achieved through multimodal segmentation driven by text prompts.

[0090] The calibration and production grasp share the same vision pipeline, reducing systematic deviations between calibration and production;

[0091] The aggregation rules for calibration reference points and the aggregation rules for grabbing reference points are intentionally distinguished: calibration uses "average value of water level + highest vertical point" to make the reference point closer to the center of the top surface of the object; grabbing uses "average value of XYZ" to make the reference point located in the middle of the object for easy clamping; both use the same thin layer point cloud of the top surface as input.

[0092] No dedicated calibration target is required; calibration and production share the same vision pipeline, avoiding discrepancies between "calibration target positioning accuracy" and "production object positioning accuracy"; when ordinary objects are used as calibration references, operators can visually verify the consistency between the reference point and the center of the object's top surface; tolerance to occlusion and changes in lighting is better than corner detection.

[0093] Example 3:

[0094] Please see Figure 4 The diagram shows a structural schematic of a computer device provided in an embodiment of this application. An embodiment of this application provides a computer device 400, including a processor 410 and a memory 420. The memory 420 stores a computer program executable by the processor 410. When the computer program is executed by the processor 410, it performs the method described above.

[0095] This application embodiment also provides a storage medium 430, on which a computer program is stored, and the computer program is executed by a processor 410 to perform the above method.

[0096] The storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0097] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0098] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0099] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0100] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0101] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0102] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0103] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0104] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for hand-eye calibration of a robotic arm that reuses vision pipelines, characterized in that, Includes the following steps: Preset at least 4 calibration points that are not all located on the same plane, and record the robot arm coordinates and corresponding text prompts for each calibration point in the robot arm base coordinate system; The object is placed sequentially at each of the specified calibration points and confirmation is triggered to collect time-synchronized color and depth maps. The same vision processing pipeline as the production capture task is invoked to process the object at each calibration point to obtain a thin point cloud of the top surface of the object. The camera coordinates of the calibration reference point are calculated for the thin point cloud on the top surface according to the calibration-specific aggregation rules: the arithmetic mean of the two horizontal coordinate components is taken, and the minimum value of the vertical coordinate component is taken, so that the calibration reference point is located at the center of the top surface of the object. The camera coordinates of the calibration reference point and the robot coordinates of the corresponding calibration point are added as a pair to the point pair set; after traversing all calibration points, rigid body registration is performed on the point pair set, and the homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system is solved.

2. The method according to claim 1, characterized in that, The at least four calibration points, which are not all located on the same plane, are distributed at at least two different height levels and are evenly distributed within the workspace accessible to the robotic arm. The non-coplanarity of calibration points is determined by calculating the volume of the convex hull formed by all calibration points. If the volume of the convex hull is greater than a preset threshold, it is determined to be non-coplanar.

3. The method according to claim 2, characterized in that, The object is a common rigid object with divisible appearance features and an approximately continuous curved surface on its top surface, requiring no special calibration target or marker.

4. The method according to claim 1, characterized in that, The vision processing pipeline includes: Based on the text prompt, a binary mask is generated from the color image using a multimodal segmentation model. The target point cloud is obtained by back-projecting the depth image using the binary mask and depth validity as joint filtering conditions. The top surface thin layer point cloud is extracted along the optical axis using the nearest point depth plus a fixed layer thickness as the slicing threshold.

5. The method according to claim 4, characterized in that, The vision processing pipeline is the same as the vision processing pipeline used to locate and grasp targets in the production grasping task, and they share the multimodal segmentation model parameters, back projection parameters, and top surface thin layer slicing parameters.

6. The method according to claim 1, characterized in that, The calibration-specific aggregation rule and the grab aggregation rule used in the production grabbing task are consistent in the horizontal direction, both taking an arithmetic mean, but are differentiated in the vertical direction: The calibration-specific aggregation rule takes the minimum value in the vertical direction to ensure that the calibration reference point is strictly located at the highest point of the object. The gripping aggregation rule takes the arithmetic mean in the vertical direction to place the gripping reference point in the middle of the upper half of the object so that the gripper can hold it stably. Both aggregation rules share the same top surface thin-layer point cloud as input.

7. The method according to claim 1, characterized in that, The rigid body registration employs a singular value decomposition algorithm, including: Remove the centroids of the set of point pairs to obtain zero-mean point pairs, construct the cross-covariance matrix, and perform singular value decomposition on the cross-covariance matrix to obtain the rotation matrix. ,when The last column of the right singular vector matrix is ​​inverted and R is recalculated to eliminate the reflection solution. Then, the translation vector t is calculated from the centroid and combined into a 4×4 homogeneous transformation matrix.

8. The method according to claim 1, characterized in that, The method further includes: The camera coordinates of all calibration points are transformed to the robotic arm coordinate system using the homogeneous transformation matrix, and the root mean square error between the transformation result and the corresponding robotic arm coordinates is calculated as the registration residual. When the registration residual is greater than a preset threshold, the operator is prompted to check the acquisition quality of the calibration points and re-execute the calibration process; When the registration residual is not greater than the preset threshold, the homogeneous transformation matrix and the registration residual are persistently saved.

9. The method according to claim 8, characterized in that, The preset threshold is 3 to 5 millimeters; the method also includes a live-machine verification step: The verification object is placed at a non-calibrated location. The camera coordinates of the verification object are calculated through the vision processing pipeline and transformed into the robot arm coordinate system through the homogeneous transformation matrix. The robot arm then moves to the actual coordinates and the deviation between the end of the robot arm and the center of the top surface of the verification object is visually observed to intuitively confirm the calibration quality.

10. A robotic arm hand-eye calibration system that reuses vision pipelines, characterized in that, Using the method of any one of claims 1 to 9, the system comprises: The camera is fixedly mounted above the working area of ​​the robotic arm to collect time-synchronized color and depth maps. A robotic arm is used to perform movements based on the camera coordinates and the robotic arm coordinates of a calibration reference point. The main control computer is used to preset calibration points and record the coordinates of the robotic arm and text prompts. It calls the same vision processing pipeline as the production grasping task to process the object image to obtain a thin point cloud on the top surface. It calculates the camera coordinates of the calibration reference point using a calibration-specific aggregation rule and performs rigid body registration on the point pair set to solve the homogeneous transformation matrix. The calibration-specific aggregation rule is to take the arithmetic mean of the horizontal coordinate components and the minimum value of the vertical coordinate components.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-9.

12. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-9.

Images