An intelligent calibration method for the multi-robot multi-mode arm-hand-eye relationship based on online reconstruction-rendering-matching

Abstract

The present invention relates to the technical field of intelligent robots, and particularly relates to a multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching. The method includes: controlling multiple robots whose calibration relationships need to be confirmed to move under a configuration sequence for calibration; establishing an online three-dimensional model of each robot according to the online joint angles, kinematic calibration of each robot, and the three-dimensional models of each link of the robot; generating respective two-dimensional images according to the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship in combination with the online three-dimensional model, comparing the respective two-dimensional images with the respective corresponding actual images, and optimizing according to the differences obtained from the comparison to obtain an accurate hand-eye calibration relationship, an accurate arm-eye calibration relationship, and an accurate arm-arm calibration relationship. The present invention provides a multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching, which can accurately calibrate robots.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and particularly relates to a multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching. Background Art

[0002] Currently, with the continuous improvement of the task complexity, action fineness, and object flexibility of robot operations, the perception and manipulation capabilities of robot systems urgently need to be further improved. Since multi-robot systems can achieve close-range fusion observations from multiple perspectives and collaborative operations on targets, it is of great significance to carry out collaborative operations using multi-robot systems. However, the motion chain of multi-robots is complex, with multiple types of pose relationships among multiple arms, multiple hands, and multiple eyes, and it is easy to reduce the system perception accuracy due to excessive calibration errors. Existing calibration methods have disadvantages such as insufficient depth perception accuracy, single calibration mode, non-intuitive calibration error evaluation, and the need to rely on precise calibration objects. Therefore, researching a multi-robot multi-mode arm-hand-eye intelligent calibration method is of great significance for improving the perception accuracy of multi-robot systems. Summary of the Invention

[0003] An embodiment of the present invention provides a multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching, which can accurately calibrate robots.

[0004] An embodiment of the present invention provides a multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching, including:

[0005] Controlling multi-robots with calibration relationships to be confirmed to move under a configuration sequence for calibration; wherein, the robots include arms, hands, and eyes, the eyes include cameras installed at the end of the robot and cameras installed externally for observing the arms, the calibration relationships to be determined include hand-eye calibration relationships, arm-eye calibration relationships, and arm-arm calibration relationships, the hand-eye calibration relationship is the relationship of the camera installed at the end of one robot relative to the center coordinate system of the robot end flange, the arm-eye calibration relationship is the relationship of the camera installed externally relative to the base of another robot, and the arm-arm calibration relationship is the relationship between different robot bases;

[0006] Establishing an online three-dimensional model of each robot according to the online joint angles, kinematic calibration, and three-dimensional models of each link of the robot.

[0007] Generate respective two-dimensional images based on the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship in combination with the online three-dimensional model, compare the respective two-dimensional images with the respective corresponding actual images, and optimize according to the differences obtained from the comparison to obtain an accurate hand-eye calibration relationship, an accurate arm-eye calibration relationship, and an accurate arm-arm calibration relationship.

[0008] Optionally, before controlling the multi-robot with the calibration relationship to be confirmed to move in the configuration sequence for calibration, it includes:

[0009] Select the calibration methods for the hand-eye calibration relationship, the arm-arm calibration relationship, and the arm-eye calibration relationship according to the configuration of the robot and the camera.

[0010] Optionally, the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship are obtained by the following method:

[0011] Match the online three-dimensional model with the three-dimensional point cloud of the eye online test of the robot to obtain the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship.

[0012] Optionally, the step of generating respective two-dimensional images based on the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship in combination with the online three-dimensional model, comparing the respective two-dimensional images with the respective corresponding actual images, and optimizing according to the comparison differences to obtain an accurate hand-eye calibration relationship, an accurate arm-eye calibration relationship, and an accurate arm-arm calibration relationship includes:

[0013] Respectively combine the initial hand-eye calibration relationship and the initial arm-eye calibration relationship with the online three-dimensional model, and perform rendering based on the parameters of the camera of the eye to generate a hand-eye calibration two-dimensional image and an arm-eye calibration two-dimensional image respectively;

[0014] Compare the hand-eye calibration two-dimensional image and the arm-eye calibration two-dimensional image with the actual pictures acquired by the camera respectively, quantitatively evaluate the differences between the hand-eye calibration two-dimensional image and the arm-eye calibration two-dimensional image and their respective corresponding actual pictures, and optimize the initial hand-eye calibration relationship and the initial arm-eye calibration relationship according to the differences to obtain an accurate hand-eye calibration relationship and an accurate arm-eye calibration relationship;

[0015] Based on the accurate hand-eye calibration relationship, combine the online three-dimensional model and the initial arm-arm calibration relationship, and perform rendering based on the parameters of the camera of the eye to generate an arm-arm two-dimensional image;

[0016] Compare the arm-arm two-dimensional image with the actual picture acquired by the camera, quantitatively evaluate the difference between the arm-arm two-dimensional image and its corresponding actual picture, and optimize the initial arm-arm calibration relationship according to the difference to obtain an accurate arm-arm calibration relationship.

[0017] Optionally, the camera is an RGBD camera.

[0018] Optionally, the sequence configuration is generated in the following manner:

[0019] Generate a robot sequence configuration according to the current joint angles of the robot, as well as the models of each connecting rod, camera and connecting piece, and environmental obstacle, in combination with the camera field of view range and motion obstacle avoidance characteristics.

[0020] Optionally, establishing the online three-dimensional model of each robot according to the online joint angles of each robot, kinematic calibration, and the three-dimensional models of each connecting rod of the robot includes:

[0021] According to the robot kinematic parameters and online joint angles, calculate the pose coordinates of each connecting rod based on forward kinematics, and combine the mesh models of each connecting rod to generate an online three-dimensional model of the robot for cropping based on the camera internal parameters.

[0022] Optionally, matching the online three-dimensional model with the three-dimensional point cloud of the robot's eye online test to obtain the initial hand-eye calibration relationship, initial arm-eye calibration relationship, and initial arm-arm calibration relationship includes:

[0023] Adopt a pose estimation method, based on the online three-dimensional model, and combine the online acquired image and three-dimensional point cloud for matching to obtain the pose estimation value of the three-dimensional model of the robot in the three-dimensional point cloud, that is, the initial hand-eye calibration relationship, initial arm-eye calibration relationship, and initial arm-arm calibration relationship of the robot body relative to the camera.

[0024] Optionally, comparing each respective two-dimensional image with its corresponding actual image and optimizing according to the obtained difference includes:

[0025] For each two-dimensional image and its corresponding actual image, perform the following operations:

[0026] Perform segmentation on the actual picture acquired by the camera based on semantic prompts;

[0027] Replace the background after segmentation with the same color as the background of the rendered two-dimensional image;

[0028] Use an image structure measurement index to evaluate the image difference;

[0029] Adopt a particle swarm optimization method to perform sampling optimization in the area near the initial value.

[0030] The present invention has at least the following beneficial effects compared with the prior art:

[0031] In this embodiment, move multiple robots according to the configuration for calibration to move them to a configuration convenient for calibration. Then, establish an online three-dimensional model based on the relevant parameters of each robot. Compare the two-dimensional image obtained based on the online three-dimensional model and the initial calibration relationship with the actual image actually captured by the camera, and optimize according to the comparison result to obtain the accurate calibration relationship between the camera and the robot's hand and arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0033] Figure 1 Flowchart of the intelligent calibration method for the multi-robot multi-mode arm-hand-eye relationship based on online reconstruction-rendering-matching of the present invention;

[0034] Figure 2 Initial arm-eye calibration rendering image of this embodiment of the present invention;

[0035] Figure 3 Actual arm-eye calibration image obtained by the camera in this embodiment of the present invention;

[0036] Figure 4 Comparison image of actual shooting and rendering in the case of the initial value of the arm-eye relationship in this embodiment of the present invention;

[0037] Figure 5 Comparison image of actual shooting and rendering under the final arm-eye calibration result in the embodiment of the present invention;

[0038] Figure 6 Comparison image of actual shooting and rendering under the final hand-eye calibration result in the embodiment of the present invention;

[0039] Figure 7 Comparison image of actual shooting and rendering under the final arm-arm calibration result in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0041] In the description of the embodiments of the present invention, unless otherwise clearly specified or limited, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance; unless otherwise specified or stated, the term "plurality" means two or more; the terms "connection", "fixation", etc. shall be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, an integral connection, or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0042] In the description of this specification, it should be understood that the orientation terms such as "upper" and "lower" described in the embodiments of the present invention are described from the angles shown in the accompanying drawings and should not be construed as limiting the embodiments of the present invention. In addition, in the context, it should also be understood that when it is mentioned that an element is connected "above" or "below" another element, it can not only be directly connected "above" or "below" another element, but also be indirectly connected "above" or "below" another element through an intermediate element.

[0043] As Figure 1 shown, the embodiments of the present invention provide a multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching, including:

[0044] S1. Control the multi-robots to be calibrated for the calibration relationship to move under the configuration sequence for calibration; wherein, the robots include arms, hands, and eyes, the eyes include cameras installed at the end of the robots and cameras installed externally for observing the arms, the relationships to be calibrated include hand-eye calibration relationships, arm-eye calibration relationships, and arm-arm calibration relationships, the hand-eye calibration relationship is the relationship between the camera installed at the end of one robot and the center coordinate system of the robot end flange, the arm-eye calibration relationship is the relationship between the camera installed externally and the base of another robot, and the arm-arm calibration relationship is the relationship between the bases of different robots;

[0045] S2. Establish the online 3D models of each robot according to the online joint angles, kinematic calibrations, and 3D models of each link of the robots.

[0046] S3. Based on the initial hand-eye calibration relationship, initial arm-eye calibration relationship, and initial arm-arm calibration relationship, combined with the online three-dimensional model, generate respective two-dimensional images, compare the respective two-dimensional images with the respective corresponding actual images, and optimize according to the differences obtained from the comparison to obtain an accurate hand-eye calibration relationship, an accurate arm-eye calibration relationship, and an accurate arm-arm calibration relationship.

[0047] In this embodiment, move multiple robots according to the configuration sequence for calibration to move them to a configuration convenient for calibration. Then, based on the relevant parameters of each robot, establish an online three-dimensional model. Compare the two-dimensional image obtained from the online three-dimensional model and the initial calibration relationship with the actual image actually captured by the camera, and optimize according to the comparison result to obtain an accurate calibration relationship between the camera and the robot's hand and arm.

[0048] In some embodiments of the present invention, before S1, it includes:

[0049] Select the calibration methods for the hand-eye calibration relationship, arm-arm calibration relationship, and arm-eye calibration relationship according to the configuration of the robot and the camera.

[0050] The specific calibration methods are as follows:

[0051] (1) Hand-eye calibration: The object to be calibrated is a single-robot system, and its RGBD camera ("eye") is fixedly connected to the robot's end flange ("hand") through a structural member. The calibrated value is the pose transformation relationship of the RGBD camera imaging coordinate system relative to the center coordinate system of the robot's end flange.

[0052] (2) Arm-arm calibration: The object to be calibrated is a multi-robot system, and its RGBD camera is installed at the end of one of the robots. The calibrated value is the pose transformation relationship of the coordinate system of robot 1's base ("arm") relative to the coordinate system of robot 2's base ("arm").

[0053] (3) Arm-eye calibration: The object to be calibrated is a multi-robot system or a single-robot system, and its RGBD camera ("eye") is located outside the robot ("arm"), and can be located on a fixed mounting bracket or the end of other robots. The calibrated value is the pose transformation relationship of the RGBD camera imaging coordinate system relative to the robot's base coordinate system.

[0054] In this embodiment, a robot 1, a robot 2, an RGBD camera 1, and an RGBD camera 2 are used. Among them, both robots use UR5e robotic arms, and both RGBD cameras use ZED Mini cameras. The RGBD camera 1 is installed at the end of the robot 2, and the RGBD camera 2 is installed on an external fixed bracket of the robot. To achieve precise calibration of the robot system, the values to be calibrated are: the relationship of the RGBD camera 1 relative to the robot 2 (hand-eye calibration), the relationship of the robot 2 relative to the robot 1 (arm-arm calibration), and the relationship of the RGBD camera 2 relative to the robot 1 (arm-eye calibration). A complete and precise motion relationship chain can be constructed based on the above calibration values for precise manipulation tasks.

[0055] For S1, the configurations used for calibration are represented by multiple sets of robot joint angles. The factors to be considered include:

[0056] (1) Under multiple sets of calibration configurations, the pose change of the camera is as large as possible;

[0057] (2) Under multiple sets of calibration configurations, the observation range of the camera covers the robot body as much as possible;

[0058] (3) The difference in joint angles of multiple sets of calibration configurations is as small as possible to improve the efficiency of the calibration process;

[0059] (4) During multiple sets of calibration configurations and their switching processes, it is strictly prohibited for the robot to have self-collisions, collisions with other robots, and collisions with environmental objects.

[0060] This embodiment uses an optimization algorithm to solve. The above factors (1), (2), and (3) are used as optimization sub-goals, and their weight settings decrease in turn. (4) is used as a strict constraint of the algorithm. Obtain the configuration sequences of the robots 1 and 2 for hand-eye calibration, arm-arm calibration, and arm-eye calibration respectively. In hand-eye calibration, the observation target of the camera 1 is the robot 2; in arm-arm calibration, the observation target of the camera 1 is the robot 1; in arm-eye calibration, the observation target of the camera 2 is the robot 1.

[0061] For S2, use the online joint angles, kinematic calibration of each robot, and the three-dimensional model reconstruction of each link to reconstruct the current online three-dimensional models of each robot, and the model can be cropped based on the camera internal parameters when there is a calibration initial value.

[0062] In this embodiment, the ROS system is used to read the information of the robots 1 and 2, and the Trimesh library is used for online reconstruction and cropping of the robot three-dimensional models:

[0063] (1) Use the ROS system to update the URDF file according to the kinematic calibration parameters of the robots 1 and 2;

[0064] (2) Based on the updated URDF file, use the ROS system to obtain the online joint angles of robots 1 and 2, and use the TF module of the ROS system to calculate the online poses of each link coordinate system;

[0065] (3) Based on the updated URDF file, obtain the pose transformation relationship between the coordinate system of each link Mesh grid and the link coordinate system calculated by the TF module, and calculate the coordinate system of each link Mesh grid;

[0066] (4) Use the Trimesh library to read the Mesh grids of each link of robots 1 and 2, and based on the coordinate systems of each link Mesh network, merge them into the online 3D Mesh grid models of robots 1 and 2 respectively;

[0067] (5) For the case where there is a calibration initial value, the external parameter matrix of the camera can be calculated, and according to the internal parameter matrix of the camera, the planes of the four imaging edges are calculated, and the 3D models of robots 1 and 2 are cropped in turn to obtain the part in the camera's field of view for subsequent 3D matching.

[0068] (6) According to the configurations generated by S1, move the robots respectively to obtain the online 3D models of each robot in three calibration states.

[0069] In some embodiments of the present invention, the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship are obtained in the following manner:

[0070] Match the online 3D model with the 3D point cloud of the robot's eye online test to obtain the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship.

[0071] Use the 3D point cloud and images measured by the robot online to match with the reconstructed 3D model to obtain the initial hand-eye or arm-eye pose relationship.

[0072] In this embodiment, the hand-eye and arm-eye poses are calibrated respectively. For the case of whether there is a calibration initial value, the following matching methods can be used:

[0073] (1) For the case where there is a calibration initial value, obtain the cropped online 3D model of the robot relative to the camera coordinate system obtained through S2, and use traditional point cloud matching methods such as ICP and intelligent matching methods such as the Refine network in FoundationPose to estimate the pose for the 3D point cloud obtained by the RGBD camera, and obtain the initial hand-eye and arm-eye pose calibration relationships;

[0074] (2) For the case without calibrated initial values, manually extract no less than 4 pairs of feature points from the 3D model of the robot and the 3D point cloud obtained by the RGBD camera, obtain the 3D coordinates of the feature points, use the ICP method to obtain the pose transformation matrix, and perform model pose transformation based on this matrix. Then, the method in (1) can be used to obtain the initial calibration relationships of hand-eye and arm-eye.

[0075] Of course, the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship can also be set values.

[0076] In some embodiments of the present invention, S3 includes:

[0077] S301, respectively combine the initial hand-eye calibration relationship and the initial arm-eye calibration relationship with the online 3D model, perform rendering based on the parameters of the camera of the eye, and generate the hand-eye calibration 2D image and the arm-eye calibration 2D image respectively;

[0078] S302, compare the hand-eye calibration 2D image and the arm-eye calibration 2D image with the actual pictures obtained by the camera respectively, quantitatively evaluate the differences between the hand-eye calibration 2D image and the arm-eye calibration 2D image and their respective corresponding actual pictures, and optimize the initial hand-eye calibration relationship and the initial arm-eye calibration relationship according to the differences to obtain the accurate hand-eye calibration relationship and the accurate arm-eye calibration relationship;

[0079] S303, based on the accurate hand-eye calibration relationship, combine the online 3D model and the initial arm-arm calibration relationship, perform rendering based on the parameters of the camera of the eye, and generate the arm-arm 2D image;

[0080] S304, compare the arm-arm 2D image with the actual picture obtained by the camera, quantitatively evaluate the difference between the arm-arm 2D image and its corresponding actual picture, and optimize the initial arm-arm calibration relationship according to the difference to obtain the accurate arm-arm calibration relationship.

[0081] For S301, the specific steps are as follows:

[0082] (1) For the hand-eye and arm-eye calibration types respectively, calculate the pose relationship between the camera and the robot. In the hand-eye calibration type, the initial hand-eye calibration relationship between camera 1 and robot 2 is Read through the ROS system, as the initial calibration value; in the arm-eye calibration type, the initial arm-eye calibration relationship between camera 2 and robot 1 is the initial calibration value

[0083] (2) For both hand-eye and arm-eye calibration types, parameters are set according to the light source position, brightness, etc. in the current actual scene. The environmental background color is set to white. Using the Pyrender library, the current online 3D model of the robot is loaded at the base coordinate system of the Pyrender virtual environment. According to the current external camera parameters or and the respective internal camera parameters f x 、f y 、c x 、c y , the hand-eye calibration rendered image and the arm-eye calibration rendered image under the current initial or calibration parameters are generated respectively. In this embodiment, the initial arm-eye calibration rendered image is as shown in Figure 2 , and the actual image obtained by the camera is as shown in Figure 3 .

[0084] For S302, the specific operation steps are as follows:

[0085] (1) Preprocess the actual two-dimensional image obtained by the camera. Using the GroundDINO method, detect the robot target with the prompt word "robot". Combine the obtained detection box with the SAM segmentation model to obtain the segmentation result of the robot. Based on the segmented mask result, remove the environmental background except the robot in the actual image;

[0086] (2) For the preprocessed actual two-dimensional image and the two-dimensional images rendered in step S301 (hand-eye calibration two-dimensional image and arm-eye calibration two-dimensional image), evaluate the similarity from three angles of brightness, contrast, and structure based on the SSIM structure measurement index, and the image similarity under the currently set and calibration parameters can be obtained;

[0087] (3) Denote and as tx1, ty1, tz1, r1, p1, y1 and tx2, ty2, tz2, r2, p2, y2 respectively, where txi, tyi, tzi are displacement components, and ri, pi, yi are Euler angles, and i is 1 or 2. Determine the search range of the optimization algorithm according to the deviation prediction situation. Adopt the particle swarm optimization algorithm (PSO), and the calculation of the optimization function at each particle is realized by step (2);

[0088] (4) Obtain the optimization results tx1 * 、ty1 * 、tz1 * 、r1 * 、p1 * 、y1* and tx2 * 、ty2 * 、tz2 * 、r2 * 、p2 * 、y2 * respectively represent as and that is, the precise hand-eye calibration relationship and the precise arm-eye calibration relationship respectively. It will be based on and The images generated in step (2) and the images obtained in step (1) are manually compared, and evaluated according to the pixel-level error. In this embodiment, in the case of the initial value of the arm-eye relationship, the actual captured and rendered comparison images are as Figure 4 shown. In the case of the final arm-eye calibration result, the actual captured and rendered comparison images are as Figure 5 shown. In the case of the final hand-eye calibration result, the actual captured and rendered comparison images are as Figure 6 shown.

[0089] For S303, the specific operations are as follows:

[0090] (1) For the arm-arm calibration type, calculate the pose relationship of the camera 1 fixed on the robot 2 relative to the base of the robot 1. Calculate and obtain the pose transformation relationship of the camera 1 relative to the robot 1 where can be obtained through the ROS system, is the hand-eye calibration result obtained by the robot 2 using the method in step (4) of S302

[0091] (2) For the case without an initial value, use the method in step (4) to obtain the initial value of , and then calculate and obtain the initial value of according to the pose transformation relationship in step (1);

[0092] (3) For the case with an initial value, calculate the initial value of according to the pose transformation relationship in step (1), and use the method in step (2) of S301 to obtain the rendered two-dimensional image.

[0093] For S304, the specific operation steps are as follows:

[0094] (1) Use the above method to obtain the precise calibration value of ;

[0095] (2) Based on the pose transformation relationship of the camera 1 relative to the robot 1, calculate the relative pose calibration value of "arm-arm" Under the final arm-arm calibration result, the comparison images of the actual shooting and rendering are as follows Figure 7 shown

[0096] In some embodiments of the present invention, the camera is an RGBD camera

[0097] In some embodiments of the present invention, the sequence configuration is generated in the following manner

[0098] According to the current joint angles of the robot, as well as each connecting rod, camera and connecting piece, and environmental obstacle model, combined with the camera field of view range and motion obstacle avoidance characteristics, a robot sequence configuration is generated

[0099] In some embodiments of the present invention, an online 3D model of each robot is established according to the online joint angles of each robot, kinematic calibration, and the 3D models of each connecting rod of the robot, including

[0100] According to the robot kinematic parameters and online joint angles, based on forward kinematics, calculate the pose coordinates of each connecting rod, and combine the mesh models of each connecting rod to generate an online 3D model of the robot for cropping based on the camera internal parameters

[0101] In some embodiments of the present invention, the online 3D model is matched with the 3D point cloud of the robot's eye online test to obtain the initial hand-eye calibration relationship, initial arm-eye calibration relationship, and initial arm-arm calibration relationship, including

[0102] Using a pose estimation method, based on the online 3D model, combined with the online acquired image and 3D point cloud for matching, obtain the pose estimation value of the robot 3D model in the 3D point cloud, that is, the initial hand-eye calibration relationship, initial arm-eye calibration relationship, and initial arm-arm calibration relationship of the robot body relative to the camera

[0103] In some embodiments of the present invention, compare the respective two-dimensional images with the respective corresponding actual images, and optimize according to the differences obtained from the comparison, including

[0104] For each two-dimensional image and its corresponding actual image, perform the following operations

[0105] Segment the actual picture acquired by the camera based on semantic prompts

[0106] Replace the background after segmentation with the same color as the background of the rendered two-dimensional image

[0107] Use an image structure measurement index to evaluate the image difference

[0108] Use the particle swarm optimization method to perform sampling optimization in the area near the initial value

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching, characterized in that Including: Controlling multiple robots with calibration relationships to be confirmed to move under a configuration sequence for calibration; wherein, the robots include arms, hands, and eyes, the eyes include cameras installed at the ends of the robots and cameras installed externally for observing the arms, the calibration relationships to be determined include hand-eye calibration relationships, arm-eye calibration relationships, and arm-arm calibration relationships, the hand-eye calibration relationship is the relationship between the camera installed at the end of one robot and the center coordinate system of the flange at the end of the robot, the arm-eye calibration relationship is the relationship between the camera installed externally and the base of another robot, and the arm-arm calibration relationship is the relationship between the bases of different robots; Establishing an online three-dimensional model for each robot based on the online joint angles of each robot, kinematic calibration, and the three-dimensional models of each link of the robot; Generating respective two-dimensional images by combining the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship with the online three-dimensional model, comparing the respective two-dimensional images with their corresponding actual images, and optimizing according to the obtained differences to obtain accurate hand-eye calibration relationships, accurate arm-eye calibration relationships, and accurate arm-arm calibration relationships.

2. The method according to claim 1, wherein Before the step of controlling multiple robots with calibration relationships to be confirmed to move under a configuration sequence for calibration, including: Selecting the calibration methods for the hand-eye calibration relationship, the arm-arm calibration relationship, and the arm-eye calibration relationship according to the configuration of the robots and the cameras.

3. The method according to claim 1, wherein The initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship are obtained through the following methods: Matching the online three-dimensional model with the three-dimensional point cloud obtained by the online test of the eyes of the robots to obtain the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship.

4. The method according to claim 1, wherein The step of generating respective two-dimensional images by combining the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship with the online three-dimensional model, comparing the respective two-dimensional images with their corresponding actual images, and optimizing according to the comparison differences to obtain accurate hand-eye calibration relationships, accurate arm-eye calibration relationships, and accurate arm-arm calibration relationships includes: Combining the initial hand-eye calibration relationship and the initial arm-eye calibration relationship with the online three-dimensional model respectively, and rendering based on the parameters of the cameras of the eyes to generate a hand-eye calibration two-dimensional image and an arm-eye calibration two-dimensional image respectively; Comparing the hand-eye calibration two-dimensional image and the arm-eye calibration two-dimensional image with the actual pictures obtained by the cameras respectively, quantitatively evaluating the differences between the hand-eye calibration two-dimensional image and the arm-eye calibration two-dimensional image and their corresponding actual pictures respectively, and optimizing the initial hand-eye calibration relationship and the initial arm-eye calibration relationship according to the differences to obtain accurate hand-eye calibration relationships and accurate arm-eye calibration relationships; Based on the accurate hand-eye calibration relationship, combining the online three-dimensional model and the initial arm-arm calibration relationship, and rendering based on the parameters of the cameras of the eyes to generate an arm-arm two-dimensional image; Compare the arm-arm two-dimensional image with the actual image acquired by the camera, quantitatively evaluate the difference between the arm-arm two-dimensional image and its corresponding actual image, and optimize the initial arm-arm calibration relationship according to the difference to obtain an accurate arm-arm calibration relationship.

5. The method according to claim 1, wherein The camera is an RGBD camera.

6. The method according to claim 1, wherein The configuration sequence is generated in the following manner: Generate a robot configuration sequence based on the current joint angles of the robot, as well as the three-dimensional models of each link, camera, connecting parts, and environmental obstacle model, in combination with the camera field of view range and motion obstacle avoidance characteristics.

7. The method according to claim 1, characterized in that, Establishing the online three-dimensional model of each robot according to the online joint angles of each robot, kinematic calibration, and the three-dimensional models of each link of the robot includes: Based on the robot kinematic parameters and online joint angles, calculate the pose coordinates of each link through forward kinematics, and combine the mesh models of each link to generate an online three-dimensional model of the robot for cropping based on the camera internal parameters.

8. The method according to claim 3, characterized in that, Matching the online three-dimensional model with the three-dimensional point cloud of the eye online test of the robot to obtain the initial hand-eye calibration relationship, initial arm-eye calibration relationship, and initial arm-arm calibration relationship includes: Using a pose estimation method, based on the online three-dimensional model, combine the online acquired image and three-dimensional point cloud for matching, and obtain the pose estimation value of the three-dimensional model of the robot in the three-dimensional point cloud, that is, the initial hand-eye calibration relationship, initial arm-eye calibration relationship, and initial arm-arm calibration relationship of the robot body relative to the camera.

9. The method according to claim 1, wherein Compare the respective two-dimensional images with their corresponding actual images and optimize according to the differences obtained from the comparison, including: For each two-dimensional image and its corresponding actual image, perform the following: Segment the actual image acquired by the camera based on semantic prompts; Replace the background after segmentation with the same color as the background of the rendered two-dimensional image; Use an image structure measurement index to evaluate the image difference; Use a particle swarm optimization method to perform sampling optimization in the region near the initial value.

Images