Multi-robot multi-mode arm-hand-eye relation intelligent calibration method based on online reconstruction-rendering-matching

Abstract

The invention relates to the technical field of intelligent robots, in particular to a multi-robot multi-mode arm-hand-eye relation intelligent calibration method based on online reconstruction-rendering-matching. Comprising the following steps: controlling multiple robots of which the calibration relationship is to be confirmed to move under a configuration sequence for calibration; establishing an online three-dimensional model of each robot according to the online joint angle and kinematics calibration of each robot and the three-dimensional model of each connecting rod of the robot; generating respective two-dimensional images according to the initial hand-eye calibration relationship, the initial arm-eye calibration relationship and the initial arm-arm calibration relationship in combination with an online three-dimensional model, comparing the respective two-dimensional images with respective corresponding actual images, and optimizing according to differences obtained by comparison to obtain a calibration result; and obtaining an accurate hand-eye calibration relationship, an accurate arm-eye calibration relationship and an accurate arm-arm calibration relationship. The invention provides a multi-robot multi-mode arm-hand-eye relation intelligent calibration method based on online reconstruction-rendering-matching, and the robot can be accurately calibrated.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robot technology, and in particular to a multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching. Background Art

[0002] At present, with the increasing complexity of robot operation tasks, precision of movements, and flexibility of objects, the perception and control capabilities of robot systems need to be further improved. Since multi-robot systems can realize close-range fusion observation of multiple perspectives and collaborative operations on targets, it is of great significance to use multi-robot systems to carry out collaborative operations. However, the motion chain of multi-robots is complex, and there are multiple types of posture relationships between multiple arms, multiple hands, and multiple eyes. It is easy for the system perception accuracy to be reduced due to excessive calibration errors. The existing calibration methods have the disadvantages of insufficient depth perception accuracy, single calibration mode, non-intuitive calibration error evaluation, and the need for precise calibration objects. Therefore, it is of great significance to study an arm-hand-eye intelligent calibration method that can adapt to multiple modes of multi-robots to improve the perception accuracy of multi-robot systems. Summary of the invention

[0003] The embodiment of the present invention provides a multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching, which can accurately calibrate the robot.

[0004] The embodiment of the present invention provides a multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching, comprising:

[0005] Controlling a multi-robot whose calibration relationship is to be confirmed to move in a configuration sequence for calibration; wherein the robot comprises an arm, a hand and an eye, the eye comprises a camera installed at the end of the robot and a camera installed externally for observing the arm, the relationship to be calibrated comprises a hand-eye calibration relationship, an arm-eye calibration relationship and an arm-arm calibration relationship, the hand-eye calibration relationship is a relationship between a camera installed at the end of one robot and a center coordinate system of a flange at the end of the robot, the arm-eye calibration relationship is a relationship between a camera installed externally and a base of another robot, and the arm-arm calibration relationship is a relationship between bases of different robots;

[0006] Establish an online 3D model of each robot based on the online joint angle, kinematic calibration and 3D model of each link of the robot;

[0007] According to the initial hand-eye calibration relationship, the initial arm-eye calibration relationship and the initial arm-arm calibration relationship, respective two-dimensional images are generated in combination with the online three-dimensional model, and the respective two-dimensional images are compared with the respective corresponding actual images. According to the differences obtained by the comparison, optimization is performed to obtain accurate hand-eye calibration relationship, accurate arm-eye calibration relationship and accurate arm-arm calibration relationship.

[0008] Optionally, before controlling the multi-robots with calibration relationships to be confirmed to move in a configuration sequence for calibration, the method includes:

[0009] Select the hand-eye calibration relationship, arm-arm calibration relationship, and arm-eye calibration relationship calibration method according to the configuration of the robot and camera.

[0010] Optionally, the initial hand-eye calibration relationship, the initial arm-eye calibration relationship and the initial arm-arm calibration relationship are obtained by:

[0011] The online three-dimensional model is matched with the three-dimensional point cloud of the online eye test of the robot to obtain an initial hand-eye calibration relationship, an initial arm-eye calibration relationship and an initial arm-arm calibration relationship.

[0012] Optionally, generating respective two-dimensional images according to the initial hand-eye calibration relationship, the initial arm-eye calibration relationship and the initial arm-arm calibration relationship in combination with the online three-dimensional model, comparing the respective two-dimensional images with the respective corresponding actual images, and obtaining accurate hand-eye calibration relationship, accurate arm-eye calibration relationship and accurate arm-arm calibration relationship after optimizing according to the comparison difference, comprises:

[0013] Respectively combining the initial hand-eye calibration relationship and the initial arm-eye calibration relationship with the online three-dimensional model, rendering based on the parameters of the eye camera, and generating a hand-eye calibration two-dimensional image and an arm-eye calibration two-dimensional image respectively;

[0014] The hand-eye calibration two-dimensional image and the arm-eye calibration two-dimensional image are respectively compared with the actual picture acquired by the camera, and the differences between the hand-eye calibration two-dimensional image and the arm-eye calibration two-dimensional image and their corresponding actual pictures are quantitatively evaluated, and the initial hand-eye calibration relationship and the initial arm-eye calibration relationship are optimized according to the differences to obtain an accurate hand-eye calibration relationship and an accurate arm-eye calibration relationship;

[0015] Based on the precise hand-eye calibration relationship in combination with the online three-dimensional model and the initial arm-arm calibration relationship, rendering is performed based on the parameters of the eye's camera to generate an arm-arm two-dimensional image;

[0016] The arm-arm two-dimensional image is compared with the actual picture acquired by the camera, the difference between the arm-arm two-dimensional image and the corresponding actual picture is quantitatively evaluated, and the initial arm-arm calibration relationship is optimized according to the difference to obtain an accurate arm-arm calibration relationship.

[0017] Optionally, the camera is an RGBD camera.

[0018] Optionally, the sequence configuration is generated by:

[0019] The robot sequence configuration is generated based on the current robot joint angles, connecting rods, cameras and connectors, environmental obstacle models, and the camera field of view and motion obstacle avoidance characteristics.

[0020] Optionally, the online three-dimensional model of each robot is established according to the online joint angle, kinematic calibration and three-dimensional model of each connecting rod of each robot, including:

[0021] According to the robot's kinematic parameters and online joint angles, the position coordinates of each link are calculated based on forward kinematics. Combined with the mesh model of each link, the robot's online three-dimensional model is generated for cropping based on the camera's intrinsic parameters.

[0022] Optionally, matching the online three-dimensional model with the three-dimensional point cloud of the online eye test of the robot to obtain an initial hand-eye calibration relationship, an initial arm-eye calibration relationship and an initial arm-arm calibration relationship includes:

[0023] A pose estimation method is adopted, based on the online three-dimensional model, combined with the online acquired image and the three-dimensional point cloud for matching, to obtain the pose estimation value of the three-dimensional model of the robot in the three-dimensional point cloud, that is, the initial hand-eye calibration relationship, the initial arm-eye calibration relationship and the initial arm-arm calibration relationship of the robot body relative to the camera.

[0024] Optionally, the respective two-dimensional images are compared with the respective corresponding actual images, and optimization is performed according to the difference obtained by the comparison, including:

[0025] For each two-dimensional image and its corresponding actual image, execute:

[0026] Perform semantic-cue-based segmentation on the actual images captured by the camera;

[0027] Replace the segmented background with the same color as the rendered 2D image background;

[0028] Image structure measurement indicators are used to evaluate image differences;

[0029] The particle swarm optimization method is used to perform sampling optimization in the area near the initial value.

[0030] Compared with the prior art, the present invention has at least the following beneficial effects:

[0031] In this embodiment, multiple robots are moved according to the configuration sequence for calibration to a configuration that is convenient for calibration. Then, an online three-dimensional model is established based on the relevant parameters of each robot. The two-dimensional image obtained based on the online three-dimensional model and the initial calibration relationship is compared with the actual image actually taken by the camera, and optimization is performed based on the comparison result to obtain an accurate calibration relationship between the camera and the robot hand and arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 This is a flow chart of the multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching of the present invention;

[0034] Figure 2 The initial arm-eye calibration rendering image of this embodiment of the present invention;

[0035] Figure 3 The actual arm-eye calibration image acquired by the camera of this embodiment of the present invention;

[0036] Figure 4 The comparison images are actually taken and rendered under the initial value of the arm-eye relationship in this embodiment of the present invention;

[0037] Figure 5 It is a comparison image actually shot and rendered under the final arm-eye calibration result in an embodiment of the present invention;

[0038] Figure 6 It is a comparison image actually shot and rendered under the final hand-eye calibration result in an embodiment of the present invention;

[0039] Figure 7 This is a comparison image of the actual shooting and rendering under the final arm-to-arm calibration result in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] In the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance; unless otherwise specified or explained, the term "multiple" refers to two or more; the terms "connected" and "fixed" should be understood in a broad sense, for example, "connected" can be a fixed connection, a detachable connection, an integral connection, or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0042] In the description of this specification, it should be understood that the directional words such as "upper" and "lower" described in the embodiments of the present invention are described at the angles shown in the drawings and should not be understood as limiting the embodiments of the present invention. In addition, in the context, it should also be understood that when it is mentioned that an element is connected to another element "upper" or "lower", it can not only be directly connected to another element "upper" or "lower", but also indirectly connected to another element "upper" or "lower" through an intermediate element.

[0043] like Figure 1 As shown, an embodiment of the present invention provides a multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching, comprising:

[0044] S1, controlling a multi-robot whose calibration relationship is to be confirmed to move in a configuration sequence for calibration; wherein the robot comprises an arm, a hand and an eye, the eye comprises a camera installed at the end of the robot and a camera installed externally for observing the arm, the relationship to be calibrated comprises a hand-eye calibration relationship, an arm-eye calibration relationship and an arm-arm calibration relationship, the hand-eye calibration relationship is the relationship between a camera installed at the end of a robot and the center coordinate system of the flange at the end of the robot, the arm-eye calibration relationship is the relationship between a camera installed externally and another robot base, and the arm-arm calibration relationship is the relationship between different robot bases;

[0045] S2, establishing an online three-dimensional model of each robot according to the online joint angle, kinematic calibration and three-dimensional model of each link of the robot;

[0046] S3, generating respective two-dimensional images according to the initial hand-eye calibration relationship, the initial arm-eye calibration relationship and the initial arm-arm calibration relationship in combination with the online three-dimensional model, comparing the respective two-dimensional images with the corresponding actual images, optimizing according to the differences obtained by the comparison, and obtaining accurate hand-eye calibration relationship, accurate arm-eye calibration relationship and accurate arm-arm calibration relationship.

[0047] In this embodiment, multiple robots are moved according to the configuration sequence for calibration to a configuration that is convenient for calibration. Then, an online three-dimensional model is established based on the relevant parameters of each robot. The two-dimensional image obtained based on the online three-dimensional model and the initial calibration relationship is compared with the actual image actually taken by the camera, and optimization is performed based on the comparison result to obtain an accurate calibration relationship between the camera and the robot hand and arm.

[0048] In some embodiments of the present invention, before S1, the following steps are included:

[0049] Select the hand-eye calibration relationship, arm-arm calibration relationship, and arm-eye calibration relationship calibration method according to the configuration of the robot and camera.

[0050] The specific calibration method is as follows:

[0051] (1) Hand-eye calibration: The object to be calibrated is a single robot system, whose RGBD camera (“eye”) is fixed to the robot end flange (“hand”) through a structural member. The calibration value is the position transformation relationship between the RGBD camera imaging coordinate system and the robot end flange center coordinate system.

[0052] (2) Arm-to-arm calibration: The object to be calibrated is a multi-robot system, where the RGBD camera is installed at the end of one of the robots. The calibration value is the position transformation relationship between the coordinate system of the base ("arm") of robot 1 and the coordinate system of the base ("arm") of robot 2.

[0053] (3) Arm-eye calibration: The object to be calibrated is a multi-robot system or a single-robot system, whose RGBD camera ("eye") is located outside the robot ("arm"), which can be located on a fixed mounting bracket or other robot end. The calibrated value is the position transformation relationship between the RGBD camera imaging coordinate system and the robot base coordinate system;

[0054] In this embodiment, robots 1, 2, RGBD cameras 1, and RGBD cameras 2 are used, wherein the robots all use UR5e robotic arms, and the RGBD cameras all use ZED Mini cameras. RGBD camera 1 is installed at the end of robot 2, and RGBD camera 2 is installed on the external fixed bracket of the robot. In order to achieve accurate calibration of the robot system, the values ​​to be calibrated are: the relationship between RGBD camera 1 and robot 2 (hand-eye calibration), the relationship between robot 2 and robot 1 (arm-arm calibration), and the relationship between RGBD camera 2 and robot 1 (arm-eye calibration). A complete and accurate motion relationship chain can be constructed based on the above calibration values ​​for precise manipulation tasks.

[0055] For S1, the configuration used for calibration is represented by multiple sets of robot joint angles. Factors that need to be considered include:

[0056] (1) Under multiple calibration configurations, the camera’s pose changes are as large as possible;

[0057] (2) Under multiple calibration configurations, the camera's observation range covers the robot body as much as possible;

[0058] (3) The differences in joint angles of multiple sets of calibration configurations should be as small as possible to improve the efficiency of the calibration process;

[0059] (4) During multiple calibration configurations and their switching, the robot is strictly prohibited from self-collision, collision with other robots, or collision with environmental objects.

[0060] This embodiment adopts an optimization algorithm to solve the problem. The above factors (1), (2), and (3) are used as optimization sub-goals, and their weight settings are reduced in sequence. (4) is used as a strict constraint of the algorithm. The configuration sequences of robots 1 and 2 for hand-eye calibration, arm-arm calibration, and arm-eye calibration are obtained. In hand-eye calibration, the observation target of camera 1 is robot 2; in arm-arm calibration, the observation target of camera 1 is robot 1; in arm-eye calibration, the observation target of camera 2 is robot 1.

[0061] For S2, the online joint angles, kinematic calibrations and 3D models of each robot’s links are used to reconstruct the current online 3D models of each robot, and the models can be cropped based on the camera’s intrinsic parameters when the initial calibration values ​​are available.

[0062] In this embodiment, the ROS system is used to read the information of robots 1 and 2, and the Trimesh library is used to perform online reconstruction and cropping of the robot's three-dimensional model:

[0063] (1) Use the ROS system to update the URDF file according to the kinematic calibration parameters of robots 1 and 2;

[0064] (2) Based on the updated URDF file, the ROS system is used to obtain the online joint angles of robots 1 and 2, and the ROS system TF module is used to calculate the online pose of each link coordinate system;

[0065] (3) Based on the updated URDF file, obtain the pose transformation relationship between the coordinate system of each link Mesh and the link coordinate system calculated by the TF module, and calculate the coordinate system of each link Mesh;

[0066] (4) The Trimesh library is used to read the mesh grids of each link of robots 1 and 2, and based on the coordinate system of each link mesh network, they are merged into online three-dimensional mesh models of robots 1 and 2 respectively;

[0067] (5) In the case where there are initial calibration values, the camera's external parameter matrix can be calculated. Based on the camera's internal parameter matrix, the planes of the four imaging edges can be calculated. The three-dimensional models of robots 1 and 2 are cropped in turn to obtain the part in the camera's field of view for subsequent three-dimensional matching.

[0068] (6) According to the configuration generated by S1, the robots are moved respectively to obtain the online three-dimensional models of each robot in the three calibration states.

[0069] In some embodiments of the present invention, the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship are obtained by:

[0070] The online 3D model is matched with the 3D point cloud of the robot's eye online test to obtain the initial hand-eye calibration relationship, the initial arm-eye calibration relationship and the initial arm-arm calibration relationship.

[0071] The three-dimensional point cloud and image measured online by the robot are matched with the reconstructed three-dimensional model to obtain the initial hand-eye or arm-eye posture relationship.

[0072] In this embodiment, the hand-eye and arm-eye postures are calibrated separately. For the case of whether or not the initial calibration value is obtained, the following matching method can be used:

[0073] (1) For the case of initial calibration values, obtain the cropped online 3D model of the robot relative to the camera coordinate system obtained by S2, and use the Refine network in traditional point cloud matching methods such as ICP and intelligent matching methods such as FoundationPose to estimate the pose of the 3D point cloud obtained by the RGBD camera according to the complexity of the scene to obtain the initial hand-eye and arm-eye pose calibration relationship;

[0074] (2) For the case of no initial calibration value. Manually extract at least 4 pairs of feature points from the robot 3D model and the 3D point cloud obtained by the RGBD camera, obtain the 3D coordinates of the feature points, use the ICP method to obtain the pose transformation matrix, and perform model pose transformation based on the matrix. Then, the method (1) can be used to obtain the hand-eye and arm-eye initial calibration relationship.

[0075] Of course, the initial hand-eye calibration relationship, the initial arm-eye calibration relationship, and the initial arm-arm calibration relationship may also be set values.

[0076] In some embodiments of the present invention, S3 includes:

[0077] S301, combining the initial hand-eye calibration relationship and the initial arm-eye calibration relationship with the online three-dimensional model, rendering based on the parameters of the eye camera, and generating a hand-eye calibration two-dimensional image and an arm-eye calibration two-dimensional image respectively;

[0078] S302, comparing the hand-eye calibration two-dimensional image and the arm-eye calibration two-dimensional image with the actual image acquired by the camera, quantitatively evaluating the differences between the hand-eye calibration two-dimensional image and the arm-eye calibration two-dimensional image and their corresponding actual images, optimizing the initial hand-eye calibration relationship and the initial arm-eye calibration relationship according to the differences, and obtaining the precise hand-eye calibration relationship and the precise arm-eye calibration relationship;

[0079] S303, based on the precise hand-eye calibration relationship combined with the online three-dimensional model and the initial arm-arm calibration relationship, rendering is performed based on the parameters of the eye camera to generate an arm-arm two-dimensional image;

[0080] S304, comparing the arm-to-arm two-dimensional image with the actual picture acquired by the camera, quantitatively evaluating the difference between the arm-to-arm two-dimensional image and its corresponding actual picture, optimizing the initial arm-to-arm calibration relationship according to the difference, and obtaining an accurate arm-to-arm calibration relationship.

[0081] For S301, the specific steps are as follows:

[0082] (1) For the hand-eye and arm-eye calibration types, calculate the position relationship of the camera relative to the robot. Under the hand-eye calibration type, the initial hand-eye calibration relationship of camera 1 relative to robot 2 is Read through the ROS system, is the initial calibration value; in the arm-eye calibration type, the initial arm-eye calibration relationship of camera 2 relative to robot 1 is the initial calibration value

[0083] (2) For both hand-eye and arm-eye calibration types, parameters are set according to the light source position and brightness in the current actual scene, and the background color of the environment is set to white. The Pyrender library is used to load the current online 3D model of the robot into the base coordinate system of the Pyrender virtual environment. or And the camera's internal parameters f x 、f y 、c x 、c y , respectively generate the current initial or The hand-eye calibration rendering image and the arm-eye calibration rendering image under the calibration parameters. In this embodiment, the initial arm-eye calibration rendering image is as follows: Figure 2 As shown, the actual image acquired by the camera is Figure 3 shown.

[0084] For S302, the specific steps are as follows:

[0085] (1) Preprocess the actual two-dimensional image acquired by the camera, use the GroundDINO method, use the prompt word "robot" to detect the robot target, combine the obtained detection frame with the SAM segmentation model, obtain the segmentation result of the robot, and remove the environmental background except the robot in the actual image based on the segmentation mask result;

[0086] (2) For the actual preprocessed 2D image and the 2D image rendered in step S301 (hand-eye calibration 2D image and arm-eye calibration 2D image), similarity evaluation is performed from three perspectives: brightness, contrast, and structure based on the SSIM structural measurement index, and the currently set and Image similarity under calibration parameters;

[0087] (3) and They are respectively expressed as tx1, ty1, tz1, r1, p1, y1 and tx2, ty2, tz2, r2, p2, y2, where txi, tyi, tzi are displacement components, ri, pi, yi are Euler angles, and i is 1 or 2. According to the deviation estimation, the search range of the optimization algorithm is determined. The particle swarm optimization algorithm (PSO) is used, and the optimization function calculation at each particle is implemented by step (2);

[0088] (4) Get the optimization result tx1 * ,ty1 * ,tz1 * 、r1 * 、p1 * ,y1* and tx2 * ,ty2 * ,tz2 * 、r2 * 、p2 * ,y2 * , respectively expressed as and That is, the precise hand-eye calibration relationship and the precise arm-eye calibration relationship. and The image generated in step (2) is manually compared with the image obtained in step (1) and evaluated based on the pixel-level error. In this embodiment, under the initial value of the arm-eye relationship, the actual photographed and rendered comparison images are as follows: Figure 4 As shown in the figure, under the final arm-eye calibration result, the comparison images of the actual shooting and rendering are as follows Figure 5 As shown in the figure, under the final hand-eye calibration result, the comparison image of the actual shooting and rendering is shown in the figure. Figure 6 shown.

[0089] For S303, the specific operations are as follows:

[0090] (1) For the arm-to-arm calibration type, calculate the pose relationship of the camera 1 fixed on the robot 2 relative to the base of the robot 1. Calculate and obtain the pose transformation relationship of the camera 1 relative to the robot 1 in Can be obtained through the ROS system, The hand-eye calibration result obtained by step (4) in S302 is used for robot 2

[0091] (2) For If there is no initial value, use step (4) to obtain The initial value of is then calculated based on the posture transformation relationship in step (1) to obtain The initial value of

[0092] (3) For In the case of an initial value, the pose transformation relationship in step (1) is used to calculate The initial value of is used, and the method of step (2) of S301 is adopted to obtain a rendered two-dimensional image.

[0093] For S304, the specific steps are as follows:

[0094] (1) Using the above method, obtain The precise calibration value of

[0095] (2) Based on the posture transformation relationship between camera 1 and robot 1, the “arm-arm” relative posture calibration value is calculated. In the final arm-to-arm calibration result, the actual shooting and rendering comparison images are as follows: Figure 7 shown.

[0096] In some embodiments of the invention, the camera is an RGBD camera.

[0097] In some embodiments of the invention, the sequence configuration is generated by:

[0098] The robot sequence configuration is generated based on the current robot joint angles, connecting rods, cameras and connectors, environmental obstacle models, and the camera field of view and motion obstacle avoidance characteristics.

[0099] In some embodiments of the present invention, an online three-dimensional model of each robot is established based on the online joint angles, kinematic calibrations and three-dimensional models of each connecting rod of each robot, including:

[0100] According to the robot's kinematic parameters and online joint angles, the position coordinates of each link are calculated based on forward kinematics. Combined with the mesh model of each link, the robot's online three-dimensional model is generated for cropping based on the camera's intrinsic parameters.

[0101] In some embodiments of the present invention, the online three-dimensional model is matched with the three-dimensional point cloud of the robot's eye online test to obtain an initial hand-eye calibration relationship, an initial arm-eye calibration relationship, and an initial arm-arm calibration relationship, including:

[0102] A pose estimation method is adopted, based on the online 3D model, combined with the online acquired image and 3D point cloud for matching, to obtain the pose estimation value of the robot's 3D model in the 3D point cloud, that is, the initial hand-eye calibration relationship, initial arm-eye calibration relationship and initial arm-arm calibration relationship of the robot body relative to the camera.

[0103] In some embodiments of the present invention, the two-dimensional images are compared with the actual images corresponding to the two images, and optimization is performed according to the difference obtained by the comparison, including:

[0104] For each two-dimensional image and its corresponding actual image, execute:

[0105] Perform semantic-cue-based segmentation on the actual images captured by the camera;

[0106] Replace the segmented background with the same color as the rendered 2D image background;

[0107] Image structure measurement indicators are used to evaluate image differences;

[0108] The particle swarm optimization method is used to perform sampling optimization in the area near the initial value.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-robot multi-mode arm-hand-eye relationship intelligent calibration method based on online reconstruction-rendering-matching, characterized in that: include: Controlling a multi-robot whose calibration relationship is to be confirmed to move in a configuration sequence for calibration; wherein the robot comprises an arm, a hand and an eye, the eye comprises a camera installed at the end of the robot and a camera installed externally for observing the arm, the relationship to be calibrated comprises a hand-eye calibration relationship, an arm-eye calibration relationship and an arm-arm calibration relationship, the hand-eye calibration relationship is a relationship between a camera installed at the end of one robot and a center coordinate system of a flange at the end of the robot, the arm-eye calibration relationship is a relationship between a camera installed externally and a base of another robot, and the arm-arm calibration relationship is a relationship between bases of different robots; Establish an online 3D model of each robot based on the online joint angle, kinematic calibration and 3D model of each link of the robot; According to the initial hand-eye calibration relationship, the initial arm-eye calibration relationship and the initial arm-arm calibration relationship, respective two-dimensional images are generated in combination with the online three-dimensional model, and the respective two-dimensional images are compared with the respective corresponding actual images. According to the differences obtained by the comparison, optimization is performed to obtain accurate hand-eye calibration relationship, accurate arm-eye calibration relationship and accurate arm-arm calibration relationship.

2. The method according to claim 1, characterized in that Before controlling the multi-robots with calibration relationships to be confirmed to move in the configuration sequence for calibration, the method includes: Select the hand-eye calibration relationship, arm-arm calibration relationship, and arm-eye calibration relationship calibration method according to the configuration of the robot and camera.

3. The method according to claim 1, characterized in that: The initial hand-eye calibration relationship, the initial arm-eye calibration relationship and the initial arm-arm calibration relationship are obtained by: The online three-dimensional model is matched with the three-dimensional point cloud of the online eye test of the robot to obtain an initial hand-eye calibration relationship, an initial arm-eye calibration relationship and an initial arm-arm calibration relationship.

4. The method according to claim 1, characterized in that The generating of respective two-dimensional images according to the initial hand-eye calibration relationship, the initial arm-eye calibration relationship and the initial arm-arm calibration relationship in combination with the online three-dimensional model, comparing the respective two-dimensional images with the respective corresponding actual images, and obtaining accurate hand-eye calibration relationship, accurate arm-eye calibration relationship and accurate arm-arm calibration relationship after optimizing according to the comparison difference, including: Respectively combining the initial hand-eye calibration relationship and the initial arm-eye calibration relationship with the online three-dimensional model, rendering based on the parameters of the eye camera, and generating a hand-eye calibration two-dimensional image and an arm-eye calibration two-dimensional image respectively; The hand-eye calibration two-dimensional image and the arm-eye calibration two-dimensional image are respectively compared with the actual picture acquired by the camera, and the differences between the hand-eye calibration two-dimensional image and the arm-eye calibration two-dimensional image and their corresponding actual pictures are quantitatively evaluated, and the initial hand-eye calibration relationship and the initial arm-eye calibration relationship are optimized according to the differences to obtain an accurate hand-eye calibration relationship and an accurate arm-eye calibration relationship; Based on the precise hand-eye calibration relationship in combination with the online three-dimensional model and the initial arm-arm calibration relationship, rendering is performed based on the parameters of the eye's camera to generate an arm-arm two-dimensional image; The arm-arm two-dimensional image is compared with the actual picture acquired by the camera, the difference between the arm-arm two-dimensional image and the corresponding actual picture is quantitatively evaluated, and the initial arm-arm calibration relationship is optimized according to the difference to obtain an accurate arm-arm calibration relationship.

5. The method according to claim 1, characterized in that The camera is an RGBD camera.

6. The method according to claim 1, characterized in that The sequence configuration is generated in the following manner: The robot sequence configuration is generated based on the current robot joint angles, connecting rods, cameras and connectors, environmental obstacle models, and the camera field of view and motion obstacle avoidance characteristics.

7. The method according to claim 1, characterized in that The online three-dimensional model of each robot is established according to the online joint angle, kinematic calibration and three-dimensional model of each connecting rod of each robot, including: According to the robot's kinematic parameters and online joint angles, the position coordinates of each link are calculated based on forward kinematics. Combined with the mesh model of each link, the robot's online three-dimensional model is generated for cropping based on the camera's intrinsic parameters.

8. The method according to claim 3, characterized in that The online three-dimensional model is matched with the three-dimensional point cloud of the online eye test of the robot to obtain an initial hand-eye calibration relationship, an initial arm-eye calibration relationship and an initial arm-arm calibration relationship, including: A pose estimation method is adopted, based on the online three-dimensional model, combined with the online acquired image and the three-dimensional point cloud for matching, to obtain the pose estimation value of the three-dimensional model of the robot in the three-dimensional point cloud, that is, the initial hand-eye calibration relationship, the initial arm-eye calibration relationship and the initial arm-arm calibration relationship of the robot body relative to the camera.

9. The method according to claim 1, characterized in that: Compare the two-dimensional images with the corresponding actual images, and perform optimization based on the differences obtained by the comparison, including: For each two-dimensional image and its corresponding actual image, execute: Perform semantic-cue-based segmentation on the actual images captured by the camera; Replace the segmented background with the same color as the rendered 2D image background; Image structure measurement indicators are used to evaluate image differences; The particle swarm optimization method is used to perform sampling optimization in the area near the initial value.

Images