A calibration method and system for irregular asymmetric tools connected to a manipulator based on stereo vision

Abstract

The present invention discloses a calibration method and system for an irregular asymmetric tool externally connected to a manipulator based on stereoscopic vision. The method calculates the relative movement transformation matrix of the manipulator end, in which the center point of the tool moves from the shooting position to the center of the target object and the direction of the manipulator end is aligned with the direction of the target object coordinate system. The relative movement is calculated according to the relative movement transformation matrix, and the manipulator is controlled to move again, and the compensation matrix is ​​calculated. The compensation matrix is ​​used to control the manipulator to perform relative movement, so that the tool can be directly aligned with the target object. The present invention solves the problem that the tool coordinate system of irregular asymmetric external tools in manipulator applications is difficult to calibrate.

Description

Technical Field

[0001] The present invention relates to the technical field of robot calibration, and more specifically, to a calibration method and system for an irregular asymmetric tool connected to a robot based on stereo vision. Background Art

[0002] In my country, industrial robots are widely used in manufacturing, not only in the automotive industry, but also in the production of space shuttles, military equipment, high-speed rail, and ballpoint pens. Compared with fixed-program and fixed-position robot applications, combined with machine vision, the position and direction of components can be detected and reported by machine vision, which can achieve robot positioning guidance and further improve the level of automation. In robot positioning guidance, the vision (camera) and tools (suction cups, welding guns, fixtures, pokers, etc.) are generally fixed to the end joints of the manipulator through flanges. Therefore, it is necessary to calibrate the position relationship between the vision (camera) and the manipulator, the manipulator and the tools, etc.

[0003] There are still some problems to be solved in the application of the calibration of the manipulator and tool coordinate system. The origin of the tool coordinate system is called the tool center point TCP (Tool Center Point), such as the center of the suction cup, the virtual center of the fixture, etc. The general robot system itself provides TCP calibration, which is achieved by manually aligning the center to a certain point and making several special gestures. There are also some other high-precision TCP calibration methods that are relatively mature. However, TCP calibration only aligns the tip position, and the conversion relationship between the tool coordinate system and the manipulator end coordinate system is only a translation transformation, not a rotation transformation, that is, the direction of the tool coordinate system obtained only through TCP calibration is the same as that of the manipulator end. In actual applications, it is not enough to align the TCP. There are many scenarios where the manipulator needs to grasp or align irregular and asymmetric targets, such as "D"-shaped holes, and there are also scenarios where the target orientation (normal) is not fixed. In addition to TCP (center point) alignment, the tool and the target need to be aligned in all directions.

[0004] The existing calibration methods do not solve the problem that the tool coordinate system of irregular asymmetric external tools in robot applications is difficult to calibrate. Therefore, there is an urgent need for a calibration method for irregular asymmetric external tools of robots based on stereo vision. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a calibration method and system for irregular and asymmetric tools connected to a robot based on stereo vision, which can accurately match and align the tool with the target, cope with complex scenes, and improve the level of automation in industrial applications.

[0006] The first aspect of the present invention provides a calibration method for an irregular asymmetric tool connected to a manipulator based on stereo vision, comprising the following steps:

[0007] Determine the direction and position of the target object coordinate system, and calibrate the tool center point and camera respectively;

[0008] Calculate the relative movement transformation matrix of the end of the manipulator so that the center point of the tool moves from the shooting position to the center of the target object and the direction of the end of the manipulator is aligned with the direction of the target object coordinate system;

[0009] Calculate the relative movement according to the relative movement transformation matrix, wherein the relative movement includes: a relative translation amount and a rotation amount;

[0010] The end of the manipulator performs the first relative motion according to the relative translation obtained, and the position of the end of the manipulator in the manipulator base coordinate system at this time is recorded as P1;

[0011] Control the manipulator to rotate around the center point of the tool so that the direction of the tool is aligned with the direction of the target object; and record the position of the end of the manipulator in the manipulator base coordinate system as P2;

[0012] Calculate the compensation matrix based on the poses P1 and P2;

[0013] The relative motion Euler angles are calculated using the compensation matrix, and the robot is controlled to perform relative motion, so that the tool and the target object can be directly aligned.

[0014] In this solution, the direction and position of the target object coordinate system are determined, and the tool center point and the camera are calibrated respectively, specifically including:

[0015] Determine the direction of the target object's coordinate system according to the target object's characteristics;

[0016] Determine the direction and position of the target object coordinate system in the camera coordinate system;

[0017] Calibrate the tool center point in the robot end coordinate system to obtain the calibration result;

[0018] Perform hand-eye calibration on the camera and the end of the manipulator to obtain the hand-eye calibration result;

[0019] In this solution, determining the direction of the target object coordinate system according to the target object characteristics includes:

[0020] The target object coordinate system includes the normal direction Vec_Z, the Y direction Vec_Y, and the X direction Vec_X, wherein the directions are the normal direction of the point cloud at the target center point or the normal direction extending from the target virtual center point; the Y direction is a direction perpendicular to the normal direction that is easy to grasp and has a fixed position on the target; the X direction is a direction perpendicular to both the normal direction and the Y direction.

[0021] In this scheme, the direction and position of the target object coordinate system in the camera coordinate system are expressed as:

[0022]

[0023] In this scheme, the tool center point is calibrated in the robot end coordinate system, and the calibration result is expressed as:

[0024] In this scheme, the relative movement transformation matrix of the manipulator end is calculated so that the tool center point moves from the shooting position to the center of the target object, and the direction of the manipulator end is aligned with the direction of the target object coordinate system;

[0025] The expression is: in Indicates the hand-eye calibration result between the camera and the end-point of the robot.

[0026] In this scheme, the relative movement is calculated according to the relative movement transformation matrix, and the relative movement includes: relative translation and rotation. The specific process is: the relative movement transformation matrix is ​​a rigid transformation, the first three rows and three columns are the rotation matrix R, and the last column is the translation t. The position change of the relative movement is the translation t in the transformation matrix. The rotation is calculated using the matlab function rotm2eul, and the directional rotation of the manipulator is represented by the three Euler angles Rz, Ry, and Rx.

[0027] In this solution, the compensation matrix is ​​calculated according to the postures P1 and P2, and the specific process is:

[0028] The (x, y, z) position of pose P1 is t1, where x, y, z represent the coordinates of pose P1, and the Euler angle pose is converted into the rotation matrix R1 using the matlab function eul2rotm; the position of pose P2 is t2, and the pose rotation matrix is ​​R2. Then the compensation matrix from pose P1 to pose P2 is:

[0029]

[0030] In this scheme, the compensation matrix is ​​used to calculate the relative motion Euler angle, and the manipulator is controlled to perform relative motion based on the end coordinate system, so that the tool and the target object can be directly aligned. The specific process is:

[0031] Determine the fixed conversion relationship: camera-end of robot Tool TCP-Manipulator end Tool orientation and end-of-arm rotation compensation

[0032] Get the position and direction of the target object captured by the camera

[0033] Calculate the relative motion transformation matrix of the manipulator end from the shooting position to the irregular asymmetric tool aligned with the target object:

[0034]

[0035] By using existing tools to calculate the Euler angles of the robot's motion and controlling the robot to perform relative motion according to the calculated Euler angles, the tool can be directly aligned with the target object.

[0036] A second aspect of the present invention provides a calibration system for an irregular asymmetric tool connected to a manipulator based on stereo vision, comprising a memory and a processor, wherein the memory comprises a calibration program for an irregular asymmetric tool connected to a manipulator based on stereo vision, and the calibration method program for an irregular asymmetric tool connected to a manipulator based on stereo vision is executed by the processor to implement the following steps:

[0037] Determine the direction and position of the target object coordinate system, and calibrate the tool center point and camera respectively;

[0038] Calculate the relative movement transformation matrix of the end of the manipulator so that the center point of the tool moves from the shooting position to the center of the target object and the direction of the end of the manipulator is aligned with the direction of the target object coordinate system;

[0039] Calculate the relative movement according to the relative movement transformation matrix, wherein the relative movement includes: a relative translation amount and a rotation amount;

[0040] The end of the manipulator performs the first relative motion according to the relative translation obtained, and the position of the end of the manipulator in the manipulator base coordinate system at this time is recorded as P1;

[0041] Control the manipulator to rotate around the center point of the tool so that the direction of the tool is aligned with the direction of the target object; and record the position of the end of the manipulator in the manipulator base coordinate system as P2;

[0042] Calculate the compensation matrix based on the poses P1 and P2;

[0043] The relative motion Euler angles are calculated using the compensation matrix, and the robot is controlled to perform relative motion, so that the tool and the target object can be directly aligned.

[0044] The present invention discloses a calibration method and system for an irregular asymmetric tool attached to a robot based on stereo vision. The method calculates the relative movement transformation matrix of the robot end so that the center point of the tool moves from a shooting position to the center of a target object and the direction of the robot end is aligned with the direction of the target object coordinate system. The relative movement is calculated according to the relative movement transformation matrix. The robot is controlled to move again, a compensation matrix is ​​calculated, and the compensation matrix is ​​used to control the robot to perform relative movement. The tool can be directly aligned with the target object, thereby solving the problem that the tool coordinate system of irregular asymmetric external tools in robot applications is difficult to calibrate. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flow chart of a calibration method for an irregular asymmetric tool connected to a manipulator based on stereo vision according to the present application is shown.

[0046] Figure 2 The flowchart of determining the direction and position of the target object coordinate system and calibrating the tool center point and the camera respectively in this application is shown.

[0047] Figure 3 A block diagram of a calibration system for an irregular asymmetric tool connected to a manipulator based on stereo vision is shown in the present application. DETAILED DESCRIPTION

[0048] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0050] Figure 1 A flow chart of a calibration method for an irregular asymmetric tool connected to a manipulator based on stereo vision according to the present application is shown.

[0051] like Figure 1 As shown, the present application discloses a calibration method for an irregular asymmetric tool connected to a manipulator based on stereo vision, comprising the following steps:

[0052] S102, determining the direction and position of the target object coordinate system, and calibrating the tool center point and the camera respectively;

[0053] S104, calculating a relative movement transformation matrix of the end of the manipulator so that the center point of the tool moves from the shooting position to the center of the target object and the direction of the end of the manipulator is aligned with the direction of the target object coordinate system;

[0054] S106, calculating the relative movement according to the relative movement transformation matrix, wherein the relative movement includes: a relative translation amount and a rotation amount;

[0055] S108, the end of the manipulator performs the first relative motion according to the obtained relative translation, and the position of the end of the manipulator in the manipulator base coordinate system at this time is recorded as P1;

[0056] S110, controlling the manipulator to rotate around the center point of the tool so that the direction of the tool is aligned with the direction of the target object; and recording the position of the end of the manipulator in the manipulator base coordinate system at this time as P2;

[0057] S112, calculating a compensation matrix according to the postures P1 and P2;

[0058] S114, using the compensation matrix to calculate the relative motion Euler angle, control the manipulator to perform relative motion, and directly align the tool with the target object.

[0059] It should be noted that the method described in the embodiment of the present invention needs to be calibrated by the tool center point (i.e. TCP calibration) and hand-eye calibration (i.e. hand-eye calibration of the camera and the end of the manipulator) before implementation, and the position, normal, and Y direction (easily grasped feature edges) of the target object in the camera coordinate system (i.e. point cloud) are converted to the position, normal, and Y direction in the tool coordinate system. The manipulator is allowed to move relatively. At this time, since the direction of the tool coordinate system is the same as the direction of the manipulator end coordinate system, the tool and the target may not be aligned. The manipulator needs to rotate relatively around the TCP to align the target. This relative rotation is called the rotation compensation between the tool coordinate system and the end of the manipulator. Since the tool is fixedly installed at the end of the manipulator, this rotation compensation is fixed.

[0060] It should be noted that the camera described in the embodiment of the present invention is a 3D camera, and the manipulator described in the present invention is not limited to a specific type and usage scenario. At the same time, the tool is an external tool, and there is no restriction on the type of tool. All irregular and asymmetric tools suitable for manipulators are suitable for the present invention.

[0061] According to an embodiment of the present invention, determining the direction and position of the target object coordinate system and calibrating the tool center point and the camera respectively specifically includes:

[0062] S202, determining the direction of the target object coordinate system according to the target object characteristics;

[0063] S204, determining the direction and position of the target object coordinate system in the camera coordinate system;

[0064] S206, calibrating the tool center point in the manipulator end coordinate system to obtain a calibration result;

[0065] S208, performing hand-eye calibration on the camera and the end of the manipulator to obtain a hand-eye calibration result;

[0066] It should be noted that, in a specific embodiment, the external tool at the end of the manipulator needs to be aligned with the target object in actual application. The direction of the tool coordinate system can be considered to be consistent with the direction of the target object coordinate system when aligned with the target object, and the direction of the target object's coordinate system can be specified according to the characteristics of the target object.

[0067] According to an embodiment of the present invention, determining the direction of the target object coordinate system based on the target object characteristics includes: the target object coordinate system includes a normal direction Vec_Z, a Y direction Vec_Y, and an X direction Vec_X, wherein the directions are the normal direction of the point cloud at the target center point or the normal direction extending from the target virtual center point; the Y direction is a direction on the target that is easy to grasp and has a fixed position and is perpendicular to the normal direction; the X direction is a direction perpendicular to both the normal direction and the Y direction.

[0068] According to an embodiment of the present invention, the direction and position of the target object coordinate system in the camera coordinate system are expressed as:

[0069]

[0070] It should be noted that the direction and position of the target coordinate system in the camera coordinate system (i.e., the point cloud coordinate system) are obtained by analyzing the point cloud. For example, if the target is a hole, the center of the hole is the origin of the target coordinate system, and the outward direction of the hole is the Z direction of the target coordinate system.

[0071] According to an embodiment of the present invention, the tool center point is calibrated in the manipulator end coordinate system, and the calibration result is expressed as:

[0072] It should be noted that the calibration result of the tool center point in the robot end coordinate system, that is, the TCP calibration result, can be performed by the robot system itself, and what is obtained is the position of the TCP in the robot end (End) coordinate system (because the tool center point is a point, there is no concept of direction).

[0073] According to an embodiment of the present invention, a relative movement transformation matrix of the end of the manipulator is calculated so that the center point of the tool moves from the shooting position to the center of the target object and the direction of the end of the manipulator is aligned with the direction of the target object coordinate system;

[0074] The expression is: in Indicates the hand-eye calibration result between the camera and the end-point of the robot.

[0075] It should be noted that the hand-eye calibration result between the camera and the end of the robot is obtained through the general hand-eye calibration process to obtain the position and direction of the camera coordinate system in the robot end (End) coordinate system. Because the camera is installed at the end of the robot, this relationship is fixed and can be obtained through calibration.

[0076] According to an embodiment of the present invention, the relative movement is calculated based on the relative movement transformation matrix, and the relative movement includes: relative translation and rotation. The specific process is: the relative movement transformation matrix is ​​a rigid transformation, the first three rows and three columns are the rotation matrix R, and the last column is the translation t. The position change of the relative movement is the translation t in the transformation matrix. The rotation is calculated using the matlab function rotm2eul, and the directional rotation of the manipulator is represented by the three Euler angles Rz, Ry, and Rx.

[0077] According to the embodiment of the present invention, the compensation matrix is ​​calculated according to the postures P1 and P2. The specific process is as follows: let the (x, y, z) position of posture P1 be t1, where x, y, z represent the coordinates of posture P1, and the Euler angle posture is converted into the rotation matrix R1 using the matlab function eul2rotm; the position of posture P2 is t2, and the posture rotation matrix is ​​R2. Then the compensation matrix from posture P1 to posture P2 is:

[0078]

[0079] According to an embodiment of the present invention, the relative motion Euler angle is calculated using a compensation matrix, and the manipulator is controlled to perform relative motion, so that the tool and the target object can be directly aligned. The specific process is as follows:

[0080] Determine the fixed conversion relationship: camera-end of robot Tool TCP-Manipulator end Tool orientation and end-of-arm rotation compensation

[0081] Get the position and direction of the target object captured by the camera

[0082] Calculate the relative motion transformation matrix of the manipulator end from the shooting position to the irregular asymmetric tool aligned with the target object:

[0083]

[0084] By using existing tools to calculate the Euler angles of the robot's motion and controlling the robot to perform relative motion according to the calculated Euler angles, the tool can be directly aligned with the target object.

[0085] It should be noted that in the embodiment of the present invention, when the manipulator is controlled to perform relative motion, it is performed based on the manipulator end coordinate system. The existing calculation tool, for example, calculates the relative motion Euler angle through the matlab function rotm2eul.

[0086] A second aspect of the present invention provides a calibration system for an irregular asymmetric tool connected to a manipulator based on stereo vision, comprising a memory and a processor, wherein the memory comprises a calibration program for an irregular asymmetric tool connected to a manipulator based on stereo vision, and the calibration method program for an irregular asymmetric tool connected to a manipulator based on stereo vision is executed by the processor to implement the following steps:

[0087] S102, determining the direction and position of the target object coordinate system, and calibrating the tool center point and the camera respectively;

[0088] S104, calculating a relative movement transformation matrix of the end of the manipulator so that the center point of the tool moves from the shooting position to the center of the target object and the direction of the end of the manipulator is aligned with the direction of the target object coordinate system;

[0089] S106, calculating the relative movement according to the relative movement transformation matrix, wherein the relative movement includes: a relative translation amount and a rotation amount;

[0090] S108, the end of the manipulator performs the first relative motion according to the obtained relative translation, and the position of the end of the manipulator in the manipulator base coordinate system at this time is recorded as P1;

[0091] S110, controlling the manipulator to rotate around the center point of the tool so that the direction of the tool is aligned with the direction of the target object; and recording the position of the end of the manipulator in the manipulator base coordinate system at this time as P2;

[0092] S112, calculating a compensation matrix according to the postures P1 and P2;

[0093] S114, using the compensation matrix to calculate the relative motion Euler angle, the manipulator is controlled to perform relative motion based on the end coordinate system, so that the tool can be directly aligned with the target object.

[0094] It should be noted that the method described in the embodiment of the present invention needs to be calibrated by the tool center point (i.e. TCP calibration) and hand-eye calibration (i.e. hand-eye calibration of the camera and the end of the manipulator) before implementation, and the position, normal, and Y direction (easily grasped feature edges) of the target object in the camera coordinate system (i.e. point cloud) are converted to the position, normal, and Y direction in the tool coordinate system. The manipulator is allowed to move relatively. At this time, since the direction of the tool coordinate system is the same as the direction of the manipulator end coordinate system, the tool and the target may not be aligned. The manipulator needs to rotate relatively around the TCP to align the target. This relative rotation is called the rotation compensation between the tool coordinate system and the end of the manipulator. Since the tool is fixedly installed at the end of the manipulator, this rotation compensation is fixed.

[0095] It should be noted that the camera described in the embodiment of the present invention is a 3D camera, and the manipulator described in the present invention is not limited to a specific type and usage scenario. At the same time, the tool is an external tool, and there is no restriction on the type of tool. All irregular and asymmetric tools suitable for manipulators are suitable for the present invention.

[0096] According to an embodiment of the present invention, determining the direction and position of the target object coordinate system and calibrating the tool center point and the camera respectively specifically includes:

[0097] S202, determining the direction of the target object coordinate system according to the target object characteristics;

[0098] S204, determining the direction and position of the target object coordinate system in the camera coordinate system;

[0099] S206, calibrating the tool center point in the manipulator end coordinate system to obtain a calibration result;

[0100] S208, performing hand-eye calibration on the camera and the end of the manipulator to obtain a hand-eye calibration result;

[0101] It should be noted that, in a specific embodiment, the external tool at the end of the manipulator needs to be aligned with the target object in actual application. The direction of the tool coordinate system can be considered to be consistent with the direction of the target object coordinate system when aligned with the target object, and the direction of the target object's coordinate system can be specified according to the characteristics of the target object.

[0102] According to an embodiment of the present invention, determining the direction of the target object coordinate system based on the target object characteristics includes: the target object coordinate system includes a normal direction Vec_Z, a Y direction Vec_Y, and an X direction Vec_X, wherein the directions are the normal direction of the point cloud at the target center point or the normal direction extending from the target virtual center point; the Y direction is a direction on the target that is easy to grasp and has a fixed position and is perpendicular to the normal direction; the X direction is a direction perpendicular to both the normal direction and the Y direction.

[0103] According to an embodiment of the present invention, the direction and position of the target object coordinate system in the camera coordinate system are expressed as:

[0104]

[0105] It should be noted that the direction and position of the target coordinate system in the camera coordinate system (i.e., the point cloud coordinate system) are obtained by analyzing the point cloud. For example, if the target is a hole, the center of the hole is the origin of the target coordinate system, and the outward direction of the hole is the Z direction of the target coordinate system.

[0106] According to an embodiment of the present invention, the tool center point is calibrated in the manipulator end coordinate system, and the calibration result is expressed as:

[0107] It should be noted that the calibration result of the tool center point in the robot end coordinate system, that is, the TCP calibration result, can be performed by the robot system itself, and what is obtained is the position of the TCP in the robot end (End) coordinate system (because the tool center point is a point, there is no concept of direction).

[0108] According to an embodiment of the present invention, a relative movement transformation matrix of the end of the manipulator is calculated so that the center point of the tool moves from the shooting position to the center of the target object and the direction of the end of the manipulator is aligned with the direction of the target object coordinate system;

[0109] The expression is: in Indicates the hand-eye calibration result between the camera and the end-point of the robot.

[0110] It should be noted that the hand-eye calibration result between the camera and the end of the robot is obtained through the general hand-eye calibration process to obtain the position and direction of the camera coordinate system in the robot end (End) coordinate system. Because the camera is installed at the end of the robot, this relationship is fixed and can be obtained through calibration.

[0111] According to an embodiment of the present invention, the relative movement is calculated based on the relative movement transformation matrix, and the relative movement includes: relative translation and rotation. The specific process is: the relative movement transformation matrix is ​​a rigid transformation, the first three rows and three columns are the rotation matrix R, and the last column is the translation t. The position change of the relative movement is the translation t in the transformation matrix. The rotation is calculated using the matlab function rotm2eul, and the directional rotation of the manipulator is represented by the three Euler angles Rz, Ry, and Rx.

[0112] According to the embodiment of the present invention, the compensation matrix is ​​calculated according to the postures P1 and P2. The specific process is as follows: let the (x, y, z) position of posture P1 be t1, where x, y, z represent the coordinates of posture P1, and the Euler angle posture is converted into the rotation matrix R1 using the matlab function eul2rotm; the position of posture P2 is t2, and the posture rotation matrix is ​​R2. Then the compensation matrix from posture P1 to posture P2 is:

[0113]

[0114] According to an embodiment of the present invention, the relative motion Euler angle is calculated using a compensation matrix, and the manipulator is controlled to perform relative motion, so that the tool and the target object can be directly aligned. The specific process is as follows:

[0115] Determine the fixed conversion relationship: camera-end of robot Tool TCP-Manipulator end Tool orientation and end-of-arm rotation compensation

[0116] Get the position and direction of the target object captured by the camera

[0117] Calculate the relative motion transformation matrix of the manipulator end from the shooting position to the irregular asymmetric tool aligned with the target object:

[0118]

[0119] By using existing tools to calculate the Euler angles of the robot's motion and controlling the robot to perform relative motion according to the calculated Euler angles, the tool can be directly aligned with the target object.

[0120] It should be noted that in the embodiment of the present invention, when the manipulator is controlled to perform relative motion, it is performed based on the manipulator end coordinate system. The existing calculation tool, for example, calculates the relative motion Euler angle through the matlab function rotm2eul.

[0121] The present invention discloses a calibration method and system for an irregular asymmetric tool attached to a robot based on stereo vision. The method calculates a relative movement transformation matrix of the robot end in which the center point of the tool moves from a shooting position to the center of a target object and the direction of the robot end is aligned with the direction of the target object coordinate system. The relative movement is calculated according to the relative movement transformation matrix. The robot is controlled to move again, a compensation matrix is ​​calculated, and the compensation matrix is ​​used to control the robot to perform relative movement, so that the tool and the target object can be directly aligned, thereby solving the problem that the tool coordinate system of irregular asymmetric external tools in robot applications is difficult to calibrate.

[0122] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0123] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0124] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0125] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0126] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. A calibration method for an irregular asymmetric tool connected to a manipulator based on stereo vision, characterized in that: The following steps are involved: Determine the direction and position of the target object coordinate system, and calibrate the tool center point and camera respectively; Calculate the relative movement transformation matrix of the end of the manipulator so that the center point of the tool moves from the shooting position to the center of the target object and the direction of the end of the manipulator is aligned with the direction of the target object coordinate system; The relative movement is calculated according to the relative movement transformation matrix, and the relative movement includes: relative translation and rotation; the specific process is: the relative movement transformation matrix is ​​a rigid transformation, the first three rows and three columns are the rotation matrix R, and the last column is the translation t. The position change of the relative movement is the translation t in the transformation matrix. The rotation is calculated using the matlab function rotm2eul, and the directional rotation of the manipulator is represented by three Euler angles Rz, Ry, and Rx; The end of the manipulator performs the first relative motion according to the relative translation obtained, and the position of the end of the manipulator in the manipulator base coordinate system at this time is recorded as P1; Control the manipulator to rotate around the center point of the tool so that the direction of the tool is aligned with the direction of the target object; and record the position of the end of the manipulator in the manipulator base coordinate system as P2; Calculate the compensation matrix based on the poses P1 and P2; the specific process is: The (x, y, z) position of pose P1 is t1, where x, y, and z represent the coordinates of pose P1. The Euler angle pose is converted into the rotation matrix R1 using the matlab function eul2rotm. The position of pose P2 is t2, and the pose rotation matrix is ​​R2. The compensation matrix from pose P1 to pose P2 is: The relative motion Euler angles are calculated using the compensation matrix, and the manipulator is controlled to perform relative motion, so that the tool and the target can be directly aligned; the process is: Determine the fixed conversion relationship: camera-end of robot Tool TCP-Manipulator end Tool orientation and end-of-arm rotation compensation Get the position and direction of the target object captured by the camera Calculate the relative motion transformation matrix of the manipulator end from the shooting position to the irregular asymmetric tool aligned with the target object: By using existing tools to calculate the Euler angles of the robot's motion and controlling the robot to perform relative motion according to the calculated Euler angles, the tool can be directly aligned with the target object.

2. The calibration method of a stereoscopic vision-based manipulator external irregular asymmetric tool according to claim 1, characterized in that: Determining the direction and position of the target object coordinate system and calibrating the tool center point and the camera respectively specifically includes: Determine the direction of the target object's coordinate system according to the target object's characteristics; Determine the direction and position of the target object coordinate system in the camera coordinate system; Calibrate the tool center point in the robot end coordinate system to obtain the calibration result; Perform hand-eye calibration between the camera and the end of the robot to obtain the hand-eye calibration result.

3. The calibration method of a stereoscopic vision-based manipulator external irregular asymmetric tool according to claim 2, characterized in that: The method of determining the direction of the target object coordinate system according to the target object characteristics includes: the target object coordinate system includes a normal direction Vec_Z, a Y direction Vec_Y, and an X direction Vec_X, wherein the direction is the normal direction of the point cloud at the target center point or the normal direction extending from the target virtual center point; the Y direction is a direction on the target that is easy to grasp and has a fixed position and is perpendicular to the normal direction; the X direction is a direction perpendicular to both the normal direction and the Y direction.

4. The calibration method of a stereoscopic vision-based manipulator external irregular asymmetric tool according to claim 2, characterized in that: The direction and position of the target object coordinate system in the camera coordinate system are expressed as: in, A column vector of length 3, which is a unit vector and represents the X-axis direction of the target object coordinate system in the camera coordinate system. A column vector of length 3, which is a unit vector and represents the y-axis direction of the target object coordinate system in the camera coordinate system. A column vector of length 3, which is a unit vector and represents the z-axis direction of the target object coordinate system in the camera coordinate system. A column vector of length 3, representing the position of the target object's coordinate system in the camera's coordinate system.

5. The calibration method of a stereoscopic vision-based manipulator external irregular asymmetric tool according to claim 2, characterized in that: The tool center point is calibrated in the robot end coordinate system, and the calibration result is expressed as: where t x ,t y ,t z They correspond to the x-coordinate, y-coordinate and z-coordinate of the tool center point in the robot end coordinate system respectively.

6. The calibration method of a robot arm externally connected to an irregular asymmetric tool based on stereo vision according to claim 1, characterized in that: Calculate the relative movement transformation matrix of the end of the manipulator so that the center point of the tool moves from the shooting position to the center of the target object and the direction of the end of the manipulator is aligned with the direction of the target object coordinate system; The expression is: in Indicates the hand-eye calibration result between the camera and the end-point of the robot.

7. A calibration system for an irregular asymmetric tool connected to a manipulator based on stereo vision, further characterized in that it comprises a memory and a processor, wherein the memory comprises a calibration program for an irregular asymmetric tool connected to a manipulator based on stereo vision, and when the calibration method program for an irregular asymmetric tool connected to a manipulator based on stereo vision is executed by the processor, the following steps are implemented: Determine the direction and position of the target object coordinate system, and calibrate the tool center point and camera respectively; Calculate the relative movement transformation matrix of the end of the manipulator so that the center point of the tool moves from the shooting position to the center of the target object and the direction of the end of the manipulator is aligned with the direction of the target object coordinate system; The relative movement is calculated according to the relative movement transformation matrix, and the relative movement includes: Relative translation and rotation; the specific process is: the relative movement transformation matrix is ​​a rigid transformation, the first three rows and three columns are the rotation matrix R, and the last column is the translation t. The position change of the relative movement is the translation t in the transformation matrix. The rotation is calculated using the matlab function rotm2eul. The direction rotation of the manipulator is represented by the three Euler angles Rz, Ry, and Rx. The end of the manipulator performs the first relative motion according to the relative translation obtained, and the position of the end of the manipulator in the manipulator base coordinate system at this time is recorded as P1; Control the manipulator to rotate around the center point of the tool so that the direction of the tool is aligned with the direction of the target object; and record the position of the end of the manipulator in the manipulator base coordinate system as P2; Calculate the compensation matrix based on the poses P1 and P2; the specific process is: The (x, y, z) position of pose P1 is t1, where x, y, and z represent the coordinates of pose P1. The Euler angle pose is converted into the rotation matrix R1 using the matlab function eul2rotm. The position of pose P2 is t2, and the pose rotation matrix is ​​R2. The compensation matrix from pose P1 to pose P2 is: The relative motion Euler angles are calculated using the compensation matrix, and the manipulator is controlled to perform relative motion, so that the tool and the target can be directly aligned; the process is: Determine the fixed conversion relationship: camera-end of robot Tool TCP-Manipulator end Tool orientation and end-of-arm rotation compensation Get the position and direction of the target object captured by the camera Calculate the relative motion transformation matrix of the manipulator end from the shooting position to the irregular asymmetric tool aligned with the target object: By using existing tools to calculate the Euler angles of the robot's motion and controlling the robot to perform relative motion according to the calculated Euler angles, the tool can be directly aligned with the target object.

Images