Hand-eye calibration method and device, computer readable storage medium and mechanical arm

Abstract

The application belongs to the technical field of mechanical arms, and particularly relates to a hand-eye calibration method and device, a computer readable storage medium and a mechanical arm. The method comprises: acquiring measurement data of two groups of first calibration objects and second calibration objects respectively; wherein the first calibration object is a calibration object arranged on a visual recognition station, the second calibration object is a calibration object arranged on a mechanical arm movement station, and the first calibration object and the second calibration object maintain a constant relative pose relationship; and performing hand-eye calibration according to the measurement data. In the above method, since the first calibration object arranged on the visual recognition station and the second calibration object arranged on the mechanical arm movement station always maintain a constant relative pose relationship, when the mechanical arm is far away from the camera, resulting in that there is no intersection in the detection space, the hand-eye calibration at a long distance can be performed according to the measurement data of the first calibration object and the second calibration object.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of mechanical arms, and particularly relates to a hand-eye calibration method and device, a computer readable storage medium, and a mechanical arm. BACKGROUND

[0002] In the hand-eye coordination application of a mechanical arm, the process of determining the relationship between the base coordinate system of the mechanical arm and the camera coordinate system is called hand-eye calibration. The existing hand-eye calibration method is usually based on the invariance of the pose of a calibration board in the base coordinate system and the camera coordinate system to establish a calibration equation.

[0003] However, when the mechanical arm is far away from the camera, the motion space of the mechanical arm and the detection space of the vision do not intersect, resulting in the inability to use the existing hand-eye calibration method for hand-eye calibration. SUMMARY

[0004] Therefore, the embodiments of the present application provide a hand-eye calibration method, device, computer readable storage medium, and mechanical arm to solve the problem that the existing hand-eye calibration method cannot be applied to a long-distance hand-eye calibration scene.

[0005] The first aspect of the embodiments of the present application provides a hand-eye calibration method, which can include:

[0006] Obtaining a first set of measurement data of a first calibration object and a second calibration object at a first set of measurement positions; obtaining a second set of measurement data of the first calibration object and the second calibration object at a second set of measurement positions; wherein the first calibration object is a calibration object arranged on a visual recognition station, the second calibration object is a calibration object arranged on a mechanical arm motion station, the first calibration object and the second calibration object maintain a constant relative pose relationship; the second set of measurement positions are positions obtained by moving the first set of measurement positions in a preset direction by a preset distance and / or rotating a preset angle;

[0007] Performing hand-eye calibration according to the first set of measurement data and the second set of measurement data.

[0008] In a specific implementation manner of the first aspect, the first set of measurement positions includes a first camera measurement position and a first mechanical arm measurement position.

[0009] Obtaining a first set of measurement data of a first calibration object and a second calibration object at a first set of measurement positions can include:

[0010] Measuring the pose of the first calibration object at the first camera measurement position;

[0011] Measuring the pose of the second calibration object at the first mechanical arm measurement position.

[0012] In an implementation form of the first aspect, the second set of measurement positions comprises a second camera measurement position and a second robot arm measurement position.

[0013] The obtaining the second set of measurement data of the first calibration object and the second calibration object at the second set of measurement positions can comprise:

[0014] measuring the pose of the first calibration object at the second camera measurement position;

[0015] measuring the pose of the second calibration object at the second robot arm measurement position.

[0016] In an implementation form of the first aspect, the performing hand-eye calibration according to the first set of measurement data and the second set of measurement data can comprise:

[0017] calculating a corresponding homogeneous transformation matrix according to the first set of measurement data and the second set of measurement data;

[0018] establishing a calibration equation between the camera and the end effector according to the corresponding homogeneous transformation matrix;

[0019] solving the calibration equation to obtain a hand-eye calibration result.

[0020] In an implementation form of the first aspect, the calculating a corresponding homogeneous transformation matrix according to the first set of measurement data and the second set of measurement data can comprise:

[0021] calculating a homogeneous transformation matrix from a first calibration object coordinate system to a camera coordinate system at the first camera measurement position according to the pose of the first calibration object at the first camera measurement position;

[0022] calculating a homogeneous transformation matrix from a second calibration object coordinate system to an end effector coordinate system at the first robot arm measurement position according to the pose of the second calibration object at the first robot arm measurement position;

[0023] calculating a homogeneous transformation matrix from a first calibration object coordinate system to a camera coordinate system at the second camera measurement position according to the pose of the first calibration object at the second camera measurement position;

[0024] calculating a homogeneous transformation matrix from a second calibration object coordinate system to an end effector coordinate system at the second robot arm measurement position according to the pose of the second calibration object at the second robot arm measurement position.

[0025] In an implementation form of the first aspect, the establishing a calibration equation between the camera and the end effector according to the corresponding homogeneous transformation matrix can comprise:

[0026] The calibration equation is established according to a homogeneous transformation matrix of a first calibration object coordinate system to a camera coordinate system at the first camera measurement position, a homogeneous transformation matrix of a second calibration object coordinate system to an end effector coordinate system at the first robot arm measurement position, a homogeneous transformation matrix of the first calibration object coordinate system to the camera coordinate system at the second camera measurement position, and a homogeneous transformation matrix of the second calibration object coordinate system to the end effector coordinate system at the second robot arm measurement position.

[0027] A second aspect of the embodiment of the application provides a hand-eye calibration device, which can include:

[0028] The measurement data acquisition module is configured to acquire a first set of measurement data of the first calibration object and the second calibration object at a first set of measurement positions, and acquire a second set of measurement data of the first calibration object and the second calibration object at a second set of measurement positions, wherein the first calibration object is a calibration object arranged on a visual recognition station, the second calibration object is a calibration object arranged on a robot arm movement station, the first calibration object and the second calibration object maintain a constant relative pose relationship, and the second set of measurement positions are positions obtained by moving the first set of measurement positions by a preset distance in a preset direction and / or rotating by a preset angle.

[0029] The hand-eye calibration module is configured to perform hand-eye calibration according to the first set of measurement data and the second set of measurement data.

[0030] In a specific implementation manner of the second aspect, the measurement data acquisition module can include:

[0031] The first measurement data acquisition unit is configured to acquire a first set of measurement data of the first calibration object and the second calibration object at a first set of measurement positions.

[0032] The second measurement data acquisition unit is configured to acquire a second set of measurement data of the first calibration object and the second calibration object at a second set of measurement positions.

[0033] In a specific implementation manner of the second aspect, the first set of measurement positions includes a first camera measurement position and a first robot arm measurement position.

[0034] The first measurement data acquisition unit can include:

[0035] The first pose measurement subunit is configured to measure a pose of the first calibration object at the first camera measurement position.

[0036] The second pose measurement subunit is configured to measure a pose of the second calibration object at the first robot arm measurement position.

[0037] In an implementation form of the second aspect, the second set of measurement positions comprises a second camera measurement position and a second robot arm measurement position.

[0038] The second measurement data obtaining unit can comprise:

[0039] a third pose measurement sub-unit configured to measure the pose of the first calibration object at the second camera measurement position;

[0040] a fourth pose measurement sub-unit configured to measure the pose of the second calibration object at the second robot arm measurement position.

[0041] In an implementation form of the second aspect, the hand-eye calibration module can comprise:

[0042] a matrix calculation unit configured to calculate corresponding homogeneous transformation matrices according to the first set of measurement data and the second set of measurement data;

[0043] an equation establishing unit configured to establish a calibration equation between the camera and the end effector according to the corresponding homogeneous transformation matrices;

[0044] an equation solving unit configured to solve the calibration equation to obtain a hand-eye calibration result.

[0045] In an implementation form of the second aspect, the matrix calculation unit can comprise:

[0046] a first matrix calculation sub-unit configured to calculate a homogeneous transformation matrix from a first calibration object coordinate system to a camera coordinate system at the first camera measurement position according to the pose of the first calibration object at the first camera measurement position;

[0047] a second matrix calculation sub-unit configured to calculate a homogeneous transformation matrix from a second calibration object coordinate system to an end effector coordinate system at the first robot arm measurement position according to the pose of the second calibration object at the first robot arm measurement position;

[0048] a third matrix calculation sub-unit configured to calculate a homogeneous transformation matrix from a first calibration object coordinate system to a camera coordinate system at the second camera measurement position according to the pose of the first calibration object at the second camera measurement position;

[0049] a fourth matrix calculation sub-unit configured to calculate a homogeneous transformation matrix from a second calibration object coordinate system to an end effector coordinate system at the second robot arm measurement position according to the pose of the second calibration object at the second robot arm measurement position.

[0050] In an implementation form of the second aspect, the equation establishing unit can comprise:

[0051] The equation establishing sub-unit is configured to establish the calibration equation according to a homogeneous transformation matrix of a first calibration object coordinate system to a camera coordinate system at the first camera measurement position, a homogeneous transformation matrix of a second calibration object coordinate system to an end effector coordinate system at the first robot arm measurement position, a homogeneous transformation matrix of the first calibration object coordinate system to the camera coordinate system at the second camera measurement position, and a homogeneous transformation matrix of the second calibration object coordinate system to the end effector coordinate system at the second robot arm measurement position.

[0052] A third aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of any of the hand-eye calibration methods.

[0053] A fourth aspect of the embodiments of the present application provides a robot arm, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of any of the hand-eye calibration methods when executing the computer program.

[0054] A fifth aspect of the embodiments of the present application provides a computer program product, which, when executed on a robot arm, causes the robot arm to perform the steps of any of the hand-eye calibration methods.

[0055] Compared with the prior art, the embodiments of the present application have the beneficial effects that: the embodiments of the present application acquire measurement data of two groups of first calibration objects and second calibration objects respectively; the first calibration objects are calibration objects arranged on a visual recognition station, the second calibration objects are calibration objects arranged on a robot arm movement station, and the first calibration objects and the second calibration objects maintain a constant relative pose relationship; and hand-eye calibration is performed according to the measurement data. In the embodiments of the present application, since the first calibration objects arranged on the visual recognition station and the second calibration objects arranged on the robot arm movement station always maintain a constant relative pose relationship, when the robot arm is far away from the camera and the detection space does not have an intersection, remote hand-eye calibration can be performed according to the measurement data of the first calibration objects and the second calibration objects, which has strong usability and practicality. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0057] Figure 1 It is a schematic diagram of a common hand-eye calibration scene.

[0058] Figure 2 An embodiment flow chart of a hand-eye calibration method in embodiments of the present application;

[0059] Figure 3 A schematic diagram of a robot application scenario in embodiments of the present application;

[0060] Figure 4 A schematic flow chart of a measurement data acquisition process;

[0061] Figure 5 A schematic diagram of a first calibration object and a second calibration object position change;

[0062] Figure 6 A schematic flow chart of a hand-eye calibration process;

[0063] Figure 7 An embodiment structure diagram of a hand-eye calibration device in embodiments of the present application;

[0064] Figure 8 A schematic block diagram of a robot in embodiments of the present application. DETAILED DESCRIPTION

[0065] In order to make the purposes, characteristics and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the embodiments described below are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0066] It should be understood that when used in the specification and the appended claims, the term “comprising” indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0067] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clearly indicated by the context, the singular forms “a”, “an” and “the” are intended to include the plural forms.

[0068] It should be further understood that the term “and / or” used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0069] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0070] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0071] In the hand-eye coordination application of robotic arms, the process of determining the relationship between the base coordinate system of the robotic arm and the camera coordinate system is called hand-eye calibration.

[0072] Understandably, in existing technologies, cameras and robotic arms are typically positioned close together, resulting in an overlap between the robotic arm's motion space and the vision detection space. Therefore, the pose invariance of the calibration plate in both the base coordinate system and the camera coordinate system can be used to establish calibration equations. For specific hand-eye calibration scenarios, please refer to... Figure 1 A common calibration scenario in existing technologies is eye-in-hand, such as... Figure 1 As shown in (a), the camera is mounted on the end effector of the robotic arm. Another common calibration scenario is eye-to-hand, such as... Figure 1 As shown in (b), the camera is mounted in a fixed position, while the calibrator is mounted on the end effector of the robotic arm.

[0073] In practical use, after the camera identifies the corresponding object to be operated on, the robotic arm can perform corresponding operations based on the camera's recognition results. However, between obtaining the camera's recognition results and performing the corresponding operation on the object, the robotic arm may need to perform certain preliminary operations. For example, if the object to be operated on is a package, the robotic arm needs to affix a label to the package. Between obtaining the camera's recognition results and performing the labeling operation, the robotic arm needs to first print the label corresponding to the package. To ensure that the preliminary operations are completed when the package arrives at the robotic arm's workstation, the camera and robotic arm can be set up at a distance, allowing the robotic arm sufficient time to perform the preliminary operations after obtaining the camera's recognition results.

[0074] However, when the camera is far away from the robot arm, the motion space of the robot arm and the detection space of the vision do not intersect, so that the hand-eye calibration method in the prior art cannot be used for hand-eye calibration.

[0075] Therefore, the embodiments of the present application provide a hand-eye calibration method, device, computer readable storage medium and robot arm. According to the embodiments of the present application, since the first calibration object arranged on the vision recognition station and the second calibration object arranged on the robot arm motion station always maintain a constant relative pose relationship, when the robot arm is far away from the camera and the detection space does not intersect, the long-distance hand-eye calibration can be performed according to the measurement data of the first calibration object and the second calibration object, which has strong usability and practicality.

[0076] It should be noted that the execution subject of the method of the present application is a robot arm, specifically, it can be a common multi-joint robot arm or other industrial robot arm.

[0077] Referring to Figure 2 An embodiment of the hand-eye calibration method in the embodiments of the present application can include:

[0078] In step S201, the measurement data of two or more groups of first calibration objects and second calibration objects is obtained respectively.

[0079] In the embodiments of the present application, the measurement data of two or more groups of first calibration objects and second calibration objects can be obtained respectively. The first calibration object is a calibration object arranged on a vision recognition station, and the second calibration object is a calibration object arranged on a robot arm motion station. The first calibration object and the second calibration object always maintain a constant relative pose relationship.

[0080] Specifically, the application scenario of the robot arm shown in Figure 3 The vision recognition station can be installed with a camera, and the robot arm motion station can be installed with a robot arm. After the camera on the vision station recognizes the object to be operated, the detection result can be transmitted to the robot arm on the robot arm motion station through a pre-set communication module or a terminal device connected with the camera and the robot arm at the same time. At this time, the robot arm can perform corresponding operation on the object to be operated according to the recognition result of the camera.

[0081] It can be understood that in the embodiments of the present application, calibration objects for hand-eye calibration can be arranged on the vision recognition station and the robot arm motion station respectively. Here, the calibration objects can be arranged on fixed reference objects.

[0082] In the embodiments of the present application, the first calibration object and the second calibration object can be arranged at different positions of the corresponding fixed calibration object, but the relative pose relationship between the first calibration object and the second calibration object is always constant, so that the calibration equation can be established through the constant relative pose relationship.

[0083] It should be noted that the solution of the calibration equation can be realized by obtaining only two groups of measurement data, but in order to improve the accuracy of the calibration result, more than two groups of measurement data can be obtained according to actual needs, the measurement data can be combined in pairs, the calibration equation can be solved multiple times, and the accurate solution of the calibration equation can be obtained according to the solutions of the multiple calibration equations. For example, five groups of measurement data can be obtained, and ten groups of measurement data can be formed by combining the measurement data in pairs, ten solutions of the calibration equation can be obtained, and then the weighted average of the ten solutions can be taken as the accurate solution of the calibration equation.

[0084] Please refer to Figure 4 , the step S201 can specifically include the following processes:

[0085] The step S2011, obtaining a first group of measurement data of the first calibration object and the second calibration object at a first group of measurement positions.

[0086] The first group of measurement positions can include a first camera measurement position and a first robot arm measurement position.

[0087] It can be understood that the first calibration object and the second calibration object can be arranged on a fixed reference object, the position of the first calibration object on the corresponding fixed reference object can be recorded as a camera measurement position, and the position of the second calibration object on the corresponding fixed reference object can be recorded as a robot arm measurement position.

[0088] It can be understood that since the relative pose relationship between the first calibration object and the second calibration object needs to be kept constant, when the camera measurement position is determined, the corresponding robot arm measurement position can also be determined accordingly. In order to facilitate representation, the camera measurement position and the robot arm measurement position corresponding to the camera measurement position can be referred to as a group of measurement positions, and the data measured at a group of measurement positions can be referred to as a group of measurement data.

[0089] In the embodiments of the present application, the pose of the first calibration object at the first camera measurement position and the pose of the second calibration object at the first robot arm measurement position can be measured to obtain a first group of measurement data at a first group of measurement positions.

[0090] In the embodiments of the present application, the pose of the first calibration object at the first camera measurement position can be measured by a camera on a visual recognition station, for example, the first calibration object can be image collected by the camera on the visual recognition station, and the pose of the first calibration object at the first camera measurement position can be identified from the collected image.

[0091] In the embodiment of the present application, the pose of the second calibration object at the second camera measurement position can also be measured by a camera on the robot arm movement station. Specifically, the camera can be pre-installed at the end effector to capture images of the second calibration object, and the pose of the second calibration object at the first robot arm measurement position can be identified from the captured images.

[0092] In step S2012, a second set of measurement data of the first calibration object and the second calibration object at a second set of measurement positions is obtained.

[0093] The second set of measurement positions can include the first camera measurement position and the first robot arm measurement position.

[0094] In the embodiment of the present application, the positions of the first calibration object and the second calibration object can be changed according to a preset method, so that the first calibration object is located at a second camera measurement position different from the first camera measurement position, and the second calibration object is located at a second robot arm measurement position different from the first robot arm measurement position.

[0095] In one possible embodiment, the first calibration object and the second calibration object can be moved by a preset distance in a preset direction. For example, as shown in FIG. 8(a), the first calibration object and the second calibration object can be moved horizontally to the right by 5 cm, and the position of the first calibration object is recorded as the second camera measurement position, and the position of the second calibration object is recorded as the second robot arm measurement position. Figure 5

[0096] In another possible embodiment, the first calibration object and the second calibration object can be moved by a preset distance in a preset direction, and then rotated by a preset angle. For example, as shown in FIG. 8(b), the first calibration object and the second calibration object can be moved horizontally to the right by 5 cm, and then rotated clockwise by 45 degrees, and the position of the first calibration object is recorded as the second camera measurement position, and the position of the second calibration object is recorded as the second robot arm measurement position. Figure 5

[0097] In the embodiment of the present application, the pose of the first calibration object at the second camera measurement position can be measured by a camera on the vision recognition station, and the pose of the second calibration object at the second robot arm measurement position can be measured by a camera on the robot arm movement station, so as to obtain the second set of measurement data at the second set of measurement positions.

[0098] In step S202, hand-eye calibration is performed according to the measurement data.

[0099] For ease of understanding, the hand-eye calibration principle of the embodiment of the present application will be described first.

[0100] ​​Since the first calibration object and the second calibration object always maintain a constant relative pose relationship, the homogeneous transformation matrix of the first calibration object coordinate system to the base coordinate system can be converted by the homogeneous transformation matrix of the first calibration object coordinate system to the second calibration object coordinate system, and can be specifically as follows:

[0101]

[0102] wherein, is the homogeneous transformation matrix of the first calibration object coordinate system to the second calibration object coordinate system to the base coordinate system, is the homogeneous transformation matrix of the first calibration object coordinate system to the camera coordinate system, is the homogeneous transformation matrix of the camera coordinate system to the base coordinate system, is the homogeneous transformation matrix of the end effector coordinate system to the base coordinate system, is the homogeneous transformation matrix of the second calibration object coordinate system to the end effector coordinate system.

[0103] The above equation is moved to obtain:

[0104]

[0105] wherein, is the inverse of the homogeneous transformation matrix of the second calibration object coordinate system to the end effector coordinate system, is the inverse of the homogeneous transformation matrix of the end effector coordinate system to the base coordinate system.

[0106] In the embodiment of the present application, based on the constant value The measurement positions of the first calibration object and the second calibration object can be changed to obtain more than two groups of measurement data, and the calibration equation can be established.

[0107] Please refer to Figure 6 , the step S202 can specifically include the following processes:

[0108] The step S2021, according to the first group of measurement data and the second group of measurement data, the corresponding homogeneous transformation matrix is calculated.

[0109] Wherein, the homogeneous transformation matrix to be calculated can include the homogeneous transformation matrix of the first calibration object coordinate system to the camera coordinate system at the first camera measurement position, the homogeneous transformation matrix of the second calibration object coordinate system to the end effector coordinate system at the first robot arm measurement position, the homogeneous transformation matrix of the first calibration object coordinate system to the camera coordinate system at the second camera measurement position, and the homogeneous transformation matrix of the second calibration object coordinate system to the end effector coordinate system at the second robot arm measurement position.

[0110] Specifically, the corresponding homogeneous transformation matrix can be calculated according to the pose of the calibration object. For example, the homogeneous transformation matrix from the first calibration object coordinate system to the camera coordinate system can be calculated according to the pose of the first calibration object in the first calibration object coordinate system and the pose of the first calibration object in the camera coordinate system.

[0111] In the embodiments of the present application, the homogeneous transformation matrix from the first calibration object coordinate system to the camera coordinate system at the first camera measurement position can be calculated according to the pose of the first calibration object at the first camera measurement position. The homogeneous transformation matrix from the second calibration object coordinate system to the end effector coordinate system at the first robot measurement position can be calculated according to the pose of the second calibration object at the first robot measurement position. The homogeneous transformation matrix from the first calibration object coordinate system to the camera coordinate system at the second camera measurement position can be calculated according to the pose of the first calibration object at the second camera measurement position. The homogeneous transformation matrix from the second calibration object coordinate system to the end effector coordinate system at the second robot measurement position can be calculated according to the pose of the second calibration object at the second robot measurement position.

[0112] In step S2022, a calibration equation between the camera and the end effector is established according to the corresponding homogeneous transformation matrix.

[0113] In the embodiments of the present application, the calibration equation between the camera and the end effector can be established according to the homogeneous transformation matrix from the first calibration object coordinate system to the camera coordinate system at the first camera measurement position, the homogeneous transformation matrix from the second calibration object coordinate system to the end effector coordinate system at the first robot measurement position, the homogeneous transformation matrix from the first calibration object coordinate system to the camera coordinate system at the second camera measurement position, and the homogeneous transformation matrix from the second calibration object coordinate system to the end effector coordinate system at the second robot measurement position. Specifically, the following calibration equation can be established:

[0114]

[0115] wherein, is the inverse of the homogeneous transformation matrix from the second calibration object coordinate system to the end effector coordinate system at the first robot measurement position in the first set of measurement data, is the inverse of the homogeneous transformation matrix from the second calibration object coordinate system to the end effector coordinate system at the second robot measurement position in the second set of measurement data, is the inverse of the homogeneous transformation matrix from the second calibration object coordinate system to the end effector coordinate system at the second robot measurement position in the second set of measurement data, is the inverse of the homogeneous transformation matrix from the second calibration object coordinate system to the end effector coordinate system at the second robot measurement position in the second set of measurement data, is the homogeneous transformation matrix from the camera coordinate system to the base coordinate system to be solved, is the inverse of the homogeneous transformation matrix from the end effector coordinate system to the base coordinate system, is the inverse of the homogeneous transformation matrix from the end effector coordinate system to the base coordinate system, The equation can be designed in the development process of the robot arm or can be obtained through pre-measurement calculation.

[0116] In the embodiment of the application, the above formula can be converted into the standard AX = XB form, as follows:

[0117]

[0118] wherein,

[0119] In step S2023, the calibration equation is solved to obtain the hand-eye calibration result.

[0120] In the embodiment of the application, after the calibration equation is established, the calibration equation can be solved to obtain the hand-eye calibration result.

[0121] It can be understood that there are various solutions to the calibration equation in mathematics, and any one of the solutions can be selected for solving according to actual conditions, and the embodiment of the application does not make a specific limitation.

[0122] In one possible embodiment, the calibration equation can be solved by using a known solving library to obtain the hand-eye calibration result. For example, the Eigen library can be used to solve the calibration equation.

[0123] In another possible embodiment, the rotation matrix can be estimated first, and then the translation vector is solved. For example, the Shui method or the Tsai method can be used to solve the calibration equation.

[0124] In another possible embodiment, the rotation matrix and the translation vector can be solved simultaneously. For example, the Levenberg-Marquardt method or the Daniilidis method can be used to solve the calibration equation.

[0125] It can be understood that after the hand-eye calibration result is obtained, the robot arm can also be controlled to perform corresponding operations according to the camera recognition result. For example, the robot arm can perform operations such as grabbing, labeling, spraying, assembling, etc. on the object to be operated according to the camera recognition result.

[0126] In summary, the embodiment of the present application respectively acquires measurement data of two groups of first calibration objects and second calibration objects, wherein the first calibration objects are calibration objects arranged on a visual recognition station, the second calibration objects are calibration objects arranged on a mechanical arm movement station, the first calibration objects and the second calibration objects maintain a constant relative pose relationship, and hand-eye calibration is performed according to the measurement data. In the embodiment of the present application, since the first calibration objects arranged on the visual recognition station and the second calibration objects arranged on the mechanical arm movement station always maintain a constant relative pose relationship, when the mechanical arm is far away from the camera and the detection space does not exist intersection, the hand-eye calibration at a long distance can be performed according to the measurement data of the first calibration objects and the second calibration objects, which has strong usability and practicality.

[0127] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0128] According to the hand-eye calibration method described in the above embodiment, Figure 7 An embodiment structure diagram of a hand-eye calibration device provided by the embodiment of the present application is shown.

[0129] In the embodiment of the present application, a hand-eye calibration device can include:

[0130] The measurement data acquisition module 701 is configured to respectively acquire measurement data of two or more groups of first calibration objects and second calibration objects, wherein the first calibration objects are calibration objects arranged on a visual recognition station, the second calibration objects are calibration objects arranged on a mechanical arm movement station, and the first calibration objects and the second calibration objects maintain a constant relative pose relationship.

[0131] The hand-eye calibration module 702 is configured to perform hand-eye calibration according to the measurement data.

[0132] In a specific implementation manner of the embodiment of the present application, the measurement data acquisition module can include:

[0133] The first measurement data acquisition unit is configured to acquire a first group of measurement data of the first calibration objects and the second calibration objects at a first group of measurement positions.

[0134] The second measurement data acquisition unit is configured to acquire a second group of measurement data of the first calibration objects and the second calibration objects at a second group of measurement positions.

[0135] In a specific implementation manner of the embodiment of the present application, the first group of measurement positions includes a first camera measurement position and a first mechanical arm measurement position.

[0136] The first measurement data acquisition unit can include:

[0137] A first pose measurement sub-unit configured to measure a pose of the first calibration object at the first camera measurement position.

[0138] A second pose measurement sub-unit configured to measure a pose of the second calibration object at the first robot measurement position.

[0139] In a specific implementation manner of the embodiment of the present application, the second group of measurement positions includes a second camera measurement position and a second robot measurement position.

[0140] The second measurement data acquisition unit can include:

[0141] A third pose measurement sub-unit configured to measure a pose of the first calibration object at the second camera measurement position.

[0142] A fourth pose measurement sub-unit configured to measure a pose of the second calibration object at the second robot measurement position.

[0143] In a specific implementation manner of the embodiment of the present application, the hand-eye calibration module can include:

[0144] A matrix calculation unit configured to calculate corresponding homogeneous transformation matrices according to the first group of measurement data and the second group of measurement data.

[0145] An equation establishment unit configured to establish a calibration equation between the camera and the end effector according to the corresponding homogeneous transformation matrices.

[0146] An equation solving unit configured to solve the calibration equation to obtain a hand-eye calibration result.

[0147] In a specific implementation manner of the embodiment of the present application, the matrix calculation unit can include:

[0148] A first matrix calculation sub-unit configured to calculate a homogeneous transformation matrix from a first calibration object coordinate system at the first camera measurement position to a camera coordinate system according to the pose of the first calibration object at the first camera measurement position.

[0149] A second matrix calculation sub-unit configured to calculate a homogeneous transformation matrix from a second calibration object coordinate system at the first robot measurement position to an end effector coordinate system according to the pose of the second calibration object at the first robot measurement position.

[0150] A third matrix calculation sub-unit configured to calculate a homogeneous transformation matrix from a first calibration object coordinate system at the second camera measurement position to a camera coordinate system according to the pose of the first calibration object at the second camera measurement position.

[0151] The fourth matrix calculation sub-unit is configured to calculate a homogeneous transformation matrix from the second calibration object coordinate system at the second robot arm measurement position to the end effector coordinate system according to the pose of the second calibration object at the second robot arm measurement position.

[0152] In a specific implementation process of the embodiment of the present application, the equation establishing unit can include:

[0153] The equation establishing sub-unit is configured to establish the calibration equation according to the homogeneous transformation matrix from the first calibration object coordinate system at the first camera measurement position to the camera coordinate system, the homogeneous transformation matrix from the second calibration object coordinate system at the first robot arm measurement position to the end effector coordinate system, the homogeneous transformation matrix from the first calibration object coordinate system at the second camera measurement position to the camera coordinate system, and the homogeneous transformation matrix from the second calibration object coordinate system at the second robot arm measurement position to the end effector coordinate system.

[0154] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described apparatuses, modules and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0155] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can refer to the relevant description of other embodiments.

[0156] Figure 8 A schematic block diagram of a robot arm is shown, and only parts related to the embodiments of the present application are shown for the convenience of description.

[0157] As Figure 8 shown, the robot arm 8 of this embodiment includes a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. The processor 80 implements the steps in each of the above hand-eye calibration method embodiments when executing the computer program 82, for example Figure 2 shown in steps S201 to S202. Alternatively, the processor 80 implements the functions of each module / unit in each of the above apparatus embodiments when executing the computer program 82, for example Figure 7 the functions of the modules 701 to 702 shown.

[0158] For example, the computer program 82 can be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 82 in the mechanical arm 8.

[0159] Those skilled in the art can understand that, Figure 8 The mechanical arm 8 is only an example and does not constitute a limitation on the mechanical arm 8, and can include more or fewer components than those shown, or combine certain components, or different components, for example, the mechanical arm 8 can also include an input / output device, a network access device, a bus, etc.

[0160] The processor 80 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0161] The memory 81 can be an internal storage unit of the mechanical arm 8, such as a hard disk or a memory of the mechanical arm 8. The memory 81 can also be an external storage device of the mechanical arm 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 81 can include both the internal storage unit and the external storage device of the mechanical arm 8. The memory 81 is used to store the computer program and other programs and data required by the mechanical arm 8. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0163] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0164] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0165] In the embodiments provided in the present application, it should be understood that the disclosed devices / robot arms and methods can be implemented in other ways. For example, the device / robot arm embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed each other can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0166] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0167] In addition, each of the function units in each of the embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0168] The integrated module / unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be implemented by a computer program instructing related hardware to complete, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer readable storage medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable storage medium does not include electric carrier signals and telecommunication signals.

[0169] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A hand-eye calibration method, characterized in that, The method comprises: obtaining a first set of measurement data of a first calibration object and a second calibration object at a first set of measurement positions; obtaining a second set of measurement data of the first calibration object and the second calibration object at a second set of measurement positions; wherein the first calibration object is a calibration object arranged on a visual recognition station, the second calibration object is a calibration object arranged on a mechanical arm movement station, the first calibration object and the second calibration object maintain a constant relative pose relationship; the second set of measurement positions are positions obtained by moving the first set of measurement positions by a preset distance in a preset direction and / or rotating by a preset angle; performing hand-eye calibration according to the first set of measurement data and the second set of measurement data.

2. The hand-eye calibration method of claim 1, wherein, The first set of measurement positions comprises a first camera measurement position and a first mechanical arm measurement position; obtaining a first set of measurement data of the first calibration object and the second calibration object at a first set of measurement positions comprises: measuring the pose of the first calibration object at the first camera measurement position; measuring the pose of the second calibration object at the first mechanical arm measurement position.

3. The hand-eye calibration method of claim 2, wherein, The second set of measurement positions comprises a second camera measurement position and a second mechanical arm measurement position; obtaining a second set of measurement data of the first calibration object and the second calibration object at a second set of measurement positions comprises: measuring the pose of the first calibration object at the second camera measurement position; measuring the pose of the second calibration object at the second mechanical arm measurement position.

4. The hand-eye calibration method of claim 3, wherein, The hand-eye calibration according to the first set of measurement data and the second set of measurement data comprises: calculating corresponding homogeneous transformation matrices according to the first set of measurement data and the second set of measurement data; establishing a calibration equation between the camera and the end effector according to the corresponding homogeneous transformation matrices; solving the calibration equation to obtain a hand-eye calibration result.

5. The hand-eye calibration method of claim 4, wherein, The calculation of the corresponding homogeneous transformation matrices according to the first set of measurement data and the second set of measurement data comprises: calculating a homogeneous transformation matrix from a first calibration object coordinate system at the first camera measurement position to a camera coordinate system according to the pose of the first calibration object at the first camera measurement position; calculating a homogeneous transformation matrix from a second calibration object coordinate system at the first mechanical arm measurement position to an end effector coordinate system according to the pose of the second calibration object at the first mechanical arm measurement position; calculating a homogeneous transformation matrix from a first calibration object coordinate system at the second camera measurement position to a camera coordinate system according to the pose of the first calibration object at the second camera measurement position; calculating a homogeneous transformation matrix from a second calibration object coordinate system at the second mechanical arm measurement position to an end effector coordinate system according to the pose of the second calibration object at the second mechanical arm measurement position.

6. The hand-eye calibration method of claim 5, wherein, The establishment of the calibration equation between the camera and the end effector according to the corresponding homogeneous transformation matrices comprises: According to a homogeneous transformation matrix of a first calibration object coordinate system to a camera coordinate system at the first camera measurement position, a homogeneous transformation matrix of a second calibration object coordinate system to an end effector coordinate system at the first robot arm measurement position, a homogeneous transformation matrix of the first calibration object coordinate system to the camera coordinate system at the second camera measurement position, and a homogeneous transformation matrix of the second calibration object coordinate system to the end effector coordinate system at the second robot arm measurement position, the calibration equation is established.

7. A hand-eye calibration device, characterized in that Comprise: a measurement data acquisition module configured to acquire a first set of measurement data of a first calibration object and a second calibration object at a first set of measurement positions; acquire a second set of measurement data of the first calibration object and the second calibration object at a second set of measurement positions; wherein the first calibration object is a calibration object arranged on a visual recognition station, the second calibration object is a calibration object arranged on a robot motion station, the first calibration object and the second calibration object maintain a constant relative pose relationship; the second set of measurement positions are positions obtained by moving the first set of measurement positions in a preset direction by a preset distance, and / or rotating by a preset angle; an eye-hand calibration module configured to perform eye-hand calibration according to the first set of measurement data and the second set of measurement data.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to realize the steps of the eye-hand calibration method according to any one of claims 1 to 6.

9. A robot arm comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the eye-hand calibration method according to any one of claims 1 to 6.

Images