Hand-eye calibration method, device, system, storage medium and product

Abstract

The present application discloses a hand-eye calibration method, device, system, storage medium and product, and relates to the field of automated production technology. The present application uses a motion detection camera for auxiliary calibration. On the basis of not incurring additional costs, by collecting first image data and second image data of the same calibration object, and based on a known first conversion relationship, coordinate conversion is performed between different coordinate systems, thereby calculating the conversion relationship between the actuator coordinate system and the hand-eye camera coordinate system, and then realizing automatic calibration of the hand-eye camera, actuator and motion detection camera, which not only saves labor costs, but also makes full use of existing equipment, does not introduce new equipment costs, and improves calibration efficiency and calibration accuracy.

Description

Technical Field

[0001] The present application relates to the field of automated production technology, and in particular to a hand-eye calibration method, device, system, storage medium and product. Background Art

[0002] In the related technologies, manipulators are widely used in various high-precision automation scenarios, especially in process flows involving precise positioning. In order to ensure the precise positioning of actuators (such as welding heads, grippers, nozzles and other tools) relative to visual systems (such as cameras), calibration has become an indispensable part of automated production. In the related technologies, when calibrating, the position of the camera and actuator is generally adjusted manually, and correction and calibration are performed through manual measurement and other methods. This is difficult to repeat in a production environment, and it relies on manual adjustments by operators, which is inefficient and difficult to guarantee accuracy.

[0003] Therefore, how to calibrate the robot's visual system more accurately and quickly is a problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of this application is to provide a hand-eye calibration method, device, equipment, storage medium and product, aiming to solve the technical problem of how to calibrate the visual system of a robot more accurately and quickly.

[0005] To achieve the above objectives, the present application proposes a hand-eye calibration method, which includes:

[0006] Acquire first image data and second image data, wherein the first image data is obtained by photographing a calibration object with a hand-eye camera, and the second image data is obtained by photographing the calibration object with a motion detection camera; the hand-eye camera is fixedly disposed on a manipulator, an actuator is fixedly disposed at an end of the manipulator, the hand-eye camera and the actuator are both located on the same side of the calibration object, and the motion detection camera is located on the other side of the calibration object;

[0007] Based on the second image data and the first conversion relationship, obtaining the target coordinate value of the calibration object in the actuator coordinate system, wherein the first conversion relationship is the coordinate conversion relationship between the motion inspection camera coordinate system and the actuator coordinate system;

[0008] Based on the first image data and the target coordinate value, a second transformation relationship is calculated, where the second transformation relationship is a coordinate transformation relationship between a hand-eye camera coordinate system and an actuator coordinate system.

[0009] In some embodiments, before acquiring the target coordinate value of the calibration object in the actuator coordinate system based on the second image data and the first conversion relationship, the method further includes:

[0010] Controlling the actuator to move to a first optimal focus position relative to the motion inspection camera;

[0011] Control the actuator to move to at least 9 designated sampling points at different positions in the plane where the first optimal focus position is located, and control the motion detection camera to respectively collect target images of the actuator at each of the designated sampling points; wherein the plane where the first optimal focus position is located is perpendicular to the optical axis of the motion detection camera;

[0012] The first conversion relationship is calculated based on all the target images using a nine-point calibration method.

[0013] In some embodiments, before controlling the actuator to move to at least 9 designated sampling points at different positions in the plane where the first optimal focus position is located, the method further includes:

[0014] The extension directions of the two horizontal plane moving axes of the manipulator are respectively used as the row direction and the column direction of the 9-point array in the nine-point calibration method; wherein the two horizontal plane moving axes are perpendicular to each other;

[0015] The first optimal focus position is used as the center point of the 9-point array in the nine-point calibration method, and the 9 designated sampling points are determined on the plane where the first optimal focus position is located based on a preset spacing.

[0016] In some embodiments, the calculating the second conversion relationship based on the first image data and the target coordinate value includes:

[0017] Based on the first image data and the target coordinate value, the second conversion relationship is calculated using a nine-point calibration method.

[0018] In some embodiments, acquiring the first image data includes:

[0019] Controlling the calibration object to move until the calibration object moves to a second optimal focusing position relative to the hand-eye camera;

[0020] Controlling the hand-eye camera to collect first image information of the calibration object;

[0021] Identify at least nine preset feature points in the first image information, calculate the coordinates of each of the feature points in the hand-eye camera coordinate system, and obtain the first image data;

[0022] The acquiring of the second image data comprises:

[0023] Controlling the calibration object to move until the calibration object moves to a third optimal focusing position relative to the motion inspection camera;

[0024] Controlling the motion inspection camera to collect second image information of the calibration object;

[0025] Identify at least nine preset feature points in the second image information, calculate the coordinates of each of the feature points in the motion detection camera coordinate system, and obtain the second image data.

[0026] In some embodiments, before acquiring the first image data and the second image data, the method further includes:

[0027] The calibration object is controlled to move to a calibration position so that the hand-eye camera and the actuator are both located on the same side of the calibration object, and the motion detection camera is located on the other side of the calibration object.

[0028] In addition, to achieve the above purpose, the present application also proposes a hand-eye calibration device, which includes:

[0029] An image acquisition module, used to acquire first image data and second image data, wherein the first image data is obtained by photographing a calibration object with a hand-eye camera, and the second image data is obtained by photographing the calibration object with a motion detection camera; the hand-eye camera is fixedly arranged on a manipulator, and an actuator is fixedly arranged on an end of the manipulator, the hand-eye camera and the actuator are both located on the same side of the calibration object, and the motion detection camera is located on the other side of the calibration object;

[0030] A coordinate conversion module, used for acquiring a target coordinate value of the calibration object in the actuator coordinate system based on the second image data and the first conversion relationship, wherein the first conversion relationship is a coordinate conversion relationship between the motion inspection camera coordinate system and the actuator coordinate system;

[0031] A transformation relationship calibration module is used to calculate a second transformation relationship based on the first image data and the target coordinate value, where the second transformation relationship is a coordinate transformation relationship between a hand-eye camera coordinate system and an actuator coordinate system.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a hand-eye calibration system, the hand-eye calibration system comprising: a manipulator, an actuator is fixedly provided at the end of the manipulator, and the actuator is suitable for performing a preset action on a workpiece on a workbench;

[0033] A hand-eye camera, wherein the hand-eye camera is fixedly arranged on the manipulator;

[0034] A motion inspection camera, which is arranged on the side of the workbench away from the manipulator to verify whether the preset work is performed; and a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the hand-eye calibration method as described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the hand-eye calibration method described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the hand-eye calibration method described above are implemented.

[0037] One or more technical solutions proposed in this application have at least the following technical effects:

[0038] Using motion detection cameras for auxiliary calibration, without incurring additional costs, by collecting the first image data and the second image data of the same calibration object, and based on the known first transformation relationship, performing coordinate transformation between different coordinate systems, thereby calculating the transformation relationship between the actuator coordinate system and the hand-eye camera coordinate system, thereby realizing automatic calibration of the hand-eye camera, actuator and motion detection camera, which not only saves labor costs, but also makes full use of existing equipment without introducing new equipment costs, thereby improving calibration efficiency and calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A schematic diagram of the structure of a hand-eye calibration system provided in an embodiment of the present application is shown;

[0042] Figure 2 A partial structural schematic diagram of a hand-eye calibration system provided by an exemplary embodiment of the present application is shown;

[0043] Figure 3A schematic diagram of a flow chart of a hand-eye calibration method provided in an embodiment of the present application is shown;

[0044] Figure 4 A schematic diagram of the structure of a bottom calibration module provided by an exemplary embodiment of the present application is shown;

[0045] Figure 5 A schematic diagram showing a flow chart of a hand-eye calibration method provided by another embodiment of the present application is shown;

[0046] Figure 6 A schematic diagram of the structure of a hand-eye calibration device provided in an embodiment of the present application is shown;

[0047] Figure 7 Shows Figure 1 Schematic diagram of the structure of the control module.

[0048] Reference numerals:

[0049] 110. Manipulator; 120. Hand-eye camera; 130. Motion detection camera; 131. Drive module; 132. Calibration object; 133. Shell; 134. Accommodating cavity; 135. Lens hole; 136. Lifting platform; 137. Matching hole; 138. Moving seat; 131a. Lifting motor; 131b. Horizontal motor.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: obtaining first image data and second image data, the first image data is obtained by taking a hand-eye camera at the calibration object, and the second image data is obtained by taking a motion detection camera at the calibration object; the hand-eye camera is fixedly arranged on the manipulator, and the actuator is fixedly arranged at the end of the manipulator, the hand-eye camera and the actuator are both located on the same side of the calibration object, and the motion detection camera is located on the other side of the calibration object; based on the second image data and the first transformation relationship, obtaining the target coordinate value of the calibration object in the actuator coordinate system, wherein the first transformation relationship is the coordinate transformation relationship between the motion detection camera coordinate system and the actuator coordinate system; based on the first image data and the target coordinate value, calculating the second transformation relationship, the second transformation relationship is the coordinate transformation relationship between the hand-eye camera coordinate system and the actuator coordinate system.

[0054] In the related art, manipulators are widely used in various high-precision automation scenarios, especially in process flows involving precise positioning. In order to ensure the precise positioning of actuators (such as welding heads, grippers, nozzles and other tools) relative to the visual system (such as hand-eye cameras), calibration has become an indispensable part of automated production. In the related art, when calibrating, the position of the hand-eye camera and the actuator is generally adjusted manually, and correction and calibration are performed through manual measurement and other methods. This is difficult to repeat in a production environment, and it relies on manual adjustments by the operator, which is inefficient and difficult to guarantee accuracy.

[0055] In summary, how to calibrate the robot's visual system more accurately and quickly is a problem that needs to be solved urgently.

[0056] Based on this, the present application provides a solution, which uses the image data collected by the hand-eye camera and the motion detection camera for the same calibration object respectively, obtains the coordinate value of the calibration object in the motion detection camera, and converts it into the actuator coordinate system, and then combines the image data collected by the hand-eye camera to further calculate the conversion relationship between the hand-eye camera and the actuator coordinate system. The calibration can be completed automatically without human intervention, which greatly improves work efficiency and ensures the operating accuracy of the manipulator in scenarios where the actuator is frequently disassembled. In addition, the present application uses a motion detection camera to assist in calibration, which saves a certain cost for the calibration of the manipulator without introducing new hardware equipment.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a hand-eye calibration system capable of realizing the above functions, etc. The hand-eye calibration system is taken as an example to illustrate this embodiment and the following embodiments.

[0058] For ease of understanding, the hand-eye calibration system provided by the present application is first described. Figure 1 The hand-eye calibration system may include a manipulator, a hand-eye camera, a motion detection camera, a control module, and a gantry module. As a preferred example, please refer to Figure 2 :

[0059] Robot 110: A robot is a mechanical module that can simulate the movements of a human arm and is usually used for automated operations in industrial production. The robot can be controlled to perform various intended tasks through pre-designed programming (in the control module).

[0060] The manipulator usually includes a gantry module for position adjustment. The gantry module usually has two or three main axes, which are at right angles to each other (for example, Figure 2In some embodiments, the gantry module may further include a displacement track on the Z-axis to control the movement of the manipulator in the Z-axis direction; in other embodiments, the gantry module may further include a lifting device (such as a lifting motor, a screw, etc.) additionally arranged inside the manipulator to control the movement of the hand-eye camera and the actuator on the manipulator in the Z-axis direction.

[0061] In some embodiments, an actuator may be fixedly provided at the end of the manipulator. The actuator refers to a device on the manipulator that is used to directly contact the workpiece, and by cooperating with the manipulator, it can perform a preset action (i.e., a task) on the workpiece on the workbench. For example, in a chip manufacturing scenario, the actuator may be a suction nozzle, which can achieve the capture and placement of the chip by adjusting the air pressure; the actuator may also be a welding head to achieve the welding of the chip.

[0062] Hand-eye camera 120: The hand-eye camera is fixed on the manipulator, and can generally be set at the end of the manipulator. The hand-eye camera can be roughly understood as the visual system corresponding to the robot arm. When performing precision work tasks, it is difficult to achieve accurate positioning only by programming; in order to ensure the accuracy of the robot operation, the actuator on the robot is usually equipped with a hand-eye camera.

[0063] The basic working principle of the hand-eye camera after calibration is to convert the coordinates of the image captured by itself in the hand-eye camera coordinate system into the coordinates in the actuator coordinate system through a pre-calibrated conversion relationship, so that the following control module drives the actuator to move with reference to the converted coordinate values ​​to achieve fine operation.

[0064] Action inspection camera 130: The action inspection camera is generally installed on the side of the workbench away from the manipulator (generally below the workbench) and shoots toward the workbench; by shooting the work execution process on the workbench, the work status can be monitored to verify whether the preset action is executed in place. Unlike the hand-eye camera that plays the role of a visual system, the action inspection camera is used in related technologies to monitor the entire work process.

[0065] Control module 140: The control module generally refers to a control device that is connected to the aforementioned system modules such as the manipulator 110, the hand-eye camera 120, and the motion inspection camera 130, for example, a mobile terminal (computer, mobile phone, tablet computer, etc.); in some embodiments, taking the control module as a computer as an example, the staff can input a pre-set program code (including a specified working position) through the computer, and the computer compiles it, and then the computer sends a corresponding control signal based on the compilation result to control the motor in the gantry module to work, thereby controlling the movement of the manipulator with the assistance of the hand-eye camera until the actuator is in the specified position. The actuator can perform corresponding operations based on the compilation result. For example, in the case where the actuator is a nozzle, when it contacts the workpiece, the air pressure in the nozzle is reduced to pick up the workpiece, and then the gantry module controls the manipulator to move to another specified position, releases the air pressure in the nozzle, and puts down the workpiece, thereby achieving the task of transferring the workpiece.

[0066] Reference Figure 3 , Figure 3 The hand-eye calibration method provided by an embodiment of the present application is shown in a flowchart. The hand-eye calibration method can be applied to a hand-eye calibration device, and includes the following steps S110 to S130:

[0067] Step S110, acquiring first image data and second image data.

[0068] Among them, Figure 2 As shown, the hand-eye camera is fixedly arranged on the manipulator, the actuator is fixedly arranged on the end of the manipulator, the hand-eye camera and the actuator are both located on the same side of the calibration object, and the motion inspection camera is located on the other side of the calibration object.

[0069] like Figure 4 As shown, in this embodiment, in addition to the motion detection camera 130, a driving module 131 and a calibration object 132 are also provided to implement the hand-eye calibration method provided in this application.

[0070] Specifically, in this embodiment, the motion detection camera 130, the driving module 131 and the calibration object 132 can be packaged in the same module below the workbench (hereinafter referred to as the bottom calibration module). The housing 133 of the bottom calibration module remains fixed relative to the workbench.

[0071] The housing 133 of the bottom calibration module defines a housing cavity 134, and the motion detection camera 130 is fixedly disposed in the housing cavity 134, and the lens of the motion detection camera 130 faces upward. The side wall (i.e., the upper surface) of the housing 133 facing the workbench has a lens hole 135 facing the workbench, and the motion detection camera 130 faces the lens hole 135 so that the motion detection camera 130 can monitor the workbench. In some embodiments, the lens hole 135 is provided with a protective lens to protect the lens of the motion detection camera 130.

[0072] The driving module 131 includes a lifting motor 131a and a horizontal motor 131b. Among them, the fixed end of the lifting motor 131a is fixedly arranged in the accommodating cavity 134, and the lifting motor 131a and the motion detection camera 130 are arranged side by side. The output shaft of the lifting motor 131a is retractable along the Z-axis direction. The output shaft is fixedly connected to a lifting platform 136. The horizontal motor 131b is arranged on the lifting platform 136, and the lifting platform 136 is also located on the side of the motion detection camera 130. And the side wall of the shell 133 facing the workbench also has a matching hole 137 connected to the accommodating cavity 134, and the lifting platform 136 can pass through the matching hole 137 and extend out of the shell 133 during the lifting process. The lifting motor 131a is used to drive the calibration object 132 to move in the Z-axis direction, so as to achieve the focusing of the calibration object 132 by the motion detection camera 130 during the calibration operation.

[0073] The calibration object 132 is disposed on a side wall of the lifting platform 136 facing the workbench, and is movable between a calibration position and an avoidance position. When the calibration object is at the calibration position, at least a portion of the calibration object is located directly above the lens hole 135. When the calibration object 132 is at the avoidance position, the calibration object 132 is located to the side of the lens hole 135. Thus, during the calibration operation, the calibration object 132 moves to the calibration position and enters the field of view of the motion detection camera 130, or when the system is working normally, the calibration object 132 moves to the avoidance position and leaves the field of view of the motion detection camera 130, so as to avoid affecting the motion detection camera 130 in performing its normal monitoring function.

[0074] In some embodiments, the hand-eye calibration system further includes a moving seat 138, which is slidably disposed on a side wall of the lifting platform 136 facing the workbench, and the calibration object 132 is fixedly disposed on the moving seat 138 so that the calibration object 132 can move between the calibration position and the avoidance position. The moving seat 138 is coupled with the output shaft of the horizontal motor 131b so that the horizontal motor 131b drives the moving seat 138 to slide back and forth. It can be understood that the transmission connection between the output shaft of the horizontal motor 131b and the moving seat 138 can be a conveyor belt connection, etc., and this embodiment does not limit this.

[0075] Thus, the driving module controls the calibration object to move in three-dimensional space.

[0076] In this embodiment, the first image data refers to the image data captured by the hand-eye camera with respect to the calibration object; the second image data refers to the image data captured by the motion detection camera with respect to the calibration object.

[0077] There are usually many methods for calibrating the hand-eye camera and the actuator, such as the nine-point calibration method, the checkerboard method, the Zhang Zhengyou calibration method, etc. Most of them use image data for coordinate transformation to achieve calibration. As a preferred implementation, this embodiment uses the nine-point calibration method for calibration.

[0078] Based on the nine-point calibration method, in some embodiments, at least nine preset feature points can be pre-selected for the calibration object. The preset feature points can be fixed points on the contour of the calibration object, or points pre-etched on the surface of the calibration object, which is not limited in this embodiment.

[0079] In this embodiment, the movement of the calibration object can be controlled by the lifting motor and the horizontal motor until the calibration object moves to the second optimal focusing position relative to the hand-eye camera; and the hand-eye camera is controlled to collect the first image information of the calibration object.

[0080] It is understandable that in order for the hand-eye camera to clearly capture the image of the calibration object, the hand-eye camera needs to be focused before the image is captured. If the hand-eye camera is not focused, the captured image may be blurred, which will affect the subsequent extraction and recognition of the calibration object feature points.

[0081] Therefore, in this embodiment, the calibration object can be controlled to move up and down by the lifting motor, and during this process, the hand-eye camera can continuously collect image information. By executing a contrast detection algorithm on the collected image information, the height corresponding to the image information with the highest contrast can be determined as the second optimal focus position of the calibration object.

[0082] After determining the second optimal focus position, the calibration object may be fixed at the position and the hand-eye camera may be controlled to take a picture of the calibration object, that is, to collect the first image information.

[0083] After obtaining the first image information, at least nine preset feature points on the calibration object in the first image information may be identified, and the coordinates of each of the feature points in the hand-eye camera coordinate system may be calculated to obtain the first image data.

[0084] Among them, the hand-eye camera can have a pre-set hand-eye camera coordinate system. For example, the optical center of the hand-eye camera (i.e., the center of the camera lens) can be determined as the origin of the hand-eye camera coordinate system. Based on the first image data and the pre-calibrated internal parameters and external parameters of the hand-eye camera, the coordinate values ​​of the aforementioned preset feature points in the hand-eye camera coordinate system can be calculated as the first image data of this embodiment.

[0085] Similar to the acquisition of the first image data, in this embodiment, the calibration object can be controlled to move until it moves to the third optimal focus position relative to the motion detection camera; the motion detection camera is controlled to acquire the second image information for the calibration object; at least nine preset feature points in the second image information are identified, and the coordinates of each feature point in the motion detection camera coordinate system are calculated to obtain the second image data. Since the specific operations involved are basically the same as those of the hand-eye camera, this embodiment will not be repeated here.

[0086] Step S120: acquiring the target coordinate value of the calibration object in the actuator coordinate system based on the second image data and the first conversion relationship.

[0087] The first conversion relationship is the coordinate conversion relationship between the motion inspection camera coordinate system and the actuator coordinate system. In some embodiments, before step S110, refer to Figure 5 The first conversion relationship can be obtained in advance through the following steps S121 to S123:

[0088] Step S121, controlling the actuator to move to a first optimal focusing position relative to the motion inspection camera.

[0089] Among them, similar to the aforementioned step S110, in order to ensure the clarity of the target image subsequently collected, it is necessary to focus the motion detection camera and the actuator first to ensure that the actuator is in the optimal focus position of the motion detection camera.

[0090] Since the motion detection camera is fixed, in this embodiment, the actuator can be controlled to move by the above-mentioned gantry module. Specifically, the actuator can be moved to a specified reference position in the XY plane (which can be a position pre-specified by the staff and within the field of view of the motion detection camera) by the gantry module, and then the actuator can be controlled to move up and down. During the process of the actuator moving up and down, the image is collected in real time and the contrast detection algorithm as described above is executed to calculate the first optimal focus position.

[0091] Step S122, controlling the actuator to move to at least 9 designated sampling points at different positions in the plane where the first optimal focus position is located, and controlling the motion inspection camera to respectively collect target images of the actuator at each designated sampling point.

[0092] The plane where the first optimal focus position is located refers to a plane parallel to the workbench and perpendicular to the optical axis of the motion inspection camera. Assuming that the workbench is horizontal, the plane where the first optimal focus position is located refers to a horizontal plane at a height corresponding to the first optimal focus position.

[0093] In this embodiment, the distance of each movement can be set by the staff. In some implementations, the distance can be set to a fixed value (for example, 3 mm) or not. This embodiment does not limit this. It only needs to ensure that it is still within the field of view of the motion detection camera after movement.

[0094] In addition, the movement direction of each time can also be set by the staff. Specifically, the extension directions of the two horizontal plane moving axes of the manipulator are used as the row direction and column direction of the 9-point array respectively; the two horizontal plane moving axes are perpendicular to each other; the first optimal focus position is used as the center point of the 9-point array in the nine-point calibration method, and 9 designated sampling points are determined on the plane where the first optimal focus position is located based on the preset spacing.

[0095] As an implementation method, the overall movement strategy may be set to "move at intervals of 3 mm in the X and Y directions, respectively, and cover a total of 9 sampling points including the origin."

[0096] In this embodiment, every time the motion detection camera moves to a designated sampling point, the motion detection camera may be controlled to collect image data, ie, a target image.

[0097] Step S123, using a nine-point calibration method, based on all target images, calculate a first conversion relationship.

[0098] It can be understood that in the actuator coordinate system, the movement strategy is known, that is, the coordinates of the specified sampling point in the actuator coordinate system are known. Based on the pre-configured internal parameters and external parameters of the motion detection camera, the coordinates of the specified sampling point in the motion detection camera coordinate system can be calculated. Therefore, the conversion relationship between the motion detection camera coordinate system and the actuator coordinate system can be determined by the nine-point calibration method, that is, the first conversion relationship is obtained.

[0099] After the first conversion relationship is known in advance, when executing step S120, the target coordinate value of the calibration object in the actuator coordinate system can be directly obtained based on the second image data and the first conversion relationship.

[0100] As mentioned above, the second image data is the coordinates of at least nine preset feature points in the motion detection camera coordinate system; based on the first conversion relationship, it can be directly converted into the target coordinate value in the actuator coordinate system.

[0101] Step S130, calculating a second transformation relationship based on the first image data and the target coordinate value, where the second transformation relationship is a coordinate transformation relationship between the hand-eye camera coordinate system and the actuator coordinate system.

[0102] When the first image data is the coordinate value of the preset feature point in the hand-eye camera coordinate system, and the target coordinate value of the preset feature point in the actuator coordinate system is obtained through step S120, the transformation relationship between the hand-eye camera coordinate system and the actuator coordinate system, that is, the second transformation relationship, can be directly calculated through the nine-point calibration algorithm.

[0103] Based on this, the calibration between the hand-eye camera and the actuator motion detection camera was realized.

[0104] The present embodiment provides a hand-eye calibration method, which uses a motion detection camera for auxiliary calibration. Without incurring additional costs, the first image data and the second image data are collected for the same calibration object, and based on a known first conversion relationship, coordinate conversion is performed between different coordinate systems, thereby calculating the conversion relationship between the actuator coordinate system and the hand-eye camera coordinate system, thereby realizing automatic calibration of the hand-eye camera, the actuator, and the motion detection camera. This not only saves labor costs, but also makes full use of existing equipment without introducing new equipment costs, thereby improving calibration efficiency and calibration accuracy.

[0105] This application also provides a hand-eye calibration device, please refer to Figure 6 , the hand-eye calibration device 200 comprises:

[0106] The image acquisition module 210 is used to acquire first image data and second image data, wherein the first image data is obtained by photographing the calibration object with the hand-eye camera, and the second image data is obtained by photographing the calibration object with the motion detection camera; the hand-eye camera is fixedly arranged on the manipulator, and the actuator is fixedly arranged on the end of the manipulator, the hand-eye camera and the actuator are both located on the same side of the calibration object, and the motion detection camera is located on the other side of the calibration object;

[0107] A coordinate conversion module 220, configured to obtain a target coordinate value of the calibration object in the actuator coordinate system based on the second image data and the first conversion relationship, wherein the first conversion relationship is a coordinate conversion relationship between the motion inspection camera coordinate system and the actuator coordinate system;

[0108] The transformation relationship calibration module 230 is used to calculate a second transformation relationship based on the first image data and the target coordinate value, where the second transformation relationship is a coordinate transformation relationship between the hand-eye camera coordinate system and the actuator coordinate system.

[0109] The hand-eye calibration device 200 provided in the present application adopts the hand-eye calibration method in the above embodiment, which can solve the technical problem of how to calibrate the visual system of the manipulator more accurately and quickly. Compared with the prior art, the beneficial effects of the hand-eye calibration device 200 provided in the present application are the same as the beneficial effects of the hand-eye calibration method provided in the above embodiment, and the other technical features in the hand-eye calibration device 200 are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0110] In some embodiments, the control module 140 in the hand-eye calibration system 100 includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the hand-eye calibration method in the above-mentioned embodiment one.

[0111] Reference below Figure 7 The control module in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The control module shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0112] like Figure 7As shown, the control module 140 may include a processing device 310 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 320 or a program loaded from a storage device 330 to a random access memory (RAM: Random Access Memory) 340. In RAM340, various programs and data required for the operation of the control module are also stored. The processing device 310, ROM320, and RAM340 are connected to each other through a bus 350. An input / output (I / O) interface 360 ​​is also connected to the bus. Generally, the following systems can be connected to the I / O interface 360: an input device 370 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 380 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 330 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 390. The communication device 390 can allow the control module to communicate with other devices wirelessly or by wire to exchange data. Although the control module with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0113] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device, or installed from a storage device 330, or installed from a ROM 320. When the computer program is executed by the processing device 310, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0114] The control module provided by the present application adopts the hand-eye calibration method in the above embodiment, which can solve the technical problem of how to calibrate the visual system of the manipulator more accurately and quickly. Compared with the prior art, the beneficial effects of the control module provided by the present application are the same as the beneficial effects of the hand-eye calibration method provided by the above embodiment, and the other technical features in the control module are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0115] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0116] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0117] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, wherein the computer-readable program instructions are used to execute the hand-eye calibration method in the above-mentioned embodiment.

[0118] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0119] The computer-readable storage medium may be included in the hand-eye calibration system; or may exist independently without being assembled into the hand-eye calibration system.

[0120] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the hand-eye calibration system, the hand-eye calibration system can write computer program codes for performing the operations of the present application in one or more programming languages ​​or a combination thereof. The programming languages ​​include object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through the Internet using an Internet service provider).

[0121] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0123] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned hand-eye calibration method, and can solve the technical problem of how to calibrate the visual system of the manipulator more accurately and quickly. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the hand-eye calibration method provided in the above-mentioned embodiment, and will not be repeated here.

[0124] The present application also provides a computer program product, including a computer program, which implements the steps of the hand-eye calibration method as described above when executed by a processor.

[0125] The computer program product provided by the present application can solve the technical problem of how to calibrate the visual system of the manipulator more accurately and quickly. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the hand-eye calibration method provided by the above embodiment, which will not be repeated here.

[0126] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A hand-eye calibration method, characterized in that: The calibration method comprises: Controlling the calibration object to move until the calibration object moves to a second optimal focusing position relative to the hand-eye camera; Controlling the hand-eye camera to collect first image information of the calibration object; Identify at least nine preset feature points in the first image information, calculate the coordinates of each of the feature points in the hand-eye camera coordinate system, and obtain first image data; Controlling the calibration object to move until the calibration object moves to a third optimal focusing position relative to the motion detection camera; Controlling the motion inspection camera to collect second image information of the calibration object; Identify at least nine preset feature points in the second image information, calculate the coordinates of each of the feature points in the motion detection camera coordinate system, and obtain second image data; wherein the hand-eye camera is fixedly set on the manipulator, the actuator is fixedly set at the end of the manipulator, the hand-eye camera and the actuator are both located on the same side of the calibration object, and the motion detection camera is located on the other side of the calibration object; Controlling the actuator to move to a first optimal focus position relative to the motion inspection camera; Control the actuator to move to at least 9 designated sampling points at different positions in the plane where the first optimal focus position is located, and control the motion detection camera to respectively collect target images of the actuator at each of the designated sampling points; wherein the plane where the first optimal focus position is located is perpendicular to the optical axis of the motion detection camera; Using a nine-point calibration method, based on all the target images, a first conversion relationship is calculated; Based on the second image data and the first conversion relationship, obtaining the target coordinate value of the calibration object in the actuator coordinate system, wherein the first conversion relationship is the coordinate conversion relationship between the motion inspection camera coordinate system and the actuator coordinate system; Based on the first image data and the target coordinate value, a second transformation relationship is calculated, where the second transformation relationship is a coordinate transformation relationship between a hand-eye camera coordinate system and an actuator coordinate system.

2. The hand-eye calibration method according to claim 1, characterized in that: Before controlling the actuator to move to at least 9 designated sampling points at different positions in the plane where the first optimal focus position is located, the method further includes: The extension directions of the two horizontal plane moving axes of the manipulator are respectively used as the row direction and the column direction of the 9-point array in the nine-point calibration method; wherein the two horizontal plane moving axes are perpendicular to each other; The first optimal focus position is used as the center point of the 9-point array in the nine-point calibration method, and the 9 designated sampling points are determined on the plane where the first optimal focus position is located based on a preset spacing.

3. The hand-eye calibration method according to claim 1, characterized in that: The calculating a second conversion relationship based on the first image data and the target coordinate value includes: Based on the first image data and the target coordinate value, the second conversion relationship is calculated using a nine-point calibration method.

4. The hand-eye calibration method according to claim 1, characterized in that: The control of the calibration object to move until the calibration object moves to a second optimal focus position relative to the hand-eye camera, the method further comprising: The calibration object is controlled to move to a calibration position so that the hand-eye camera and the actuator are both located on the same side of the calibration object, and the motion detection camera is located on the other side of the calibration object.

5. A hand-eye calibration device, characterized in that: The hand-eye calibration device comprises: An image acquisition module, used to control the movement of the calibration object until the calibration object moves to a second optimal focus position relative to the hand-eye camera; control the hand-eye camera to collect first image information for the calibration object; identify at least nine preset feature points in the first image information, calculate the coordinates of each feature point in the hand-eye camera coordinate system, and obtain first image data; control the movement of the calibration object until the calibration object moves to a third optimal focus position relative to the motion detection camera; control the motion detection camera to collect second image information for the calibration object; identify at least nine preset feature points in the second image information, calculate the coordinates of each feature point in the motion detection camera coordinate system, and obtain second image data; wherein the hand-eye camera is fixedly arranged on a manipulator, an actuator is fixedly arranged at the end of the manipulator, the hand-eye camera and the actuator are both located on the same side of the calibration object, and the motion detection camera is located on the other side of the calibration object; A coordinate conversion module, used to control the actuator to move to a first optimal focus position relative to the motion detection camera; control the actuator to move to at least 9 designated sampling points at different positions in the plane where the first optimal focus position is located, and control the motion detection camera to respectively collect target images of the actuator at each of the designated sampling points; wherein the plane where the first optimal focus position is located is perpendicular to the optical axis of the motion detection camera; a first conversion relationship is calculated based on all the target images using a nine-point calibration method; and further used to obtain the target coordinate value of the calibration object in the actuator coordinate system based on the second image data and the first conversion relationship, wherein the first conversion relationship is the coordinate conversion relationship between the motion detection camera coordinate system and the actuator coordinate system; A transformation relationship calibration module is used to calculate a second transformation relationship based on the first image data and the target coordinate value, where the second transformation relationship is a coordinate transformation relationship between a hand-eye camera coordinate system and an actuator coordinate system.

6. A hand-eye calibration system, characterized in that: The hand-eye calibration system comprises: A manipulator, wherein an actuator is fixedly disposed at the end of the manipulator, and the actuator is suitable for performing a preset action on a workpiece on the workbench; A hand-eye camera, wherein the hand-eye camera is fixedly arranged on the manipulator; An action inspection camera, which is disposed on a side of the workbench away from the manipulator to verify whether the preset action is performed in place; and A control module, the control module comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the hand-eye calibration method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the hand-eye calibration method according to any one of claims 1 to 4 are implemented.

8. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the hand-eye calibration method according to any one of claims 1 to 4 are implemented.

Images