A configurable hand-eye calibration method and device

Abstract

The present invention discloses a configurable hand-eye calibration method and device, including an offline configuration stage and an online automatic calibration stage; the offline configuration includes the following steps: offline teaching can collect multiple robot postures with clear images and record them in the posture queue to be collected; generate no less than two calibration plate images with different postures as needed; online automatic calibration includes rough calibration and fine calibration. The present invention can reduce manual intervention, avoid errors caused by manual data recording, save time in the online hand-eye calibration process, avoid collisions and path inaccessibility caused by fully automatic path planning, and solve the problem of poor universal applicability in different application scenarios.

Description

Technical Field

[0001] The present invention relates to the field of hand-eye calibration, and in particular to a configurable hand-eye calibration method and device. Background Art

[0002] With the rapid development of science and technology, the application of industrial robots has become more and more popular. In the research of industrial robots, hand-eye calibration technology is an important part of the visual guidance system. Its purpose is to measure the attitude transformation matrix from the camera coordinate system mounted on the end effector of the robot arm to the end effector coordinate system, which is an important prerequisite for ensuring the accurate positioning and grasping of the visual servo robot arm.

[0003] In modern smart factories, in order to make efficient use of production equipment, we hope that robots can participate in the work of multiple stations, or need to replace actuators regularly. This requires frequent calibration of the hand-eye system of industrial robots. In such application scenarios, hand-eye calibration should be fast and can be performed automatically.

[0004] The hand-eye calibration problem can be mathematically expressed as solving the homogeneous equation AX=XB, where A, B and X all represent rotation and translation matrices in homogeneous form, A and B are known observations, and X is the hand-eye transformation matrix to be solved. There are many classic algorithms that can solve this equation, which can be summarized into two stages: in the first stage, a linear method is used to solve the rotation axis and angle of the X matrix according to the rotation axis and angle of the A and B matrices; in the second stage, the translation part of the X matrix is ​​linearly solved using the result of the rotation part of the X matrix obtained in the first stage. This type of method can transform complex nonlinear problems into linear equations with simple forms and faster calculation speeds. However, the error generated by the calculation of the rotation part in the first stage will be diffused in the solution of the translation part in the second stage, so the calibration result is easily affected by the noise of the observation data. Of course, if enough observation data can be collected, the influence of observation errors can be eliminated to a certain extent.

[0005] To conveniently obtain the relative posture relationship between the robot and the visual system, not only a feasible hand-eye transformation matrix solution algorithm is required, but also a system that can be automatically executed in order to obtain enough observation data. During the automated execution process, a series of robot postures and corresponding calibration plate images need to be collected. An important part is the motion planning of the image posture collection. In order to improve the accuracy of the hand-eye calibration results, four important principles must be followed: first, increase the angle between the relative motion rotation axes as much as possible; second, increase the relative motion rotation angle as much as possible; third, reduce the distance from the calibration plate to the center of the camera optical axis as much as possible; fourth, reduce the distance between the end center in different postures as much as possible.

[0006] In the past, there were two types of hand-eye calibration solutions: manual data collection, which required high professional knowledge of operators, was prone to errors in data recording, and took a long time to calibrate. The other type was automatic data collection. In most of these solutions, the method of directly planning the robot's motion path online was adopted, which was prone to two problems:

[0007] (1) Collision or path inaccessibility problems, interference with other devices at the workstation;

[0008] (2) Since the calibration results are limited by the data collected during calibration, the actual accuracy is not as ideal as calculated in different application scenarios. Summary of the invention

[0009] In order to solve the above problems, the present invention proposes a new hand-eye calibration method and device to avoid collision and path inaccessibility problems and improve the accuracy of actual positioning and grasping.

[0010] To achieve the above object, the technical solution of the present invention is:

[0011] A configurable hand-eye calibration method includes an offline configuration phase and an online automatic calibration phase;

[0012] Offline configuration includes the following steps:

[0013] Offline teaching can collect multiple robot poses with clear images and record them in the pose queue to be collected;

[0014] Generate no less than two calibration plate images with different poses as needed;

[0015] Online automatic calibration includes rough calibration and fine calibration;

[0016] The rough calibration includes the following steps:

[0017] Step a: The robot moves to the first position in the queue to be collected when the offline configuration result is initialized;

[0018] Step b, collect the calibration plate image and remove the pose from the queue;

[0019] Step c: determine the quality of the calibration plate image. If it meets the requirements, save the current actual posture and calibration plate image. Otherwise, execute step a.

[0020] Step d, determine whether the queue to be collected is empty, if so, the robot movement ends and executes step e, otherwise executes step a;

[0021] Step e: determine whether there are at least N valid images. If so, calculate the hand-eye relationship matrix and its reprojection error based on the collected robot posture and calibration plate image data, calibrate the camera internal parameters, and calculate the posture of the calibration plate in the robot coordinate system. Otherwise, the calibration fails and re-perform offline configuration.

[0022] Fine calibration includes the following steps:

[0023] Step f, according to the camera intrinsic parameters of the rough calibration, the hand-eye calibration results, and the position and posture of the calibration plate in the robot coordinate system, the corresponding robot arm end position and posture are calculated and added to the robot position and posture queue in sequence;

[0024] Step g: the robot moves to the position and collects the image of the calibration plate;

[0025] Step h: determine whether the calibration plate image can be correctly identified. If so, save the current actual posture and the actual calibration plate image. Otherwise, execute step g.

[0026] Step i: determine whether the calibration plate image queue is empty. If so, execute step j; otherwise, execute step g;

[0027] Step j: determine whether the number of valid images is greater than or equal to N. If so, calculate the camera intrinsic parameters, hand-eye transformation matrix and its reprojection error. Otherwise, terminate the fine calibration directly.

[0028] Preferably, offline configuration teaching robot posture includes the following steps:

[0029] S1, initialize the pose queue and the minimum length of the pose queue;

[0030] S2, the robot moves to a position and collects the image of the calibration plate;

[0031] S3, determine whether the image is clear, if so, add the current posture to the posture queue, otherwise execute step S2;

[0032] S4. Determine whether the length of the posture queue is equal to the minimum length. If so, the offline configuration teaching robot ends, otherwise execute step S2.

[0033] Preferably, generating calibration plate images with different poses comprises the following steps:

[0034] Step 1: Initialize the camera internal parameters, nine-square grid area and logo;

[0035] Step 2: Randomly generate a camera pose;

[0036] Step 3: Generate a calibration plate image;

[0037] Step 4: Determine whether the center of the calibration plate falls within the center area of ​​the nine-square grid; if so, save the calibration plate image, otherwise execute step 2;

[0038] Step 5: Determine whether each area is covered by at least two images with different poses. If so, end; otherwise, proceed to step 2.

[0039] Preferably, in the fine calibration stage, the formula for calculating the end position of the robotic arm in step f is as follows:

[0040]

[0041] In the formula, is the pose of the tool end in the robot base coordinate system; is the position and posture of the calibration plate in the robot base coordinate system; It is the position and posture of the calibration plate in the camera coordinate system, which is calculated based on the camera intrinsic parameters and the physical model of the calibration plate; is the pose of the tool end in the camera coordinate system.

[0042] Preferably, the value of N is greater than or equal to 9.

[0043] In order to implement the above-mentioned hand-eye calibration method, the present invention also proposes a configurable hand-eye calibration device, including an image acquisition module, a display screen 1, a display screen 2, a calibration board with a backlight source, a light source controller, a robotic arm, an embedded control system, and a PC-based control system;

[0044] The image acquisition module, the embedded control system and the PC-based control system are connected via Ethernet; the calibration board with a backlight source is connected to the light source controller; the robotic arm is connected to the embedded control system; display screen one is connected to the embedded control system, and parameters are set through display screen one; display screen two is connected to the PC-based control system, and parameters are set through display screen two.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] The present invention can reduce manual intervention, avoid errors caused by manual data recording, and save time for the online hand-eye calibration process. After offline configuration, the time for completing an online hand-eye calibration can be controlled within three minutes. Secondly, it can avoid collisions and path inaccessibility problems caused by fully automatic path planning, and solve the problem of poor universal applicability in different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 Offline configuration teaching robot posture flow chart;

[0049] Figure 2 Randomly generate calibration plate image flow chart;

[0050] Figure 3 This is the rough calibration flow chart;

[0051] Figure 4 This is the flow chart for precise calibration;

[0052] Figure 5 Schematic diagram of the structure of a configurable automatic hand-eye calibration device.

[0053] In the figure: 100, image acquisition module; 110, display screen 1; 120, display screen 2; 130, calibration board with backlight source; 140, source controller; 150, robotic arm; 160, embedded control system; 170, PC-based control system. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0055] like Figure 1-Figure 4 As shown, this embodiment discloses a configurable hand-eye calibration method, including an offline configuration stage and an online automatic calibration stage;

[0056] 1. Offline configuration includes the following steps:

[0057] (1) According to the attached Figure 1 The process of teaching a set of robot poses that can capture clear images is as follows. 18 to 24 poses are selected, and the poses are different from each other. The corresponding images should cover the entire nine-square grid partition of the field of view, and each partition has a different rotation angle. After teaching, the poses are recorded in the queue of poses to be collected offline. Specifically, the offline configuration of teaching robot poses includes the following steps:

[0058] S1. Initialize the pose queue and the minimum length of the pose queue.

[0059] S2: The robot moves to a certain position and collects the image of the calibration plate.

[0060] S3. Determine whether the image is clear. If so, add the current posture to the posture queue, otherwise execute step S2.

[0061] S4. Determine whether the length of the posture queue is equal to the minimum length. If so, the offline configuration teaching robot ends, otherwise execute step S2.

[0062] (2) According to the actual application scenario, Figure 2 The process is to generate a set of calibration plate images with large posture differences when the physical size of the calibration plate is known, and the calibration plate images should be no less than two. Specifically, generating calibration plate images with different postures includes the following steps:

[0063] Step 1: Initialize the camera internal parameters, nine-square grid area and logo.

[0064] Step 2: Randomly generate a camera pose.

[0065] Step 3: Generate the calibration plate image.

[0066] Step 4: Determine whether the center of the calibration plate falls in the central area of ​​the nine-square grid; if so, save the calibration plate image, otherwise execute step 2.

[0067] Step 5: Determine whether each area is covered by at least two images with different poses. If so, end; otherwise, proceed to step 2.

[0068] In the above step 1, the initialized intrinsic parameters do not need to be the same as the actual ones. A set of extrinsic parameters is set according to the intrinsic parameters. The coordinates of the calibration plate in the camera coordinate system are first calculated by projection transformation, and then the pixel coordinates are calculated according to the intrinsic parameters. The calculation formula is as follows:

[0069]

[0070] in, is the camera extrinsic parameter, is the world homogeneous coordinate of a point on the calibration plate, is the camera homogeneous coordinate of a point on the calibration plate.

[0071]

[0072] Among them, dx and dy represent the pixel sizes in the x and y directions respectively, u0 and v0 are the pixel coordinates of the center point of the image, f is the focal length, and γ is the distortion factor, which is set to 0 here according to the ideal model.

[0073] The above process is completed automatically by the program. As an extension, you can also configure a set of manually posed calibration plate images that meet the requirements.

[0074] 2. Online automatic calibration includes rough calibration and fine calibration;

[0075] (1) Rough calibration includes the following steps:

[0076] Step a: The robot moves to the first position in the queue of poses to be collected after the offline configuration result is initialized.

[0077] Step b: collect the calibration plate image and remove the pose from the queue.

[0078] Step c: Determine the quality of the calibration plate image. If it meets the requirements, save the current actual pose and calibration plate image. Otherwise, execute step a.

[0079] Step d: determine whether the queue to be collected is empty. If so, the robot movement ends and step e is executed; otherwise, step a is executed.

[0080] Step e: determine whether there are at least N valid images. If so, calculate the hand-eye relationship matrix and its reprojection error based on the collected robot posture and calibration plate image data, calibrate the camera internal parameters, and calculate the posture of the calibration plate in the robot coordinate system. Otherwise, the calibration fails and reconfigure offline. The calculation process of the hand-eye relationship matrix is ​​the process of solving AX=XB, which will not be described in detail here.

[0081] (2) Precision calibration includes the following steps:

[0082] Step f: Calculate the corresponding robot end pose according to the camera intrinsic parameters of the rough calibration, the hand-eye calibration results, and the pose of the calibration plate in the robot coordinate system, and add them to the robot pose queue in sequence. The calculation formula is as follows:

[0083]

[0084] In the above formula, is the pose of the tool end in the robot base coordinate system; is the position and posture of the calibration plate in the robot base coordinate system; It is the position and posture of the calibration plate in the camera coordinate system, which is calculated based on the camera intrinsic parameters and the physical model of the calibration plate; It is the pose of the tool end in the camera coordinate system, that is, the hand-eye transformation relationship.

[0085] Step g: The robot moves to the position and collects the calibration plate image.

[0086] Step h: determine whether the calibration plate image can be correctly identified. If so, save the current actual posture and the actual calibration plate image. Otherwise, execute step g.

[0087] Step i: determine whether the calibration plate image queue is empty. If so, execute step j; otherwise, execute step g.

[0088] Step j: determine whether the number of valid images is greater than or equal to N. If so, calculate the camera intrinsic parameters, hand-eye transformation matrix and its reprojection error. Otherwise, terminate the fine calibration directly.

[0089] In order to implement the above-mentioned hand-eye calibration method, the present invention also proposes a configurable hand-eye calibration device, including an image acquisition module 100, a display screen 110, a display screen 2 120, a calibration board with a backlight source 130, a light source controller 140, a robotic arm 150, an embedded control system 160, and a PC-based control system 170;

[0090] The image acquisition module 100, the embedded control system 160 and the PC-based control system 170 are connected via Ethernet; the backlight source calibration board 130 is connected to the light source controller 140; the robotic arm 150 is connected to the embedded control system 160; the display screen 110 is connected to the embedded control system 160, and parameter settings are performed through the display screen 110; the display screen 2 120 is connected to the PC-based control system 170, and parameter settings are performed through the display screen 2 120.

[0091] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A configurable hand-eye calibration method, characterized in that: It includes offline configuration phase and online automatic calibration phase; Offline configuration includes the following steps: Offline teaching can collect multiple robot poses with clear images and record them in the pose queue to be collected; Generate no less than two calibration plate images with different poses as needed; Online automatic calibration includes rough calibration and fine calibration; The rough calibration includes the following steps: Step a: The robot moves to the first position in the queue to be collected when the offline configuration result is initialized; Step b, collect the calibration plate image and remove the pose from the queue; Step c: determine the quality of the calibration plate image. If it meets the requirements, save the current actual posture and calibration plate image. Otherwise, execute step a. Step d, determine whether the queue to be collected is empty, if so, the robot movement ends and executes step e, otherwise executes step a; Step e: determine whether there are at least N valid images. If so, calculate the hand-eye relationship matrix and its reprojection error based on the collected robot posture and calibration plate image data, calibrate the camera internal parameters, and calculate the posture of the calibration plate in the robot coordinate system. Otherwise, the calibration fails and re-perform offline configuration. Fine calibration includes the following steps: Step f, according to the camera intrinsic parameters of the rough calibration, the hand-eye calibration results, and the position and posture of the calibration plate in the robot coordinate system, the corresponding robot arm end position and posture are calculated and added to the robot position and posture queue in sequence; Step g: the robot moves to the position and collects the image of the calibration plate; Step h, determine whether the calibration plate image can be correctly identified, if so, save the current actual posture and the actual calibration plate image, otherwise execute step g; Step i: determine whether the calibration plate image queue is empty. If so, execute step j; otherwise, execute step g; Step j, determine whether the number of valid images is greater than or equal to N. If so, calculate the camera intrinsic parameters, hand-eye transformation matrix and its reprojection error, otherwise terminate the fine calibration directly; Offline configuration teaching robot posture includes the following steps: S1, initialize the pose queue and the minimum length of the pose queue; S2, the robot moves to a position and collects the image of the calibration plate; S3, determine whether the image is clear, if so, add the current posture to the posture queue, otherwise execute step S2; S4, determine whether the length of the posture queue is equal to the minimum length, if so, the offline configuration teaching robot ends, otherwise execute step S2; Generating calibration plate images with different poses includes the following steps: Step 1: Initialize the camera internal parameters, nine-square grid area and logo; Step 2: Randomly generate a camera pose; Step 3: Generate a calibration plate image; Step 4: Determine whether the center of the calibration plate falls within the center area of ​​the nine-square grid; if so, save the calibration plate image, otherwise execute step 2; Step 5: Determine whether each area is covered by at least two images with different positions. If so, end the process; otherwise, proceed to step 2. In the fine calibration stage, the formula for calculating the end-arm pose in step f is as follows: In the formula, is the pose of the tool end in the robot base coordinate system; is the position of the calibration plate in the robot base coordinate system It is the position and posture of the calibration plate in the camera coordinate system, which is calculated based on the camera intrinsic parameters and the physical model of the calibration plate; is the pose of the tool end in the camera coordinate system.

2. A configurable hand-eye calibration method according to claim 1, characterized in that: The value of N is greater than or equal to 9.

3. A configurable hand-eye calibration device, used to implement the configurable hand-eye calibration method according to claim 1, characterized in that: It comprises an image acquisition module (100), a first display screen (110), a second display screen (120), a calibration plate with a backlight source (130), a light source controller (140), a mechanical arm (150), an embedded control system (160), and a PC-based control system (170); The image acquisition module (100), the embedded control system (160) and the PC-based control system (170) are connected via Ethernet; the backlight source calibration board (130) is connected to the light source controller (140); the mechanical arm (150) is connected to the embedded control system (160); the first display screen (110) is connected to the embedded control system (160), and parameter settings are performed via the first display screen (110); the second display screen (120) is connected to the PC-based control system (170), and parameter settings are performed via the second display screen (120).

Images