A hand-eye calibration method based on deep learning

Abstract

The present invention provides a hand-eye calibration method based on deep learning, comprising the steps of: collecting and marking training images containing calibration objects; training yolo3 model; setting moving step length S and distance threshold D; control device randomly adjusts the position and posture of mechanical arm, and camera shoots to obtain adjustment image; calling trained yolo3 model to analyze the adjustment image, if the size of the calibration object is not less than half of the camera field of view and the actual distance is not greater than the distance threshold D, then shoot the calibration image, and record the position and posture data of the mechanical arm at this time, otherwise, shoot the adjustment image again after adjusting the mechanical arm according to the moving step length S, until it meets the shooting calibration image again; after the calibration image is calibrated to meet the required quantity, hand-eye calibration matrix calculation is performed. In the process of the present invention, no manual participation is required, manpower and material resources are saved, and the amount of calculation is small, the operation speed is fast, the shooting requirements of the calibration image are relatively low, the result accuracy is high, and the robustness is better.

Description

Technical Field

[0001] The invention relates to a hand-eye calibration method based on deep learning. Background Art

[0002] With the continuous improvement of the level of industrialization, the application of vision guidance technology in the field of industrial robots is becoming more and more extensive. Through visual guidance of industrial robots, robots can not only perform preset actions, but also locate the position of the grasped target object through machine vision for flexible grasping. Industrial robot hand-eye calibration is the basis for realizing machine vision guidance, that is, determining the conversion matrix between the camera coordinate system and the robot coordinate system through hand-eye calibration. Most of the existing hand-eye calibration methods use manual intervention to collect data, that is, manually control the movement of the robotic arm to make the calibration object in a suitable position, and change the posture of the robotic arm to obtain multiple sets of robotic arm postures and calibration object images corresponding to their postures, and finally realize the conversion between the camera coordinate system and the robot coordinate system. This method consumes more manpower and material resources, and requires operators to have certain professional skills. Interference or incorrect operation during manual operation will affect the final calibration result.

[0003] In the prior art, there are also ways to achieve automatic calibration, such as the application number 202010022816.X (the patent name is a fully automatic robot hand-eye calibration method) and the application number 202010079136.1 (the patent name is an automatic hand-eye calibration system and method for a robot motion vision system). However, they still have certain defects in actual use: the former needs to obtain the center position information of the current calibration object and the size of the calibration object image through corner point calculation. This method has a large amount of calculation and takes a long time. When the calibration object is too small in the camera field of view, it will be difficult to extract the corner points, that is, it is impossible to obtain the position information of the calibration object, and the robustness is poor; the latter solves the cantilever effect of the robot arm through three-point calibration, and three-point calibration is required before each calibration, and manual intervention by staff with certain professional knowledge is still required. Summary of the invention

[0004] The purpose of the present invention is to propose a hand-eye calibration method based on deep learning in response to the shortcomings of the prior art. The method does not require human intervention in the process, saving manpower and material resources, and has a small amount of calculation, fast calculation speed, and low requirements for the shooting of calibration images. The result has high accuracy and better robustness.

[0005] The present invention is achieved through the following technical solutions:

[0006] A hand-eye calibration method based on deep learning includes the following steps:

[0007] A. Collect training images containing calibration objects and mark the calibration object areas in each training image;

[0008] B. Use training images to train the deep learning model YOLO3, so that the YOLO3 model has the ability to identify and locate calibration objects;

[0009] C. Set the moving step length S of the robot arm and the distance threshold D between the center of the calibration object and the center of the camera field of view;

[0010] D. The control device randomly adjusts the position and posture of the robot arm, and after the calibration object is within the camera field of view, the camera takes pictures to obtain the adjustment image and feeds the adjustment image back to the control device;

[0011] E. The control device calls the yolo3 model trained in step B to analyze the adjustment image, and obtains the size of the calibration object on the adjustment image, and the actual distance between the center of the calibration object and the center of the camera field of view. If the size of the calibration object is not less than half of the camera field of view and the actual distance is not greater than the distance threshold D, then enter step G, otherwise, enter step F;

[0012] F. The control device adjusts the position and posture of the robotic arm once according to the size of the calibration object and the actual distance between the center of the calibration object and the center of the camera field of view according to the moving step length S of the robotic arm, and then the camera takes pictures to obtain the adjusted image and feeds the adjusted image back to the control device; and then enters step E;

[0013] G. The camera takes pictures to obtain the calibration image and feeds the calibration image back to the control device. The control device also records the position and posture data of the robot arm at this time, and determines whether the calibration images obtained meet the required number. If so, proceed to step H, otherwise, proceed to step D;

[0014] H. Calculate the hand-eye calibration matrix based on all calibration images and the position and posture data corresponding to each calibration image to complete the hand-eye calibration.

[0015] Furthermore, the moving step length S ranges from 1 cm to 5 cm.

[0016] Furthermore, the distance threshold D is determined according to the selected camera parameters.

[0017] Furthermore, the distance threshold D ranges from 10 pixels to 100 pixels.

[0018] Furthermore, in step D, the control device randomly adjusts the position and posture of the robotic arm and ensures that a complete calibration object image exists in the camera field of view.

[0019] Furthermore, in step A, the calibration object is placed in different scenes, and the calibration object is photographed from different perspectives and different fields of view to obtain training images, and the calibration object area in each training image is annotated using a picture annotation tool.

[0020] Furthermore, in the step B, the training images with the calibration object areas marked in step A are randomly divided according to a ratio of 2:8, 80% of the training images are used for training the yolo3 model, and 20% of the training images are used for performance testing of the yolo3 model.

[0021] Furthermore, in step H, the hand-eye calibration matrix calculation formula is:

[0022]

[0023]

[0024] in, b T g : The homogeneous matrix of the robot end coordinate system relative to the base coordinate system;

[0025] c T t : The homogeneous matrix of the calibration plate coordinate system relative to the camera coordinate system;

[0026] b T t : The homogeneous matrix of the calibration plate coordinate system relative to the base coordinate system;

[0027] g T c : The homogeneous matrix of the camera coordinate system relative to the end of the robotic arm (i.e. the solved hand-eye calibration matrix);

[0028] Furthermore, in step G, the number of calibrated images must be greater than 10.

[0029] Furthermore, the control device is a robotic arm controller.

[0030] The present invention has the following beneficial effects:

[0031] 1. The present invention first shoots an adjustment image, and analyzes the adjustment image through the deep learning model yolo3 to obtain the size of the calibration object on the adjustment image, and the actual distance between the center of the calibration object and the center of the camera field of view. If the size of the calibration object is not less than half of the camera field of view and the actual distance is not greater than a preset distance threshold D, the camera is controlled to shoot to obtain the calibration image, and the position and posture data of the mechanical arm at this time are recorded. Otherwise, the position and posture of the mechanical arm are continued to be adjusted according to the preset moving step S, and the process of shooting and analyzing the adjustment image is repeated until the requirement for shooting the calibration image is met. In this way, a control closed loop is formed. After obtaining a sufficient number of calibration images, the hand-eye calibration matrix calculation can be performed. This process does not require manual participation, and naturally does not need to be subject to the professional skills of the operator. It can also greatly save manpower and material resources. The trained yolo3 model is used to analyze the adjustment image, with small calculation amount, fast calculation speed, and low shooting requirements for the calibration image. As long as the size of the calibration object is not less than half of the camera field of view, a correct analysis result can be obtained, so the accuracy is higher and the robustness is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present invention is further described in detail below with reference to the accompanying drawings.

[0033] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0034] like Figure 1 As shown, the hand-eye calibration method based on deep learning includes the following steps:

[0035] A. Collect training images containing calibration objects, and mark the calibration object areas in each training image; specifically, place the calibration object in different scenes, and adjust the calibration object position to shoot the calibration object from different perspectives (such as placing the calibration object in the middle of the training image, and part of the calibration object in the training image), and adjust the camera model to shoot the calibration object in different fields of view to obtain a sufficient number of training images, and use image annotation tools (such as labelimage software) to mark the calibration object area in each training image; the calibration object can be an object with a special mark, which is a prior art;

[0036] B. Use the training images to train the deep learning model YOLO3, so that the YOLO3 model has the ability to identify and locate the calibration object; specifically: the training image with the calibration object area marked in step A is randomly divided according to the ratio of 2:8, 80% of the training images are used for the training of the YOLO3 model, and 20% of the training images are used for the performance test of the YOLO3 model; in this embodiment, the parameters of the YOLO3 model are set as follows: the training image size is scaled to 416×416×3, the batch training sample number batch is set to 64, the learning rate learning_rate is set to 0.001, the maximum number of iterations is set to 100000, and loss(object) is used as the loss function, and its calculation formula is as follows:

[0037]

[0038] in, Indicates whether the jth anchor box of the ith grid is responsible for this object. If so, Otherwise it is 0, and the loss function is the existing technology;

[0039] C. Set the moving step S of the robot arm and the distance threshold D between the center of the calibration object and the center of the camera field of view, set the camera at the end of the robot arm, and place the calibration object on the calibration plane; wherein the moving step S ranges from 1 cm to 5 cm, and in this embodiment, S is 3 cm; the distance threshold D is determined according to the selected camera parameters, and its value range is from 10 pixels to 100 pixels. In this embodiment, D is 55 pixels; the center of the calibration object is obtained when the training image is annotated in step A;

[0040] D. The control device randomly adjusts the position and posture of the robot arm (i.e., starts a new round of calibration image acquisition), and after the calibration object is within the camera field of view, the camera takes pictures to obtain the adjustment image and feeds the adjustment image back to the control device; in order to enable the robot arm to adjust to a state where the calibration image can be taken more quickly during the subsequent adjustment process, it is usually ensured that a complete calibration object image exists in the camera field of view when the position and posture of the robot arm is randomly adjusted;

[0041] E. The control device calls the yolo3 model trained in step B to analyze the adjustment image, and obtains the size of the calibration object on the adjustment image, and the actual distance between the center of the calibration object and the center of the camera field of view. If the size of the calibration object is not less than half of the camera field of view and the actual distance is not greater than the distance threshold D, then enter step G, otherwise, enter step F;

[0042] F. The control device adjusts the position and posture of the robotic arm once according to the size of the calibration object and the actual distance between the center of the calibration object and the center of the camera field of view according to the moving step length S of the robotic arm, and then the camera takes pictures to obtain the adjusted image and feeds the adjusted image back to the control device; and then enters step E;

[0043] G. The camera takes pictures to obtain calibration images and feeds the calibration images back to the control device. The control device also records the position and posture data of the robot arm at this time, and determines whether the obtained calibration images meet the required number. If so, proceed to step H, otherwise, proceed to step D; wherein the number of calibration images must be greater than 10. In this embodiment, the number of calibration images is 30;

[0044] H. Calculate the hand-eye calibration matrix according to all calibration images and the position and posture data corresponding to each calibration image to complete the hand-eye calibration, wherein the hand-eye calibration matrix calculation formula (this formula is the prior art) is:

[0045]

[0046]

[0047] in, b T g : The homogeneous matrix of the robot end coordinate system relative to the base coordinate system;

[0048] c T t : The homogeneous matrix of the calibration plate coordinate system relative to the camera coordinate system;

[0049] b T t : The homogeneous matrix of the calibration plate coordinate system relative to the base coordinate system;

[0050] g T c : The homogeneous matrix of the camera coordinate system relative to the end of the robotic arm (i.e. the hand-eye calibration matrix to be solved), matrix b T g , c T t , b T t The position and posture data of the robot arm corresponding to each calibration image are obtained, and this process is a prior art.

[0051] Among them, the control device is a robotic arm controller.

[0052] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A hand-eye calibration method based on deep learning, characterized in that: The steps include: A. Collect training images containing calibration objects and mark the calibration object areas in each training image; B. Use training images to train the deep learning model YOLO3, so that the YOLO3 model has the ability to identify and locate calibration objects; C. Set the moving step length S of the robot arm and the distance threshold D between the center of the calibration object and the center of the camera field of view; D. The control device randomly adjusts the position and posture of the robot arm, and after the calibration object is within the camera field of view, the camera takes pictures to obtain the adjustment image and feeds the adjustment image back to the control device; E. The control device calls the yolo3 model trained in step B to analyze the adjustment image, and obtains the size of the calibration object on the adjustment image, and the actual distance between the center of the calibration object and the center of the camera field of view. If the size of the calibration object is not less than half of the camera field of view and the actual distance is not greater than the distance threshold D, then enter step G, otherwise, enter step F; F. The control device adjusts the position and posture of the robotic arm once according to the size of the calibration object and the actual distance between the center of the calibration object and the center of the camera field of view, and the robotic arm's moving step length S. The camera then takes a picture to obtain an adjusted image and feeds the adjusted image back to the control device. And proceed to step E; G. The camera takes pictures to obtain the calibration image and feeds the calibration image back to the control device. The control device also records the position and posture data of the robot arm at this time, and determines whether the calibration images obtained meet the required number. If so, proceed to step H, otherwise, proceed to step D; H. Calculate the hand-eye calibration matrix based on all calibration images and the position and posture data corresponding to each calibration image to complete the hand-eye calibration; In step H, the hand-eye calibration matrix calculation formula is: in, b T g : The homogeneous matrix of the robot end coordinate system relative to the base coordinate system; c T t : The homogeneous matrix of the calibration plate coordinate system relative to the camera coordinate system; b T t : The homogeneous matrix of the calibration plate coordinate system relative to the base coordinate system; g T c : The homogeneous matrix of the camera coordinate system relative to the end of the robotic arm, that is, the solved hand-eye calibration matrix.

2. The hand-eye calibration method based on deep learning according to claim 1, characterized in that: The moving step length S ranges from 1 cm to 5 cm.

3. The hand-eye calibration method based on deep learning according to claim 1, characterized in that: The distance threshold D is determined according to the selected camera parameters.

4. The hand-eye calibration method based on deep learning according to claim 3, characterized in that: The distance threshold D ranges from 10 pixels to 100 pixels.

5. A hand-eye calibration method based on deep learning according to claim 1, 2, 3 or 4, characterized in that: In step D, the control device randomly adjusts the position and posture of the robotic arm and ensures that a complete calibration object image exists in the camera field of view.

6. A hand-eye calibration method based on deep learning according to claim 1, 2, 3 or 4, characterized in that: In step A, the calibration object is placed in different scenes, and the calibration object is photographed from different perspectives and different fields of view to obtain training images, and the calibration object area in each training image is annotated using a picture annotation tool.

7. A hand-eye calibration method based on deep learning according to claim 1 or 2 or 3 or 4, characterized in that: In the step B, the training images with the calibration object area marked in step A are randomly divided according to a ratio of 2:8, 80% of the training images are used for training the yolo3 model, and 20% of the training images are used for performance testing of the yolo3 model.

8. A hand-eye calibration method based on deep learning according to claim 1, 2, 3 or 4, characterized in that: In step G, the number of calibrated images must be greater than 10.

9. A hand-eye calibration method based on deep learning according to claim 1, 2, 3 or 4, characterized in that: The control device is a robotic arm controller.

Images