Industrial robot tail end pose error detection method based on binocular camera

By combining binocular cameras and lidar sensors, edge extraction method and stereo matching algorithm optimization, and triangulation method, the problem of light sensitivity of industrial robot terminal pose error detection is solved, and the accuracy and accuracy of detection is improved to ensure that Stability under different lighting conditions.

CN119068049AInactive Publication Date: 2024-12-03YANCHENG INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410898754.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-05
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the prior art, the end position error detection method of industrial robots is sensitive to ambient light, resulting in large image deviations, affecting the accuracy and accuracy of matching. Especially in night or in scenes with insufficient light, the image quality decreases and the accuracy of depth estimation decreases.

Method used

Using the end position error detection method based on binocular cameras, combined with the lidar sensor fusion technology, the edge extraction method and stereo matching algorithm are optimized, and the precise distance of the object is calculated using triangulation method, the error of the end position is evaluated, and the illumination pair is reduced. The effect of matching results.

Benefits of technology

It improves the accuracy and accuracy of detection, reduces the impact of light changes on detection results, improves image quality, and ensures the stability and reliability of detection under various lighting conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068049A_ABST
    Figure CN119068049A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of error detection, in particular to an industrial robot tail end pose error detection method based on a binocular camera, and the method comprises the detection steps: S1, carrying out the shooting of the tail end pose of a robot through the binocular camera, and obtaining an industrial robot image; s2, uploading the images shot by the binocular camera to an image processing module, extracting the features of the images by using an edge extraction method, and obtaining the difference between the two groups of images shot by the binocular camera; s3, combining with a laser radar sensor fusion technology, fusing the data, and making up for the defect that a binocular camera is sensitive to light. The laser radar sensor fusion technology is used, the binocular camera is combined for detection, the stereo matching algorithm is optimized, the defect that the binocular camera is sensitive to light is overcome, the influence degree of environment light on the binocular camera is reduced, the situation that the image deviation is large is reduced, the matching accuracy and precision are improved, and the user experience is improved. And image quality is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of error detection, and in particular relates to a method for detecting position and posture errors of an industrial robot terminal based on a binocular camera. Background Art

[0002] Binocular camera is an important computer vision device. It simulates the stereoscopic vision system of human eyes through two cameras to obtain three-dimensional information of objects in the scene. Binocular camera has the following advantages: low hardware requirements and low cost. Ordinary CMOS camera can meet the demand, which makes the binocular camera cost-effective. It is applicable to both indoor and outdoor use. As long as the light is suitable, the binocular camera can work normally. By calculating the image difference between the two cameras, high-precision depth information can be obtained. Industrial robots are multi-joint manipulators or multi-degree-of-freedom machine devices widely used in the industrial field. They have a certain degree of automation and can rely on their own power and control capabilities to achieve various industrial processing and manufacturing functions.

[0003] The end pose error refers to the actual deviation of the end effector of an industrial robot, such as the end of a robotic arm, from the desired or target pose. This error can affect the accuracy and reliability of the industrial robot and has an important impact on the performance of the robot. In the prior art, visual measurement is usually used to detect the end pose. By acquiring images of the surface of the industrial robot and performing image processing and analysis, the geometric characteristics and pose information of the workpiece are obtained. However, this detection method is sensitive to the lighting of the environment. Line changes may cause large image deviations, which in turn affect the accuracy and precision of the matching. At night or in scenes with insufficient lighting, the image quality deteriorates, which may lead to a decrease in the accuracy of depth estimation. In this regard, we propose a method for detecting the end pose error of an industrial robot based on a binocular camera. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for detecting the position and posture errors of an industrial robot terminal based on a binocular camera.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a method for detecting the position error of an industrial robot end based on a binocular camera, the detection steps are:

[0006] S1. Use a binocular camera to capture the robot's terminal posture and obtain an industrial robot image;

[0007] S2, uploading the images taken by the binocular camera to the image processing module, using the edge extraction method to extract the features of the image, and obtaining the difference between the two sets of images taken by the binocular camera;

[0008] S3, combined with LiDAR sensor fusion technology, the data is fused to make up for the lack of light sensitivity of binocular cameras;

[0009] S4, optimize the stereo matching algorithm, reduce the influence of illumination on the matching results, and compare two images from different perspectives;

[0010] S5. Calculate the precise distance of the object through triangulation and evaluate the error of the end position.

[0011] Preferably, in step S1, the binocular camera is composed of two groups of cameras, and the steps of obtaining the image of the industrial robot by the binocular camera are:

[0012] A1. Obtain the internal and external parameters of the binocular camera. The internal parameters include focal length, principal point coordinates and distortion coefficient, and the external parameters include the conversion between the camera and the coordinate system.

[0013] A2. Calibrate the binocular camera to eliminate distortion and keep the optical axes of the two cameras parallel;

[0014] A3. Take a photo of the industrial robot and upload the captured image.

[0015] Preferably, in step A1, the internal and external parameters of the binocular camera are obtained by shooting an image with known world coordinates, and the parameters of the camera are solved through the image. The parameters are solved through EXIF ​​information. EXIF ​​includes the focal length, image size and image center point of the image, wherein the focal length value is directly read from the EXIF ​​information, and the image center point formula is:

[0016]

[0017] Where cx and cy are the center points of the image, W is the width pixel of the image, and H is the height pixel of the image. In step A2, the binocular camera calibration is performed through parallel calibration and stereo calibration to align the positions of the binocular cameras.

[0018] Preferably, in step S2, the image processing module extracts the features of the image by using an edge extraction method, and the edge extraction method uses the Canny operator for detection. The Canny operator calculates the gradient strength and direction of the pixel points in the image, performs a non-maximum suppression operation, processes the threshold, and obtains the edge image from which the features are finally extracted based on the edge candidate points determined by the high threshold and the edge candidate points connected by the low threshold, so as to obtain the features in the two groups of images, and compare the two groups of features to obtain the difference in the features, wherein the gradient calculation formula is:

[0019] G=sqrt(Gx∧2+Gy∧2)

[0020] Where Gx and Gy are the gradients in the x and y directions, and sqrt is the square root.

[0021] Preferably, the threshold processing includes two thresholds, a high threshold and a low threshold;

[0022] When the gradient amplitude of a pixel is greater than a high threshold, the pixel is considered an edge point. When the gradient amplitude of a pixel is less than a low threshold, the pixel is suppressed. When the gradient amplitude of a pixel is between two thresholds, if the pixel is connected to a determined edge point, the pixel is retained, otherwise the pixel is suppressed to ensure that the detected edge points are continuous and reduce breakpoints and noise.

[0023] The Canny operator realizes effective edge detection of images through denoising, gradient calculation, non-maximum suppression, threshold processing and edge connection steps;

[0024] The Canny operator aims to find an optimal edge detection algorithm that meets three main criteria:

[0025] Good detection,the algorithm can identify as many actual edges in the image as possible;

[0026] For good positioning, the identified edges should be as close as possible to the actual edges in the actual image;

[0027] Minimum response, an edge in the image can only be identified once, and possible image noise should not be identified as an edge.

[0028] Preferably, the LiDAR sensor fusion technology emits lasers to the industrial robot to measure the round-trip time, to perform three-dimensional perception of the error of the terminal position of the industrial robot, and to fuse it with the binocular camera to produce a three-dimensional point cloud and fuse the data.

[0029] Preferably, the stereo matching algorithm optimizes the local matching strategy and dynamically adjusts the search window size according to the characteristics of the local area to improve the accuracy of the matching algorithm, and uses the optimized stereo matching algorithm to find the difference between pixels in the two images to calculate the depth information.

[0030] Preferably, triangulation is used to calculate the position, distance and height information, and the unknown quantity is calculated by the sine theorem, cosine theorem and tangent theorem formulas, wherein the sine theorem formula is:

[0031]

[0032] In which triangle ABC, ABC is the opposite angle of the triangle, abc is the three sides of the triangle, and R is the radius of triangle ABC;

[0033] The cosine theorem formula is:

[0034] a 2 =b 2 +c 2-2bccosA

[0035] b 2 =a 2 +c 2 -2accosB

[0036] c 2 =a 2 +b 2 -2abcosC

[0037] In triangle ABC, ABC is the opposite angle of the triangle, and abc is the three sides of the triangle.

[0038] Preferably, the evaluation method in step S5 is to establish a triangle in the plane space, calculate the position of the target point relative to the reference point by triangulation, and compare it with the target position measured by the binocular camera to evaluate the error of the end posture of the industrial robot.

[0039] Preferably, the EXIF ​​information is used to record the camera settings and shooting situation data when the photo is taken.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] This application uses the technology of LiDAR sensor fusion to combine with binocular camera for detection and optimizes the stereo matching algorithm to compensate for the lack of light sensitivity of the binocular camera, reduce the impact of ambient light on the binocular camera, and improve the accuracy and precision of matching in situations with large image deviations, thereby improving image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of the detection steps of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The present invention provides a technical solution: a method for detecting the end position error of an industrial robot based on a binocular camera, wherein the detection steps are as follows:

[0045] S1. Use a binocular camera to capture the robot's terminal posture and obtain an industrial robot image;

[0046] S2, uploading the images taken by the binocular camera to the image processing module, using the edge extraction method to extract the features of the image, and obtaining the difference between the two sets of images taken by the binocular camera;

[0047] S3, combined with LiDAR sensor fusion technology, the data is fused to make up for the lack of light sensitivity of binocular cameras;

[0048] S4, optimize the stereo matching algorithm, reduce the influence of illumination on the matching results, and compare two images from different perspectives;

[0049] S5. Calculate the precise distance of the object through triangulation and evaluate the error of the end position.

[0050] Furthermore, in step S1, the binocular camera is composed of two groups of cameras, and the steps of obtaining the industrial robot image by the binocular camera are:

[0051] A1. Obtain the internal and external parameters of the binocular camera. The internal parameters include focal length, principal point coordinates and distortion coefficient, and the external parameters include the conversion between the camera and the coordinate system.

[0052] A2. Calibrate the binocular camera to eliminate distortion and keep the optical axes of the two cameras parallel;

[0053] A3. Take a photo of the industrial robot and upload the captured image.

[0054] Furthermore, in step A1, the internal and external parameters of the binocular camera are obtained by shooting an image with known world coordinates, and the camera parameters are solved through the image. The parameters are solved through EXIF ​​information. EXIF ​​includes the focal length, image size and image center point of the image. The focal length value is directly read from the EXIF ​​information. The formula for the image center point is:

[0055]

[0056] Where cx and cy are the center points of the image, W is the width pixel of the image, and H is the height pixel of the image. In step A2, the binocular camera calibration is performed by parallel calibration and stereo calibration to align the position of the binocular camera.

[0057] The binocular camera obtains depth information by calculating the difference between two images, and has high accuracy. The binocular camera does not rely on external light sources and can work stably under various lighting conditions. The binocular camera uses stereo vision technology and has strong anti-interference ability in complex environments.

[0058] The binocular camera calibration step uses a binocular camera to take a series of images containing a checkerboard from different angles. These images will be used for subsequent camera calibration. The left eye image data set is used to calibrate the left eye camera to obtain the left eye intrinsic parameter matrix K1 and the left eye distortion coefficient vector D1; the right eye image data set is used to calibrate the right eye camera to obtain the right eye intrinsic parameter matrix K2 and the right eye distortion coefficient vector D2;

[0059] Stereo calibration: Take the parameters K1, K2, D1, and D2 measured by the left and right eyes as input, and then use the chessboard image with one-to-one correspondence between the left and right eyes to call the stereoCalibrate function to output the rotation matrix R and translation vector T of the left and right eyes;

[0060] Based on the measured K1, K2, D1, D2, R, and T, binocular vision image correction is performed. The purpose of this step is to eliminate distortion so that the corresponding points in the left and right images are on the same horizontal line, thereby simplifying the subsequent 3D reconstruction or matching process;

[0061] Parallel correction: The source image pixel coordinate system is converted into the camera coordinate system through the intrinsic parameter matrix, and parallel epipolar correction is performed through the rotation matrices R1 and R2 to ensure that the corresponding epipolar lines in the left and right eye images are on the same horizontal line; after correction, the camera coordinate system is converted into the image pixel coordinate system through the intrinsic parameter matrix, and the pixel values ​​of the source image coordinates are assigned to the new image coordinates to complete the correction and eliminate the distortion of the image.

[0062] Furthermore, in step S2, the image processing module extracts the features of the image by using the edge extraction method. The edge extraction method uses the Canny operator for detection. The Canny operator calculates the gradient strength and direction of the pixel points in the image, performs non-maximum suppression operation, processes the threshold, and obtains the edge image with the final feature extraction based on the edge candidate points determined by the high threshold and the edge candidate points connected by the low threshold to obtain the features in the two groups of images. The two groups of features are compared to obtain the difference in features, where the gradient calculation formula is:

[0063] G=sqrt(Gx∧2+Gy∧2)

[0064] Where Gx and Gy are the gradients in the x and y directions, and sqrt is the square root.

[0065] Furthermore, the threshold processing includes two thresholds, a high threshold and a low threshold;

[0066] When the gradient amplitude of a pixel is greater than a high threshold, the pixel is considered an edge point. When the gradient amplitude of a pixel is less than a low threshold, the pixel is suppressed. When the gradient amplitude of a pixel is between two thresholds, if the pixel is connected to a determined edge point, the pixel is retained, otherwise the pixel is suppressed to ensure that the detected edge points are continuous and reduce breakpoints and noise.

[0067] The Canny operator realizes effective edge detection of images through denoising, gradient calculation, non-maximum suppression, threshold processing and edge connection steps;

[0068] The Canny operator aims to find an optimal edge detection algorithm that meets three main criteria:

[0069] Good detection,the algorithm can identify as many actual edges in the image as possible;

[0070] For good positioning, the identified edges should be as close as possible to the actual edges in the actual image;

[0071] Minimum response, an edge in the image can only be identified once, and possible image noise should not be identified as an edge.

[0072] Furthermore, the laser radar sensor fusion technology emits lasers to the industrial robot to measure the round-trip time, to perform three-dimensional perception of the error of the terminal position of the industrial robot, and to fuse it with the binocular camera to produce a three-dimensional point cloud and fuse the data;

[0073] Cameras can obtain color and texture information, but lack depth information. By fusing with lidar, this shortcoming of cameras can be compensated and richer environmental perception information can be provided. For example, lidar can generate high-precision three-dimensional point clouds, while cameras can provide color and texture information of these points.

[0074] Furthermore, the stereo matching algorithm optimizes the local matching strategy and dynamically adjusts the search window size according to the characteristics of the local area to improve the accuracy of the matching algorithm. The optimized stereo matching algorithm is used to find the difference between the pixels in the two images and calculate the depth information.

[0075] The basic principle of the stereo matching algorithm is to calculate the depth information by comparing the differences between the corresponding pixels in the left and right images. Specifically, it uses two cameras to obtain the left and right views of the same scene from different angles, and then uses the algorithm to find the matching pixel pairs in the left and right views. Once these matching point pairs are found, the triangulation principle can be used to restore the position of the physical point in the three-dimensional space.

[0076] Furthermore, triangulation is used to calculate the position, distance and height information, and the unknown quantities are calculated through the sine theorem, cosine theorem and tangent theorem formulas, where the sine theorem formula is:

[0077]

[0078] In which triangle ABC, ABC is the opposite angle of the triangle, abc is the three sides of the triangle, and R is the radius of triangle ABC;

[0079] The cosine theorem formula is:

[0080] a 2 =b 2 +c 2 -2bccosA

[0081] b 2 =a 2 +c 2 -2accosB

[0082] c 2 =a 2 +b 2 -2abcosC

[0083] In triangle ABC, ABC is the opposite angle of the triangle, and abc is the three sides of the triangle.

[0084] Furthermore, in step S5, the evaluation method is to establish a triangle in the plane space, calculate the position of the target point relative to the reference point by triangulation, and compare it with the target position measured by the binocular camera to evaluate the error of the end position of the industrial robot;

[0085] Triangulation is a geographical survey method based on the properties of triangles and the principles of trigonometric functions. It has high accuracy, high reliability and wide applicability.

[0086] Furthermore, EXIF ​​information is used to record the camera settings and shooting situation data when the photo was taken;

[0087] The recorded data includes:

[0088] Basic camera settings, including exposure time, aperture value, sensitivity, and focal length;

[0089] Shooting conditions, shooting date and time, white balance setting, flash status;

[0090] Image information, image pixel dimensions, file size, image format;

[0091] Camera and lens information, camera model, lens model;

[0092] GPS data, including latitude, longitude, and altitude of the shooting location.

[0093] The above description only describes the specific implementation mode of the present invention. Various examples do not limit the essential content of the present invention. After reading the description, ordinary technicians in the relevant technical field can modify or deform the specific implementation mode described above without departing from the essence and scope of the invention.

Claims

1. A method for detecting the end position error of an industrial robot based on a binocular camera, characterized in that: The detection steps are: S1. Use a binocular camera to capture the robot's terminal posture and obtain an industrial robot image; S2, uploading the images taken by the binocular camera to the image processing module, using the edge extraction method to extract the features of the image, and obtaining the difference between the two sets of images taken by the binocular camera; S3, combined with LiDAR sensor fusion technology, the data is fused to make up for the lack of light sensitivity of binocular cameras; S4, optimize the stereo matching algorithm, reduce the influence of illumination on the matching results, and compare two images from different perspectives; S5. Calculate the precise distance of the object through triangulation and evaluate the error of the end position.

2. The method for detecting the position error of an industrial robot end based on a binocular camera according to claim 1, characterized in that: In step S1, the binocular camera is composed of two groups of cameras. The steps for the binocular camera to obtain the image of the industrial robot are as follows: A1. Obtain the internal and external parameters of the binocular camera. The internal parameters include focal length, principal point coordinates and distortion coefficient, and the external parameters include the conversion between the camera and the coordinate system. A2. Calibrate the binocular camera to eliminate distortion and keep the optical axes of the two cameras parallel; A3. Take a photo of the industrial robot and upload the captured image.

3. The method for detecting the position error of an industrial robot end based on a binocular camera according to claim 2, characterized in that: In step A1, the internal and external parameters of the binocular camera are obtained by shooting an image with known world coordinates, and the camera parameters are solved through the image. The parameters are solved through EXIF ​​information. EXIF ​​includes the focal length, image size and image center point of the image. The focal length value is directly read from the EXIF ​​information. The formula for the image center point is: Where cx and cy are the center points of the image, W is the width pixel of the image, and H is the height pixel of the image. In step A2, the binocular camera calibration is performed through parallel calibration and stereo calibration to align the positions of the binocular cameras.

4. The method for detecting the position error of an industrial robot end based on a binocular camera according to claim 1, characterized in that: In step S2, the image processing module extracts the features of the image by using the edge extraction method. The edge extraction method uses the Canny operator for detection. The Canny operator calculates the gradient strength and direction of the pixel points in the image, performs non-maximum suppression operations, processes the threshold, and obtains the edge image with the final feature extraction based on the edge candidate points determined by the high threshold and the edge candidate points connected by the low threshold to obtain the features in the two groups of images. The two groups of features are compared to obtain the difference in features, where the gradient calculation formula is: G=sqrt(Gx^2+Gy∧2) Where Gx and Gy are the gradients in the x and y directions, and sqrt is the square root.

5. The method for detecting the position error of an industrial robot end based on a binocular camera according to claim 4, characterized in that: Threshold processing includes two thresholds, a high threshold and a low threshold; When the gradient amplitude of a pixel is greater than a high threshold, the pixel is considered to be an edge point. When the gradient amplitude of a pixel is less than a low threshold, the pixel is suppressed. When the gradient amplitude of a pixel is between two thresholds, if the pixel is connected to a determined edge point, the pixel is retained, otherwise the pixel is suppressed to ensure that the detected edge points are continuous and reduce breakpoints and noise.

6. The method for detecting the position error of an industrial robot end based on a binocular camera according to claim 1, characterized in that: LiDAR sensor fusion technology emits lasers to the industrial robot to measure the round-trip time, to perform three-dimensional perception of the error of the industrial robot's terminal position, and to fuse it with the binocular camera to produce a three-dimensional point cloud and fuse the data.

7. The method for detecting the position error of an industrial robot end based on a binocular camera according to claim 1, characterized in that: The stereo matching algorithm optimizes the local matching strategy and dynamically adjusts the search window size according to the characteristics of the local area to improve the accuracy of the matching algorithm. The optimized stereo matching algorithm is used to find the difference between pixels in the two images and calculate the depth information.

8. The method for detecting the position error of an industrial robot end based on a binocular camera according to claim 1, characterized in that: Triangulation is used to calculate position, distance and height information. The unknown quantities are calculated through the sine theorem, cosine theorem and tangent theorem formulas. The sine theorem formula is: In which triangle ABC, ABC is the opposite angle of the triangle, abc is the three sides of the triangle, and R is the radius of triangle ABC; The cosine theorem formula is: to 2 =b 2 +c 2 -2bccosA b 2 =a 2 +c 2 -2accosB c 2 =a 2 +b 2 -2abcosC In triangle ABC, ABC is the opposite angle of the triangle, and abc is the three sides of the triangle.

9. The method for detecting the position error of an industrial robot end based on a binocular camera according to claim 1, characterized in that: The evaluation method in step S5 is to establish a triangle in the plane space, calculate the position of the target point relative to the reference point by triangulation, and compare it with the target position measured by the binocular camera to evaluate the error of the end position of the industrial robot.

10. The method for detecting the position error of an industrial robot end based on a binocular camera according to claim 3, characterized in that: EXIF information is used to record the camera settings and shooting conditions when the photo was taken.