Robot disordered grabbing method based on binocular structure light point cloud reconstruction

Through binocular structured light point cloud reconstruction technology, the lack of flexibility and accuracy of traditional robot grasping methods is solved, and high-precision grabbing of complex environments and messy scenes is achieved, which improves the adaptability and freedom of robot grasping. Great.

CN118967758APending Publication Date: 2024-11-15JIANGSU UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410966024.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Traditional robot grasping methods lack flexibility and adaptability, cannot cope with dynamically changing environments, and have low grasping accuracy, making it difficult to adapt to complex visual environments and messy parts placement scenarios.

Method used

Using binocular structured light point cloud reconstruction technology, structured light is projected through a projector, binocular camera shoots and performs stereo matching, and the three-dimensional point cloud model is reconstructed using PCL library to perform robot hand-eye calibration and motion trajectory solution, and create a plane grabbing network. Implement unordered crawling.

Benefits of technology

It improves the accuracy and robustness of robot grasping parts, can cope with highly dynamic parts grabbing scenarios, and improves the grab accuracy and adaptability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967758A_ABST
    Figure CN118967758A_ABST
Patent Text Reader

Abstract

The invention discloses a robot disordered grabbing method based on binocular structured light point cloud reconstruction. The robot disordered grabbing method comprises the steps that structured light is projected to the surface of a to-be-grabbed part; the binocular camera shoots structured light on the surface of the part to be grabbed, stereo matching is carried out on structured light pictures shot by a left camera and a right camera in the binocular camera, and pixel points of the matched structured light pictures are obtained; restoring depth information according to the matched pixel points, and reconstructing a three-dimensional point cloud model for the matched pixel points by using a PCL library; performing hand-eye calibration and motion trail solving on the grabbing robot; and creating a plane grabbing network by using the obtained three-dimensional point cloud model and the motion trail of the grabbing robot to realize disordered grabbing of the robot. According to the method, the accuracy of disordered part grabbing of the robot can be effectively improved, and the problem that ambient light interferes with part information obtained by a camera is further solved on the basis of an existing grabbing technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a robot disordered grasping method based on binocular structured light point cloud reconstruction. Background Art

[0002] At present, the traditional robot grasping principle is usually based on pre-programmed path grasping, which requires the operator to manually program the robot's motion trajectory and grasping action according to the position and characteristics of the object. This method is relatively simple to implement, but lacks flexibility and adaptability, cannot cope with dynamically changing environments, and has low grasping accuracy. How to make robots adapt to more complex visual environments, cope with grasping scenarios with more cluttered parts placement, and how to achieve higher grasping accuracy when grasping parts, with the development of computer technology, the existing industrial production technology still has greater room for development. Summary of the invention

[0003] The present invention aims to solve the problems existing in the above-mentioned prior art and provides a robot disordered grasping method based on binocular structured light point cloud reconstruction.

[0004] The technical solutions adopted in the present invention are:

[0005] A robot disordered grasping method based on binocular structured light point cloud reconstruction includes the following steps:

[0006] S1) The projector projects structured light onto the surface of the part to be grasped;

[0007] S2) The binocular camera captures the structured light on the surface of the part to be grasped, and stereo matching is performed on the structured light images captured by the left and right cameras of the binocular camera to obtain the pixel points of the matched structured light images;

[0008] S3) restoring depth information according to the matched pixels, and reconstructing a 3D point cloud model based on the matched pixels using the PCL library;

[0009] S4) performing hand-eye calibration and motion trajectory solving for the grasping robot;

[0010] S5) Using the obtained three-dimensional point cloud model and the motion trajectory of the grasping robot, a planar grasping network is created to realize the robot's disordered grasping.

[0011] Furthermore, the projector projects a structured light including:

[0012] S21 The structured light is a cyclic complementary Gray code, which is a black and white long stripe of equal width, covering the part to be grasped. It is projected on the part to be grasped by the projector at a high rate and is a straight stripe when projected. It becomes a curved stripe when it falls on the part to be grasped, so that the binocular camera can shoot the stripe to calculate the depth information of the part;

[0013] When S22 stripes are projected, the projector projects black and white stripes with different densities at different time points, which continuously cycles from sparse to dense, and the connection between the black and white stripes of the previous frame is covered by the black or white stripes of the next frame, so that the binocular camera can shoot the stripes and calculate more detailed surface information of the parts.

[0014] Further, in S22:

[0015] The phase formula of the projector projecting Gray code, assuming that three phase differences are projected The phase formula is:

[0016]

[0017] Among them, I 1 , I 2 , I 3 is the gray value, I 0 is the average brightness, I a is the modulation amplitude of the fringes, For phase.

[0018] Furthermore, the steps of shooting structured light with the binocular camera and performing stereo matching using the structured light shot by the binocular camera are as follows:

[0019] S31) performing phase extraction on the structured light image taken by the binocular camera, and dedistorting the image to obtain a normalized image;

[0020] S32) performing phase unwrapping according to the obtained normalized image to obtain a continuous phase value;

[0021] S33) In the pictures of the left camera and the right camera, stereo matching estimation is performed using the phase value to obtain estimated matching points for pixel matching of the two pictures, and interpolation is performed within a set range of the estimated matching points to obtain accurate matching points;

[0022] S34) According to the obtained normalized image and the interpolated precise matching points, the disparity between each corresponding pair of pixels in the images taken by the left and right cameras is determined for subsequent depth information calculation.

[0023] Further, in step S31:

[0024] The formula for dedistortion is:

[0025]

[0026] Among them, (x, y) is the pixel point after dedistortion, (x distorted ,y distorted ) is the pixel before distortion, k 1 , k 2, k 3 is the radial distortion parameter, which is the internal parameter of the camera and is generally used as k 1 and k 2 , fisheye cameras use k 3 ; r is the distance from the pixel to the center of the image.

[0027] Furthermore, the steps of restoring the depth information based on the corresponding points and reconstructing the 3D point cloud model using the PCL library are as follows:

[0028] S41) calculating the depth value of each pair of pixels according to the disparity information obtained by matching and combining the geometric parameters of the binocular camera;

[0029] S42) Based on the precisely matched pixel points, point cloud 3D modeling is performed using the PCL library.

[0030] Further, in step S41:

[0031] The formula for calculating the depth value of each pair of pixels is:

[0032]

[0033] Where Z represents the depth value of the pixel, f represents the focal length of the binocular camera, and T represents the distance between the left and right cameras in the binocular camera; x l Indicates the image taken by the left camera, the distance between the pixel point of the image and the center of the image; x r Indicates the picture taken by the right camera, the distance between the pixel point of the picture and the center of the picture.

[0034] Furthermore, the steps for hand-eye calibration and motion trajectory solution of the robot grasping system are as follows:

[0035] S51 uses Zhang Zhengyou calibration method to solve the pose transformation of the grasping robot's hand and eye;

[0036] S52 uses Cartesian space planning algorithm to solve the motion trajectory of the grasping robot.

[0037] Furthermore, a planar grasping network is created for the obtained 3D point cloud model and the motion trajectory of the grasping robot. The grasping network extracts grasping points and grasping postures from the input image through a convolutional neural network to guide the manipulator to perform actual grasping operations.

[0038] The present invention has the following beneficial effects:

[0039] The present invention uses a binocular camera to capture a structured light image projected by a projector, performs deep resolution on the pixels in the image, and reconstructs a point cloud model of the part, thereby improving the accuracy of the robot grasping the part. In addition, since the structured light projected by the projector is refreshed quickly, the binocular camera also quickly refreshes the point cloud model of the part, so for highly dynamic part grasping scenes, this method improves the robustness of part grasping. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of a binocular camera and a projector to which the present invention is applied.

[0041] Figure 2 This is the schematic diagram of the depth algorithm.

[0042] Figure 3 It is a flow chart of the present invention.

[0043] Figure 4 The structured light is projected with black and white light and dark stripes.

[0044] Numbers in the figure: 1 binocular camera; 2 projector; 3 parts to be grasped. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings.

[0046] like Figures 1 to 3 The present invention provides a robot disordered grasping method based on binocular structured light point cloud reconstruction, comprising the following steps:

[0047] S1) The projector 2 projects structured light onto the surface of the part 3 to be grasped;

[0048] The structured light projected by projector 2 is a cyclic complementary Gray code, which is a long stripe of black and white and equal width, covering the part to be grasped. It is projected by the projector on the part to be grasped at a high rate, and is a straight stripe during projection, which becomes a curved stripe when it falls on the part to be grasped, so that the binocular camera can capture the stripe and calculate the depth information of the part.

[0049] When the stripes are projected, the projector projects black and white stripes with different densities at different time points, which continuously cycles from sparse to dense, and the connection between the black and white stripes of the previous frame is covered by the black or white stripes of the next frame, so that the binocular camera can capture the stripes and calculate more detailed surface information of the parts.

[0050] The phase formula of the projector projecting Gray code, assuming that three phase differences are projected The phase formula is:

[0051]

[0052] Among them, I 1 , I 2 , I 3 is the gray value, I 0 is the average brightness, I a is the modulation amplitude of the fringes, For phase.

[0053] S2) The binocular camera captures the surface structured light of the part to be grasped, and stereo matching is performed on the structured light images captured by the left and right cameras in the binocular camera to obtain the pixel points of the matched structured light images.

[0054] The specific steps are as follows:

[0055] S21) Two cameras in the binocular camera are located on both sides of the projector. In order to shoot the parts to be grasped at a specific distance, the binocular camera, the projector and the three form a certain angle relationship. This angle relationship can be appropriately adjusted according to the specific grasping scene. At the same time, the optical axes of the binocular camera, the projector and the three objects converge at one point. The focal length of the binocular camera can automatically focus on the part to be grasped, ensuring that a clear picture can be taken. At the same time, the projected black and white stripes can completely cover the surface of the part to be grasped. After the stripes projected by the projector cover the surface of the part to be grasped, the binocular camera takes a picture. At this time, the picture contains phase information due to the coverage of the black and white stripes. Then, the picture taken by the binocular camera is subjected to phase extraction. Here, only the area covered by the black and white stripes in the picture is subjected to phase extraction, and then the picture is dedistorted to obtain a normalized picture;

[0056] S22) performing phase unwrapping according to the obtained normalized image to obtain a continuous phase value. Phase unwrapping is specifically described here by using a fringe image without bending, such as Figure 4 As shown:

[0057] The purpose of phase unwrapping is to perform phase encoding on each column of pixels. Since the structured light projected by the projector is black and white stripes, the stripes are long strips from top to bottom. Adjacent black stripes and white stripes are a group, with a phase from 0 to 2π. The phase value at the junction of black and white is π. From left to right, multiple pairs of black and white stripes are repeated, and it is not appropriate to repeatedly use 0 to 2π for encoding in the column direction. Therefore, 0 to 2π is used as a cycle, and the phase of the next pair of black and white stripes after 2π is 2π to 4π, which increases successively to avoid duplication of phase values. In this way, each column of black and white pixels will have a unique phase value. These phase information corresponds to different surface heights of the parts to be grasped, which is used for the subsequent PCL library to solve the height information of the surface of the parts to be grasped.

[0058] S23) In the images of the left camera and the right camera, the phase value is used to perform stereo matching estimation. Before starting the matching, the two color images are converted into black and white images. Since the phase value of each column of pixels is the same and the pixel value of each row of pixels is increasing, a row-by-row matching strategy is adopted in stereo matching. When a pair of pixels in the left and right images are to be matched, the position of the pixel in the left image in the pixel coordinate system is first determined. Through the black and white grayscale information of the pixels around the pixel and the phase value of the pixel in the stripe code, the pixel point with the most similar grayscale value and phase value to the pixel in the left image is found around the same pixel coordinate system in the right image, and an estimated match for pixel matching of the two images is obtained. Point, since this is a stereo matching in the pixel coordinate system, there is a slight deviation between the position information expressed by the pixel coordinate system and the position of the point in the real environment. Although the pixel point in the pixel coordinate system is very small, the position information it can represent is not continuous in the metric unit. In many cases, the precise point we are looking for is between adjacent pixels. Therefore, according to the different capture accuracy required by the captured scene, the algorithm is used to find a more accurate point mapped in the real environment based on the pixel coordinate system. The overall process of the algorithm is to interpolate within the estimated matching point setting range, and then match the pixel grayscale value and phase encoding value of the interpolated pixel points one by one to obtain the precise matching point;

[0059] S24) According to the obtained normalized image and the interpolated precise matching points, the coordinate values ​​of the corresponding pixel points in the images taken by the left and right cameras in their respective pixel coordinate systems are determined. The difference between the coordinate values ​​of the two corresponding pixel points is called the disparity value, which is used for subsequent depth information solution.

[0060] Specifically, in step S21), the formula for dedistortion is:

[0061]

[0062] Among them, (x, y) is the pixel point after dedistortion, (x distorted ,y distorted ) is the pixel before distortion, k 1 , k 2 , k 3 is the radial distortion parameter, which is the internal parameter of the camera and is generally used as k 1 and k 2 , fisheye cameras use k 3 , r is the distance from the pixel to the center of the image.

[0063] S3) restores depth information based on matched pixels and reconstructs a 3D point cloud model based on matched pixels using the PCL library, specifically including:

[0064] S31) When two cameras shoot the same part, there will be a slight difference in the position of the part in the two images. This difference in position between pixels is called parallax. Then, the parallax information obtained by matching is combined with the geometric parameters of the binocular cameras to calculate the depth value of each pair of pixels.

[0065] The formula for calculating the depth value of each pair of pixels is:

[0066]

[0067] Where Z represents the depth value of the pixel, f represents the focal length of the binocular camera, and T represents the distance between the left and right cameras in the binocular camera; x l Indicates the image taken by the left camera, the distance between the pixel point of the image and the center of the image; x r Indicates the distance between the pixel point of the picture taken by the right camera and the center of the picture. Figure 2 As shown in the figure, this method uses the triangle similarity principle and the parallax (x l -x r ) and the baseline (T), and the ratio of the focal length (f) and the depth value (Z) are equal. l , x r , T, f are known, solve the depth value Z.

[0068] S32) Based on the precisely matched pixel points, the PCL library is used to perform point cloud 3D modeling. The PCL library extracts and processes 3D point cloud data from the image of the binocular camera. It first removes noise and irrelevant points in the point cloud, retains important structural information, divides the point cloud data into different objects or parts, and classifies them. Then, the point cloud data from different perspectives are aligned and fused to generate a high-quality 3D model.

[0069] S4) performing hand-eye calibration and motion trajectory solving for the grasping robot, specifically including:

[0070] S41) Using Zhang Zhengyou's calibration method, the position transformation of the grasping robot's hand and eye is solved. The grasping robot has five coordinate systems, from the base coordinate system, the gripper coordinate system, the camera coordinate system, the image coordinate system, to the pixel coordinate system, a one-to-one mapping is achieved, and finally the transformation relationship between the pixel coordinate system and the base coordinate system is obtained, and the pixel points in the image are mapped to the real grasping scene through the transformation relationship;

[0071] S42) The motion trajectory of the grasping robot is solved by using a Cartesian space planning algorithm. A three-dimensional model is first generated based on the point cloud data to identify the position and posture of the part, determine the starting point of the gripper and the end point of the part, connect them into a straight line, and identify potential obstacles on the straight line based on the information of the point cloud modeling. Interpolation is performed between two points to plan a motion trajectory that bypasses these obstacles, and then the motion trajectory is smoothly optimized to ensure smoothness and accuracy of the gripper during movement.

[0072] S5) Using the obtained 3D point cloud model and the motion trajectory of the grasping robot, a planar grasping network is created. Here, a convolutional neural network is used to extract features from the input image. These features include important information such as the shape and edge of the object. Then, based on the extracted features, the possibility of each pixel in the image being a grasping point is evaluated. The result of the evaluation is usually a probability distribution, which represents the grasping possibility of each point. Then, the grasping points with high probability are further analyzed to predict the grasping posture of the manipulator at these points. According to the output of the network, the best grasping point and grasping posture are selected. This step is usually combined with some post-processing steps, such as non-maximum suppression to remove redundant grasping points. Finally, the selected grasping point and posture are passed to the robot control system to guide the manipulator to perform actual grasping operations. At the same time, sensor feedback can be used for real-time adjustments to ensure the success rate of grasping.

[0073] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be regarded as within the protection scope of the present invention.

Claims

1. A robot disordered grasping method based on binocular structured light point cloud reconstruction, characterized in that: The steps include: S1) Projecting structured light onto the surface of the part to be grasped; S2) The binocular camera captures the structured light on the surface of the part to be grasped, and stereo matching is performed on the structured light images captured by the left and right cameras of the binocular camera to obtain the pixel points of the matched structured light images; S3) restoring depth information according to the matched pixels, and reconstructing a 3D point cloud model based on the matched pixels using the PCL library; S4) performing hand-eye calibration and motion trajectory solving for the grasping robot; S5) Using the obtained three-dimensional point cloud model and the motion trajectory of the grasping robot, a planar grasping network is created to realize the robot's disordered grasping.

2. The robot disordered grasping method based on binocular structured light point cloud reconstruction as claimed in claim 1, characterized in that: In S1), structured light is projected by a projector, which is a cyclic complementary Gray code with long stripes of black and white and equal width, covering the part to be grasped.

3. The robot disordered grasping method based on binocular structured light point cloud reconstruction as claimed in claim 1, characterized in that: In S2), a stereo matching is performed on the structured light image captured by a binocular camera, including: S21) performing phase extraction on the structured light image taken by the binocular camera, and dedistorting the image to obtain a normalized image; S22) performing phase unwrapping according to the obtained normalized image to obtain a continuous phase value; S23) In the pictures of the left camera and the right camera, stereo matching estimation is performed using the phase value to obtain estimated matching points for pixel matching of the two pictures, and interpolation is performed within a set range of the estimated matching points to obtain accurate matching points; S24) According to the obtained normalized image and the interpolated precise matching points, the disparity between each corresponding pair of pixels in the images taken by the left and right cameras is determined for subsequent depth information calculation.

4. The robot disordered grasping method based on binocular structured light point cloud reconstruction as claimed in claim 3, characterized in that: In step S21), the formula for dedistortion is: Among them, (x, y) is the pixel point after dedistortion, (x distorted ,y distorted ) is the pixel before distortion, k1, k2, k3 are radial distortion parameters, and r is the distance from the pixel to the center of the image.

5. The robot disordered grasping method based on binocular structured light point cloud reconstruction as claimed in claim 1, characterized in that: S3) includes: S31) calculating the depth value of each pair of pixels based on the disparity information obtained by matching and combining the geometric parameters of the binocular camera; S32) Based on the precisely matched pixel points, point cloud 3D modeling is performed using the PCL library.

6. The robot disordered grasping method based on binocular structured light point cloud reconstruction as claimed in claim 5, characterized in that: In step S31), the formula for calculating the depth value of each pair of pixels is: Where Z represents the depth value of the pixel, f represents the focal length of the binocular camera, and T represents the distance between the left and right cameras in the binocular camera; x l Indicates the image taken by the left camera, the distance between the pixel point of the image and the center of the image; x r Indicates the picture taken by the right camera, the distance between the pixel point of the picture and the center of the picture.

7. The robot disordered grasping method based on binocular structured light point cloud reconstruction as claimed in claim 1, characterized in that: Step S4) comprises: S41) Using Zhang Zhengyou calibration method, solve the position transformation of the hand-eye of the grasping robot; S42) Using Cartesian space planning algorithm, the motion trajectory of the grasping robot is solved.

8. The robot disordered grasping method based on binocular structured light point cloud reconstruction as claimed in claim 1, characterized in that: In step S5), the plane capture network is a 2D plane capture network.