Stacked object disordered grasping method and system

Abstract

The application discloses a kind of stacked object disorderly grabbing methods, comprising the following steps: three-dimensional point cloud information acquisition: the two-dimensional laser scanning data of linear laser emitted on stacked object by laser radar is obtained, the two-dimensional laser scanning data is converted into corresponding three-dimensional point cloud data;Point cloud pose acquisition: the laser image collected by camera is processed, according to the three-dimensional point cloud data, PPF feature extraction is carried out, and corresponding pose information is obtained;Robot motion control: the joint state of robot is detected, the motion trajectory of robot is planned according to the obtained pose information, and the motion of robot is controlled according to the motion trajectory.The stacked object disorderly grabbing method of the application obtains the two-dimensional laser scanning data of linear laser on stacked object by laser radar and converts it into corresponding three-dimensional point cloud data to improve positioning accuracy, and is not easily affected, with low cost.The application also discloses a kind of stacked object disorderly grabbing system.

Description

Technical Field

[0001] This invention relates to the field of automatic object grasping technology, and in particular to a method and system for grasping stacked objects in an unordered manner. Background Technology

[0002] Most existing unordered stacked object grasping systems use 2D vision technology, which makes it difficult to obtain depth information of stacked objects and distinguish between stacked objects, resulting in low grasping accuracy. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method and system for grasping stacked objects in an unordered manner, so as to improve the precision and accuracy of grasping.

[0004] In a first aspect, the present invention provides a method for disordered grasping of stacked objects, comprising the following steps: 3D point cloud information acquisition: acquiring two-dimensional laser scanning data of line laser emitted by a lidar onto the stacked objects, and converting the two-dimensional laser scanning data into corresponding three-dimensional point cloud data; point cloud pose acquisition: processing the laser image acquired by the camera, performing PPF feature extraction based on the three-dimensional point cloud data, and acquiring corresponding pose information; robot motion control: detecting the joint state of the robot, planning the robot's motion trajectory based on the obtained pose information, and controlling the robot to move according to the motion trajectory.

[0005] Secondly, the present invention also provides a stacked object disorder grasping system, including a three-dimensional vision function module, a point cloud pose acquisition module, and a robot tracking and grasping module. The three-dimensional vision function module includes a camera and a lidar, used to acquire two-dimensional laser scanning data of line laser emitted by the lidar onto the stacked object, convert the two-dimensional laser scanning data into corresponding three-dimensional point cloud data, and acquire laser images on the stacked object captured by the camera. The point cloud pose acquisition module is used to process the laser images acquired by the camera, perform PPF feature extraction based on the three-dimensional point cloud data, and acquire corresponding pose information. The robot tracking and grasping module is used to detect the joint state of the robot, plan the robot's motion trajectory based on the obtained pose information, and control the robot to move according to the motion trajectory.

[0006] The beneficial technical effects of this invention are as follows: The stacked object disorder grasping method of this invention utilizes LiDAR to acquire two-dimensional laser scanning data of line lasers on the stacked objects and converts it into corresponding three-dimensional point cloud data, thereby improving positioning accuracy, being less susceptible to interference, and having low cost. It processes the laser images acquired by the camera and extracts PPF features from the three-dimensional point cloud data to obtain corresponding pose information. Based on the pose information, it plans the robot's motion trajectory and controls the robot to move according to the trajectory, achieving pose acquisition using three-dimensional point clouds, thus improving grasping accuracy. The stacked object disorder grasping system of this invention also has the above-mentioned functions. Attached Figure Description

[0007] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a flowchart illustrating the method for disordered grasping of stacked objects provided in an embodiment of the present invention.

[0009] Figure 2 A schematic diagram of a sub-process of the unordered grasping method for stacked objects provided in an embodiment of the present invention.

[0010] Figure 3 This is a schematic diagram of PPF feature extraction for the stacked object disorder grasping method provided in an embodiment of the present invention;

[0011] Figure 4 This is a schematic diagram of the framework of the stacked object disorder grasping system provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Please see Figure 1 As shown, Figure 1 This is a flowchart illustrating a method for unordered grasping of stacked objects provided in an embodiment of the present invention. The method includes the following steps:

[0014] Step S11, 3D point cloud information acquisition: Acquire two-dimensional laser scanning data of the line laser emitted by the lidar onto the stacked objects, and convert the two-dimensional laser scanning data into corresponding three-dimensional point cloud data; wherein, the two-dimensional laser scanning data includes the laser angle and distance measurement values;

[0015] Step S12, Point Cloud Pose Acquisition: Process the laser image acquired by the camera, extract PPF (Point Pair Feature) features based on the three-dimensional point cloud data, and obtain the corresponding pose information;

[0016] Step S13, Robot motion control: Detect the robot's joint state, plan the robot's motion trajectory based on the obtained pose information, and control the robot to move according to the motion trajectory.

[0017] The system utilizes a lidar to emit line lasers and receive laser data, while a camera captures images of the line lasers emitted onto the stacked objects. The disordered grasping method employs lidar to acquire two-dimensional laser scanning data of the line lasers on the stacked objects and converts it into corresponding three-dimensional point cloud data. This improves positioning accuracy, reduces susceptibility to interference, and is cost-effective. The laser images captured by the camera are processed, and PPF features are extracted from the three-dimensional point cloud data to obtain corresponding pose information. Based on this pose information, the robot's motion trajectory is planned, and the robot is controlled to move according to the trajectory. Using the three-dimensional point cloud for pose acquisition improves grasping accuracy.

[0018] Combination Figure 2 Step S11 specifically includes:

[0019] Step S111: Acquisition of two-dimensional laser scanning data: Two-dimensional laser scanning data of line lasers emitted on stacked objects are acquired using a lidar.

[0020] Step S112, Grayscale or Binary Image Acquisition: The obtained two-dimensional laser scanning data is processed using the OpenCV image library to convert it into a grayscale or binary image;

[0021] Step S113, 3D point cloud data acquisition: The obtained 2D laser scanning data is converted from 2D space to 3D space using triangulation. Combined with the PCL point cloud library, the corresponding 3D point cloud data is obtained. The pcl::triangulate function in the PCL point cloud library can be used to convert the 2D laser scanning data into the corresponding 3D point cloud data.

[0022] Specifically, step S113 may further include:

[0023] Stacked Object Recognition: The 3D point cloud data is processed and analyzed to identify stacked objects and the environment within it, obtaining a net stacked object 3D point cloud. This involves filtering, feature extraction, and segmentation of the 3D point cloud data to understand and recognize the stacked objects and the environment, resulting in a net stacked object 3D point cloud. The net stacked object 3D point cloud refers to the 3D point cloud of the stacked objects obtained by separating it from the 3D point cloud of the environment.

[0024] Specifically, the stacked object recognition step may further include:

[0025] The 3D point cloud of the net stacked objects is modeled to obtain a 3D model of the stacked objects. Visualization operations are then performed on the 3D model to display the stacked objects. Specifically, functions from the PCL point cloud library can be used for 3D modeling and visualization to better understand and display the information in the point cloud.

[0026] Specifically, the step S11 may further include:

[0027] The camera is calibrated to obtain its intrinsic and extrinsic parameters;

[0028] The control line lidar emits line lasers onto stacked objects.

[0029] Specifically, step S11 may further include:

[0030] A camera is controlled to acquire laser images of the stacked objects, and a point cloud of the stacked objects is obtained based on the acquired laser images. Specifically, a depth image of the stacked objects can be obtained from the acquired laser images. The depth image contains the height information of the stacked objects, and thus the point cloud of the stacked objects can be obtained from the depth image. The point cloud of the stacked objects and the corresponding 3D point cloud data obtained from the converted 2D laser scan data are the same point cloud data. The point cloud of the stacked objects is a scene point cloud; that is, the 3D point cloud data is a scene point cloud.

[0031] Specifically, the method preceding step S12 also includes:

[0032] Point cloud preprocessing: The obtained 3D point cloud data is downsampled, and the downsampled 3D point cloud data is preprocessed using the RANSAC algorithm to obtain a preprocessed 3D point cloud. By downsampling and processing the 3D point cloud data using the RANSAC algorithm, the number of point clouds is reduced while preserving basic point cloud information, thus improving work efficiency.

[0033] Point cloud segmentation: The preprocessed 3D point cloud is segmented into corresponding homogeneous regions to obtain scene point cloud and template point cloud. Homogeneity means having similar properties, so point clouds segmented into the same homogeneous region have similar properties. Template point cloud is the 3D point cloud of stacked objects.

[0034] Specifically, step S12 includes:

[0035] PPF Feature Extraction: The PPF algorithm is used to sample and obtain the corresponding PPF features between the scene point cloud and the template point cloud, and the obtained PPF features are written into a hash table. Specifically, the PPF features are written into the hash table with the feature as the key and the point pair as the value. The point cloud of the stacked objects is the scene point cloud, which is the 3D point cloud data. The template point cloud refers to the point cloud of a single object in the stacked objects. Specifically, combined with... Figure 3 The corresponding PPF features between the scene point cloud and the template point cloud can be calculated using formula (1):

[0036]

[0037] In the formula, F(m1,m2) represents the corresponding PPF features between the scene point cloud and the template point cloud. and These correspond to the normals of point m1 in the scene point cloud and point m2 in the template point cloud, respectively, where ||d||2 is the Euclidean distance between the pair of points. and These represent the angles between the corresponding vectors.

[0038] Sampling Matching: Scene sampling points are selected in the scene point cloud. These scene sampling points are then matched multiple times with different template matching points in the template point cloud to form multiple different sampling point pairs. Similar point pairs similar to the sampling point pairs are searched in a hash table. The rotation angle between each sampling point pair and its corresponding similar point pair is recorded. A voting table is constructed based on the similar sampling points and rotation angles of the similar point pairs. According to the Hough voting rule, the similar sampling point and rotation angle with the highest votes are selected. The normal vectors of the scene sampling points and similar sampling points are aligned with the x-axis of the world coordinate system, and the rotation angle is a rotation around the x-axis. A similar sampling point refers to the point corresponding to the scene sampling point in the similar point pair found in the hash table. The voting table in the Hough voting is a two-dimensional matrix used to record the voting status of each point in the point cloud corresponding to the Hough transformation space. In this embodiment, the voting table is actually used to statistically analyze the transformation matrix corresponding to each point in the point cloud.

[0039] Point-to-point distance calculation: The ICP algorithm is used to iterate the similar sampling points and rotation angles of the highest vote corresponding to the scene sampling points to calculate and obtain the pose of the stacked objects.

[0040] Preferably, the step of calculating the distance between the points is as follows:

[0041] Construct K-dimensional trees for the template point cloud and the scene point cloud respectively;

[0042] Traverse the K-tree of the template point cloud and the K-tree of the scene point cloud to find the nearest point pairs between each point in the template point cloud and the scene point cloud.

[0043] The pose of the stacked objects is calculated from the nearest point pairs between the template point cloud and the scene point cloud using the singular value decomposition algorithm.

[0044] Specifically, by constructing K-trees for template point clouds and scene point clouds respectively to form bidirectional K-trees, the corresponding nearest point pairs can be determined, significantly reducing iteration time and erroneous neighbor point searches, thereby improving the efficiency and accuracy of point cloud search.

[0045] Specifically, the step of traversing the K-tree of the template point cloud and the K-tree of the scene point cloud to obtain the nearest point pairs corresponding to each point in the template point cloud and the scene point cloud can be specifically as follows:

[0046] Closest point acquisition: Search for the scene point closest to the random template point in the K-tree of the scene point cloud, and search for the template point closest to the scene point closest to the random template point in the K-tree of the template point cloud; where the template point is a point in the template point cloud, the random template point is a random template point, and the scene point is a point in the scene point cloud.

[0047] Closest point determination: Determine whether the random template point is the closest template point to the scene point closest to it found in the K-tree of the template point cloud; if yes, record the random template point and the corresponding scene point; if no, execute the random template point update step; where, when the random template point is the closest template point to the scene point closest to it found in the K-tree of the template point cloud, it means that the random template point and the scene point closest to it found in the K-tree of the scene point cloud are the closest points with a corresponding relationship.

[0048] Random template point update: Determine if there are any unsearched random template points. If so, update the random template points and return to the step of obtaining the nearest point. If there are no unsearched random template points, end the operation.

[0049] Please see Figure 4 As shown, Figure 4This is a schematic diagram of the framework of a stacked object disorder grasping system 10 provided in an embodiment of the present invention. The stacked object disorder grasping system 10 includes a three-dimensional vision function module 11, a point cloud pose acquisition module 12, and a robot tracking and grasping module 13. The three-dimensional vision function module 11 includes a camera and a lidar, used to acquire two-dimensional laser scanning data of line laser emitted by the lidar on the stacked object, convert the two-dimensional laser scanning data into corresponding three-dimensional point cloud data, and acquire laser images on the stacked object captured by the camera. The point cloud pose acquisition module 12 is used to process the laser images acquired by the camera, perform PPF feature extraction based on the three-dimensional point cloud data, and acquire corresponding pose information. The robot tracking and grasping module 13 is used to detect the joint state of the robot, plan the robot's motion trajectory based on the obtained pose information, and control the robot to move according to the motion trajectory.

[0050] The system utilizes a lidar to emit and receive laser data. The stacked object disorder grasping system employs a 3D vision module to acquire 2D laser scanning data of the line lasers on the stacked objects using the lidar, converting it into corresponding 3D point cloud data. This improves positioning accuracy, reduces susceptibility to interference, and reduces cost. A point cloud pose acquisition module processes the laser images captured by the camera and extracts PPF features from the 3D point cloud data to obtain corresponding pose information. A robot tracking and grasping module then plans the robot's motion trajectory based on this pose information and controls the robot to move according to the trajectory. This achieves pose acquisition using 3D point clouds, thereby improving grasping accuracy.

[0051] In summary, the stacked object disorder grasping method of the present invention improves positioning accuracy by using LiDAR to acquire two-dimensional laser scanning data of line lasers on the stacked objects and converting it into corresponding three-dimensional point cloud data. This method is less susceptible to interference and has low cost. It processes the laser images acquired by the camera and extracts PPF features from the three-dimensional point cloud data to obtain corresponding pose information. Based on the pose information, it plans the robot's motion trajectory and controls the robot to move according to the trajectory, thus achieving pose acquisition using three-dimensional point clouds and improving grasping accuracy. The stacked object disorder grasping system of the present invention also has the above-mentioned functions.

[0052] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method of grasping a stack of objects in disorder, characterized in that, Includes the following steps: 3D point cloud information acquisition: acquire 2D laser scanning data of the line laser emitted by the lidar onto the stacked objects, and convert the 2D laser scanning data into corresponding 3D point cloud data; Point cloud pose acquisition: The laser image acquired by the camera is processed, and PPF features are extracted based on the three-dimensional point cloud data to obtain the corresponding pose information; Robot motion control: Detect the robot's joint states, plan the robot's motion trajectory based on the obtained pose information, and control the robot to move according to the motion trajectory; The step of acquiring the point cloud pose also includes: Point cloud preprocessing: The obtained 3D point cloud data is downsampled, and the downsampled 3D point cloud data is preprocessed using the RANSAC algorithm to obtain the preprocessed 3D point cloud. Point cloud segmentation: The preprocessed 3D point cloud is segmented into corresponding homogeneous regions to obtain scene point cloud and template point cloud. The steps for obtaining the point cloud pose include: PPF Feature Extraction: The PPF algorithm is used to sample and obtain the corresponding PPF features between the scene point cloud and the template point cloud, and the obtained PPF features are written into a hash table; Sampling and matching: Select scene sampling points in the scene point cloud, and match the scene sampling points with different template matching points in the template point cloud multiple times to form multiple different sampling point pairs. Search for similar point pairs that are similar to the sampling point pairs in the hash table, and record the rotation angle between each sampling point pair and the corresponding similar point pair found. Construct a voting table based on the similar sampling points and rotation angles of the similar point pairs, and select the similar sampling point and rotation angle with the highest vote according to the Hough voting rule. Point-to-point distance calculation: The ICP algorithm is used to iterate the similar sampling points and rotation angles of the highest vote corresponding to the scene sampling points to calculate and obtain the pose of the stacked objects.

2. The stacked object disorderly grasping method according to claim 1, characterized by, The steps for acquiring the 3D point cloud information specifically include: Two-dimensional laser scanning data acquisition: Two-dimensional laser scanning data of line lasers emitted on stacked objects are acquired using lidar; Grayscale or binary image acquisition: The obtained two-dimensional laser scanning data is processed using the OpenCV image library to convert it into a grayscale or binary image; 3D point cloud data acquisition: The obtained 2D laser scanning data is converted from 2D space to 3D space using triangulation. Combined with the PCL point cloud library, the corresponding 3D point cloud data is obtained.

3. The stacked object disorderly grasping method according to claim 2, characterized by, The step of acquiring the three-dimensional point cloud data also includes: Stacked object recognition: The three-dimensional point cloud data is processed and analyzed to identify stacked objects and the environment in the three-dimensional point cloud data, and to obtain a net stacked object three-dimensional point cloud.

4. The stacked object disorderly grasping method according to claim 3, characterized by, The stacked object recognition step is followed by: The three-dimensional point cloud of the net stacked object is modeled to obtain a three-dimensional model of the stacked object. The three-dimensional model of the stacked object is then visualized to display the three-dimensional model of the stacked object.

5. The stacked object disorderly grasping method according to claim 1, characterized by, The specific steps for calculating the distance between point pairs are as follows: Construct K-dimensional trees for the template point cloud and the scene point cloud respectively; Traverse the K-tree of the template point cloud and the K-tree of the scene point cloud to find the nearest point pairs between each point in the template point cloud and the scene point cloud. The pose of the stacked objects is calculated from the nearest point pairs between the template point cloud and the scene point cloud using the singular value decomposition algorithm.

6. The stacked object disorderly grasping method according to claim 5, characterized by, The specific steps for traversing the K-tree of the template point cloud and the K-tree of the scene point cloud to find the nearest point pairs between each point in the template point cloud and the scene point cloud are as follows: Closest point acquisition: Search for the scene point closest to the random template point in the K-tree of the scene point cloud, and search for the template point closest to the scene point closest to the random template point in the K-tree of the template point cloud; Closest point determination: Determine whether the random template point is the closest template point to the scene point found in the K-dimensional tree of the template point cloud; if yes, record the random template point and the corresponding scene point; if no, execute the random template point update step; Random template point update: Determine if there are any unsearched random template points. If so, update the random template points and return to the step of obtaining the nearest point. If there are no unsearched random template points, end the operation.

7. A stacked object disordered grasping system, comprising: include: The three-dimensional vision function module includes a camera and a lidar, used to acquire two-dimensional laser scanning data of line laser emitted by the lidar onto the stacked object, convert the two-dimensional laser scanning data into corresponding three-dimensional point cloud data, and acquire laser images of the stacked object captured by the camera. The point cloud pose acquisition module is used to process the laser image acquired by the camera, and to extract PPF features based on the 3D point cloud data to obtain the corresponding pose information. This includes: downsampling the obtained 3D point cloud data, and preprocessing the downsampled 3D point cloud data using the RANSAC algorithm to obtain a preprocessed 3D point cloud; segmenting the obtained preprocessed 3D point cloud into corresponding homogeneous regions to obtain scene point clouds and template point clouds; and using the PPF algorithm to sample and obtain the corresponding PPF features between the scene point cloud and the template point cloud, and then extracting the PPF features. PF features are written into a hash table; scene sampling points are selected in the scene point cloud, and different template matching points are matched multiple times for the scene sampling points in the template point cloud to form multiple different sampling point pairs. Similar point pairs similar to the sampling point pairs are searched in the hash table, and the rotation angle between each sampling point pair and the corresponding similar point pair is recorded. A voting table is constructed based on the similar sampling points and rotation angles of the similar point pairs. According to the Hough voting rule, the similar sampling point and rotation angle with the highest vote are selected; the ICP algorithm is used to iterate the similar sampling point and rotation angle with the highest vote corresponding to the scene sampling point to calculate and obtain the pose of the stacked objects. The robot tracking and grasping module is used to detect the robot's joint status, plan the robot's motion trajectory based on the obtained pose information, and control the robot to move according to the motion trajectory.

Images