A robot disorderly grasping method

Abstract

The present invention relates to the field of artificial intelligence and industrial application technology, and more particularly to a robotic random grasping method comprising the following steps: step S10, real-time three-dimensional point cloud data acquisition; step S20, point cloud preprocessing; step S30, point cloud ROI extraction; step S40, LCCP-based point cloud segmentation; step S50, point cloud registration priority decision-making; step S60, target pose solution based on point cloud registration; and step S70, robotic grasping. The present invention is ingeniously designed and, when in operation, can significantly improve the efficiency and accuracy of industrial grasping. It can efficiently and automatically identify and grasp industrial products, avoiding the tedious and time-consuming grasping required by traditional methods.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and industrial application technology, and in particular to a robot disorderly grasping method. Background Art

[0002] Robotic grasping in random order is a key task in industrial automation and industrial automation upgrades. In industrial scenes where objects are cluttered and occluded, using traditional 2D images for random grasping presents numerous challenges. For example, objects in industrial scenes generally lack color information and have weak textures. To overcome these challenges, a growing number of research techniques are trending towards using 3D point cloud information, which is unaffected by lighting conditions, to guide robotic grasping in random order.

[0003] Using 3D point clouds to guide robots in random grasping has the advantages of rich geometric information, being unaffected by lighting conditions, and being applicable to a variety of object shapes in the field of industrial automation. Although 3D point clouds have many advantages in random grasping by robots, existing random grasping methods based on 3D point clouds generally have some technical problems. First, low point cloud segmentation accuracy and efficiency are a common problem: due to the complexity of point cloud data and noise interference, existing segmentation algorithms often compromise between accuracy and speed, making it difficult to simultaneously achieve high-precision segmentation and real-time grasping operations. Secondly, pose estimation is one of the key issues in random grasping, but existing pose estimation methods often cannot meet the requirements of real-time grasping. Due to the complexity of object shapes and the incompleteness of point cloud data, existing pose estimation methods often have problems such as inaccurate positioning and incorrect pose estimation, which lead to errors and failures in the robot grasping process. Summary of the Invention

[0004] The present invention addresses the problems of the prior art and provides a robot disordered grasping method with an ingenious design. When working, the present invention can greatly improve the efficiency and accuracy of industrial grasping; it can efficiently and automatically identify and grasp industrial products, avoiding the tedious and time-consuming grasping methods of traditional methods.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] The present invention provides a robot disordered grasping method, which comprises the following steps:

[0007] Step S10: real-time three-dimensional point cloud data acquisition;

[0008] Step S20: point cloud preprocessing;

[0009] Step S30, point cloud ROI extraction;

[0010] Step S40: point cloud segmentation based on LCCP;

[0011] Step S50, point cloud registration priority decision;

[0012] Step S60: solving the target pose based on point cloud registration;

[0013] Step S70: Robot grasping.

[0014] Among them, in step S10, real-time three-dimensional point cloud data acquisition is performed by using sensors to obtain three-dimensional point cloud data of objects in the scene. The three-dimensional point cloud data is obtained by scanning and measuring the scene by the sensor, wherein each point contains the three-dimensional spatial information and color information of the object. By obtaining a set of distance information of the object points, the three-dimensional point cloud data is formed.

[0015] Wherein, in the step S20, the point cloud preprocessing includes a straight-through filtering method, a statistical filtering method and an octree downsampling method.

[0016] The point cloud ROI extraction method in step S30 includes step S31, converting screen coordinates to world coordinates; converting screen coordinates to world coordinates refers to the process of converting two-dimensional coordinate points on the screen into three-dimensional world coordinate points; the conversion method includes:

[0017] The first step is to convert the x and y values ​​of the screen coordinates to normalized device coordinates; use the following formula to convert screen coordinates to NDC coordinates:

[0018]

[0019] Where screen_x is the x-axis coordinate of the screen, screen_y is the y-axis coordinate of the screen, NDC_x is the normalized x-axis coordinate, and NDC_y is the normalized y-axis coordinate;

[0020] The second step is to convert the NDC coordinates into camera coordinates through inverse projection transformation; for perspective projection, use perspective division to achieve inverse projection:

[0021]

[0022] Among them, left, right, bottom, top are the boundary values ​​of the camera's viewing volume, and near is the distance from the camera to the near clipping plane;

[0023] The third step is to convert the camera coordinates into world coordinates; multiply the camera coordinates by the inverse matrix of the view matrix to get the world coordinates:

[0024] world_position=inverse(view_matrix)*(camera_x,camera_y,camera_z,1); where view_matrix is ​​the camera’s view matrix and inverse(view_matrix) is the inverse of the view matrix.

[0025] The method for extracting the point cloud ROI in step S30 further includes step S32, cropping the area according to the selected point; using the mouse event box to select the polygonal ROI area and store it, traversing each point of the real-time point cloud and judging whether it is within the ROI polygon area, deleting it if it is not, and retaining it if it is not; the specific judgment method is: assuming that the coordinates of the point are (x, y), and the vertex coordinates of the polygon are (x1, y1)(x2, y2)…(x n ,y n ), each side of the polygon is represented by the coordinates of two vertices (x i ,y i ) and (x i+1 ,y i+1 ), determine the intersection of the ray and the edge, use the following formula: If (y i >y)≠(y i+1 >y), and x<(x i+1 -x i )*(yy i ) / (y i+1 -y i )+x i , then the ray intersects the edge, and the number of times the ray intersects the edge of the polygon is counted. If it is an odd number, the point is inside the polygon; if it is an even number, the point is outside the polygon.

[0026] Among them, the LCCP-based point cloud segmentation method in step S40 is: by calculating the local convexity and concavity of each point, the points in the point cloud can be divided into convex points and concave points; then, by merging points with similar convexity and concavity, the point cloud is divided into different parts to obtain multiple point cloud sets.

[0027] The decision weight of the point cloud registration priority decision in step S50 is calculated as follows:

[0028]

[0029] Among them, w v is the volume score weight, w h is the height score weight, V i is the volume of different point cloud subsets, V template is the volume of the point cloud template, H i is the average height of different point cloud subsets, Haverage is the average height of the entire point cloud.

[0030] The method for solving the target pose based on point cloud registration in step S60 includes: step S61, coarse point cloud registration based on SAC-IA: the SAC-IA algorithm estimates the initial transformation matrix between two point clouds by sampling a set of point pairs;

[0031] Step S62, based on ICP point cloud coarse registration: gradually optimize the correspondence between the two point clouds to obtain the optimal transformation matrix.

[0032] Beneficial effects of the present invention:

[0033] The present invention is cleverly designed and can greatly improve the efficiency and accuracy of industrial grasping when in operation; it can efficiently and automatically identify and grasp industrial products, avoiding the tedious and time-consuming grasping methods of traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The figure is a flow chart of a robot disorderly grasping method of the present invention. DETAILED DESCRIPTION

[0035] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the embodiments and the accompanying drawings. The contents mentioned in the embodiments are not intended to limit the present invention. The present invention will be described in detail below with reference to the accompanying drawings.

[0036] The present application provides a method for robotic random grasping, comprising the following steps: step S10, real-time three-dimensional point cloud data acquisition; step S20, point cloud preprocessing; step S30, point cloud ROI extraction; step S40, LCCP-based point cloud segmentation; step S50, point cloud registration priority decision-making; step S60, target pose solution based on point cloud registration; and step S70, robotic grasping. Specifically, the present application embodiment is cleverly designed. When in operation, the present application embodiment can greatly improve the efficiency and accuracy of industrial grasping; it can efficiently and automatically identify and grasp industrial products, avoiding the tedious and time-consuming grasping methods of traditional methods.

[0037] In an embodiment of the present application, in step S10, real-time three-dimensional point cloud data acquisition is performed by using a depth sensor (such as a lidar or an RGB-D camera) to obtain three-dimensional point cloud data of objects in the scene. The three-dimensional point cloud data is obtained by scanning and measuring the scene by the sensor, wherein each point contains the three-dimensional spatial information (XYZ coordinates) and color (RGB) information of the object. By obtaining a set of distance information of the object points, three-dimensional point cloud data is formed; by processing and analyzing the point cloud data, the features of the object, such as surface normals, key points or descriptors, etc., can be extracted, and these features can be used to solve the position and posture of the target in the camera coordinate system.

[0038] In the embodiment of the present application, in step S20, the point cloud preprocessing includes a straight-through filtering method, a statistical filtering method, and an octree downsampling method; the straight-through filtering method is used in the embodiment of the present application to remove outliers or invalid data in the point cloud data, and to extract point cloud data within the region of interest; by setting thresholds for the directions of the three coordinate axes, such as the x, y, and z coordinates, point cloud data within the threshold range of the three coordinate axes is retained, for example, redundant point cloud data such as the edge of the material box, the ground, and the placement table are removed. Due to the acquisition equipment and environment, point cloud data may contain noise and outliers, which will have a negative impact on the alignment results; the statistical filtering method can effectively remove noise and outliers in point cloud data. It should be noted that in actual applications, the statistical filtering parameters should be set according to the measurement unit and installation conditions of the actual acquisition equipment selected. In the octree downsampling method, even after preprocessing the original point cloud dataset using methods like straight-through filtering and statistical filtering, a large number of redundant points may still remain. These redundant points not only reduce the efficiency of pose estimation but may also negatively impact pose accuracy. Therefore, further processing of the point cloud data is necessary to reduce the data volume while preserving key features. Therefore, octree downsampling is used to preprocess the point cloud. Octree downsampling effectively removes redundant points from the point cloud data, significantly reducing the number of points. This reduces the number of points while preserving the valid information required for pose estimation, improving both the efficiency and accuracy of pose estimation.

[0039] In the embodiment of the present application, the point cloud ROI extraction method in step S30 includes step S31, converting screen coordinates to world coordinates; converting screen coordinates to world coordinates refers to the process of converting two-dimensional coordinate points on the screen into three-dimensional world coordinate points; the conversion method includes:

[0040] The first step is to convert the x and y values ​​of the screen coordinates to normalized device coordinates; use the following formula to convert screen coordinates to NDC coordinates:

[0041]

[0042] Where screen_x is the x-axis coordinate of the screen, screen_y is the y-axis coordinate of the screen, NDC_x is the normalized x-axis coordinate, and NDC_y is the normalized y-axis coordinate;

[0043] The second step is to convert the NDC coordinates into camera coordinates through inverse projection transformation; for perspective projection, use perspective division to achieve inverse projection:

[0044]

[0045] Among them, left, right, bottom, top are the boundary values ​​of the camera's viewing volume, and near is the distance from the camera to the near clipping plane;

[0046] The third step is to convert the camera coordinates into world coordinates; multiply the camera coordinates by the inverse matrix of the view matrix to get the world coordinates:

[0047] world_position=inverse(view_matrix)*(camera_x,camera_y,camera_z,1); where view_matrix is ​​the camera’s view matrix and inverse(view_matrix) is the inverse of the view matrix.

[0048] In the embodiment of the present application, the method for extracting the point cloud ROI in step S30 further includes step S32, cropping the area according to the selected point; using the mouse event box to select the polygonal ROI area and store it, traversing each point of the real-time point cloud and judging whether it is within the ROI polygon area, deleting it if it is not, and retaining it if it is not; the specific judgment method is: assuming that the coordinates of the point are (x, y), and the vertex coordinates of the polygon are (x1, y1)(x2, y2)…(x n ,y n ), each side of the polygon is represented by the coordinates of two vertices (x i ,y i ) and (x i+1 ,y i+1 ), determine the intersection of the ray and the edge, use the following formula: If (y i >y)≠(y i+1 >y), and x<(x i+1 -x i )*(yy i ) / (y i+1 -y i )+x i , then the ray intersects the edge, and the number of times the ray intersects the edge of the polygon is counted. If it is an odd number, the point is inside the polygon; if it is an even number, the point is outside the polygon.

[0049] In the embodiment of the present application, the LCCP-based point cloud segmentation method in step S40 is as follows: by calculating the local convexity and concavity of each point, the points in the point cloud can be divided into convex points and concave points; then, by merging points with similar convexity and concavity, the point cloud can be divided into different parts to obtain multiple point cloud sets. The LCCP (Local Convexity Concavity Preserving) algorithm is an algorithm for point cloud segmentation, which can divide the points in the point cloud into different parts, each part representing an object or a part of an object; the principle of the LCCP algorithm is to perform segmentation based on the convexity and concavity properties in the point cloud; a convex point means that all adjacent points are on the same side of its local area; a concave point means that there are adjacent points on both sides of its local area; by calculating the local convexity and concave of each point, the points in the point cloud can be divided into convex points and concave points; then, by merging points with similar convexity and concavity, the point cloud can be segmented into different parts. This embodiment utilizes the characteristics of the LCCP algorithm and applies it to the segmentation of disordered and complex point clouds in industrial scenarios.

[0050] In the embodiment of the present application, multiple point cloud sets can be obtained through step S40. If the polling matching criteria are directly performed, there will be problems of high time consumption and low matching efficiency. Therefore, the present invention proposes a point cloud matching priority decision method to prioritize the sets that originally had no priority. The specific principle is: first, a certain priority is assigned according to the volume of the point cloud template and the volume of the point cloud template; secondly, a certain priority is assigned according to the ratio of the z-axis average value of the point cloud subset to the z-axis average value of the entire point cloud. Finally, the matching priority is decided by calculating the average height and point cloud volume in each point cloud set. The higher the calculated value, the higher the priority. The decision weight calculation method for the point cloud matching priority decision in step S50 is as follows:

[0051]

[0052] Among them, w v is the volume score weight, w h is the height score weight, V i is the volume of different point cloud subsets, V template is the volume of the point cloud template, H i is the average height of different point cloud subsets, H average is the average height of the entire point cloud.

[0053] In the embodiment of the present application, step S60, the method for solving the target pose based on point cloud registration includes:

[0054] Step S61: SAC-IA point cloud coarse registration: The SAC-IA algorithm estimates the initial transformation matrix between the two point clouds by sampling a set of point pairs. This sampling consistency algorithm can effectively handle noise and mismatching issues, thereby obtaining a relatively accurate initial transformation matrix. However, due to the limited number of sampled point pairs, the initial transformation matrix may contain certain errors. Step S62: ICP point cloud coarse registration: The correspondence between the two point clouds is gradually optimized to obtain the optimal transformation matrix. To further improve the accuracy of registration, the ICP algorithm is introduced for fine registration. The ICP algorithm gradually optimizes the correspondence between the two point clouds through iterative optimization to obtain the optimal transformation matrix. Specifically, the ICP algorithm first applies the initial transformation matrix to the moving point cloud to be registered, then establishes point pairs by finding the nearest neighbors of each point in the moving point cloud. Next, the optimal transformation matrix is ​​calculated by minimizing the distance between the point pairs and applied to the moving point cloud, iterating continuously until convergence conditions are met.

[0055] The target pose solution based on point cloud registration can be divided into two steps: SAC-IA (Sample Consensus Initial Alignment) coarse registration and ICP (Iterative Closest Point) fine registration to improve the accuracy and robustness of the registration.

[0056] In an embodiment of the present application, in step S70, after completing the point cloud registration, we can obtain the pose of the target in the camera coordinate system; however, the pose of the robot's end effector is relative to the robot's base coordinate system; therefore, before grasping the target, we need to convert the registered pose to the robot's base coordinate system through the hand-eye relationship matrix; finally, the robot completes the grasping task based on this pose.

[0057] In this invention, the conversion relationship between the screen coordinate system and the world coordinate system is utilized to quickly and accurately extract the target ROI area through manual polygon pre-selection; the convexity and concavity segmentation of LCCP is used to segment disordered point clouds in industrial scenes due to its efficient and accurate characteristics; a new point cloud registration priority decision method is proposed, which can greatly reduce the time of point cloud registration, thereby improving the rhythm of the entire method and better meeting the needs of actual production.

[0058] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention is disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes by using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments according to the technology of the present invention are all within the scope of the technical solution of the present invention without departing from the content of the technical solution of the present invention.

Claims

1. A robot disordered grasping method, characterized in that: The following steps are involved: Step S10: real-time three-dimensional point cloud data acquisition; Step S20: point cloud preprocessing; Step S30, point cloud ROI extraction; Step S40: point cloud segmentation based on LCCP; Step S50, point cloud registration priority decision; Step S60: solving the target pose based on point cloud registration; Step S70: robot grasping; The method for extracting the point cloud ROI in step S30 includes step S31, converting screen coordinates to world coordinates; converting screen coordinates to world coordinates refers to the process of converting two-dimensional coordinate points on the screen into three-dimensional world coordinate points; the conversion method includes: The first step is to convert the x and y values ​​of the screen coordinates to normalized device coordinates; use the following formula to convert screen coordinates to NDC coordinates: Where screen_x is the x-axis coordinate of the screen, screen_y is the y-axis coordinate of the screen, NDC_x is the normalized x-axis coordinate, and NDC_y is the normalized y-axis coordinate; The second step is to convert the NDC coordinates into camera coordinates through inverse projection transformation; for perspective projection, use perspective division to achieve inverse projection: Among them, left, right, bottom, top are the boundary values ​​of the camera's viewing volume, and near is the distance from the camera to the near clipping plane; The third step is to convert the camera coordinates into world coordinates; multiply the camera coordinates by the inverse matrix of the view matrix to get the world coordinates: world_position = inverse(view_matrix)*(camera_x,camera_y,camera_z,1); where view_matrix is ​​the camera’s view matrix and inverse(view_matrix) is the inverse of the view matrix. The method for extracting the point cloud ROI in step S30 further includes step S32, cropping the area according to the selected point; using the mouse event box to select the polygonal ROI area and store it, traversing each point of the real-time point cloud and judging whether it is within the ROI polygon area, deleting it if it is not, and retaining it if it is not; the specific judgment method is: assuming that the coordinates of the point are (x, y), the coordinates of the vertices of the polygon are (x1, y1)(x2, y2)…(x n ,y n ), each side of the polygon is represented by the coordinates of two vertices (x i ,y i ) and (x i+1 ,y i+1 ), determine the intersection of the ray and the edge, use the following formula: If (y i >y)≠(y i+1 >y), and x<(x i+1 -x i )*(yy i ) / (y i+1 -y i )+x i , then the ray intersects the edge, and the number of times the ray intersects the edge of the polygon is counted. If it is an odd number, the point is inside the polygon; if it is an even number, the point is outside the polygon; The decision weight for the point cloud registration priority decision in step S50 is calculated as follows: Among them, w v is the volume score weight, w h is the height score weight, V i is the volume of different point cloud subsets, V template is the volume of the point cloud template, H i is the average height of different point cloud subsets, H average is the average height of the entire point cloud.

2. A robot disorderly grasping method according to claim 1, characterized in that: In step S10, real-time three-dimensional point cloud data acquisition is performed by using sensors to obtain three-dimensional point cloud data of objects in the scene. The three-dimensional point cloud data is obtained by scanning and measuring the scene by the sensor, wherein each point contains the three-dimensional spatial information and color information of the object. The three-dimensional point cloud data is formed by obtaining a set of distance information of the object points.

3. A robot disorderly grasping method according to claim 1, characterized in that: In step S20, the point cloud preprocessing includes a straight-through filtering method, a statistical filtering method, and an octree downsampling method.

4. A robot disorderly grasping method according to claim 1, characterized in that: The LCCP-based point cloud segmentation method in step S40 is as follows: by calculating the local convexity and concavity of each point, the points in the point cloud can be divided into convex points and concave points; then, by merging points with similar convexity and concavity, the point cloud is segmented into different parts to obtain multiple point cloud sets.

5. A robot disorderly grasping method according to claim 1, characterized in that: The method for solving the target pose based on point cloud registration in step S60 includes: Step S61, coarse registration of point clouds based on SAC-IA: the SAC-IA algorithm estimates the initial transformation matrix between two point clouds by sampling a set of point pairs; Step S62, based on ICP point cloud coarse registration: gradually optimize the correspondence between the two point clouds to obtain the optimal transformation matrix.

Images