Target pose recognition method for robot grabbing of parts on conveyor belt, processing device, equipment and medium

Abstract

The invention discloses a target pose recognition method for robot grabbing of parts on a conveyor belt, and the method comprises the following steps: obtaining camera stream pictures shot at two sides of the conveyor belt, and obtaining RGB visible light images and D depth image data; reconstruction is carried out according to the depth image, and three-dimensional point cloud reconstruction is carried out on the depth image data to generate point cloud data; segmenting the point cloud data of the maximum plane part in the point cloud data by using an algorithm, calculating the average depth of the point cloud data, and filtering out contour data obtained from the obtained point cloud data of the irregular part on the plane; and according to contour data obtained from the point cloud data, an SAM model is used for conducting segmentation on the RGB visible light image to obtain irregular part mask data, the target pose of the irregular part is obtained through calculation, and a mechanical arm is used for grabbing an object. According to the invention, highly accurate spatial information is provided; the irregular parts to be grabbed are accurately identified and segmented under a complex background; and irregular parts with different shapes and sizes can be flexibly adapted.

Description

Technical Field

[0001] The present invention relates to the technical field of posture recognition, and in particular to a target posture recognition method, processing device, equipment and medium for robot grasping of parts on a conveyor belt. Background Art

[0002] With the popularization of industrial automation, robots are increasingly used in production lines. Automated production lines usually include conveyor belts that continuously transport irregular parts that need to be processed or assembled. In order for robots to accurately grasp and operate these irregular parts, accurate recognition of the target pose is particularly important.

[0003] However, since the irregular parts on the conveyor belt often have different shapes, sizes, and materials, and may rotate, flip, or be randomly placed during the conveyance process, it is impossible for the robot to accurately grasp each irregular part based solely on the preset posture information. Computer vision, deep learning, or other sensing technologies are needed to identify the real-time posture of irregular parts.

[0004] In view of this, it is necessary to design a target pose recognition method for robot grasping of parts on a conveyor belt to solve the above problems. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0006] To this end, the present invention combines point cloud reconstruction with the SAM model, and RGB visible light images with depth images. This method addresses the problem that parts on the conveyor belt have different shapes, sizes, and types, and may rotate, flip, or be randomly placed during transportation. The present invention does not rely on pre-set templates and is more adaptable to changes in the posture and shape of irregular parts.

[0007] According to a first aspect of the present invention, a target posture recognition method for robot grasping of parts on a conveyor belt comprises the following steps:

[0008] S1, obtain the camera stream images taken by the depth cameras at the ends of the robotic arms on both sides of the conveyor belt to obtain RGB visible light images and D depth image data;

[0009] S2, reconstructing the depth image according to the depth image, reconstructing the three-dimensional point cloud of the captured depth image data to generate point cloud data;

[0010] S3. Use an algorithm to segment the point cloud data of the largest plane part in the point cloud data, estimate the normal vector of the segmented point cloud data, adjust the normal vector to point to the same direction of the z-axis, calculate the average normal vector, calculate the rotation matrix, rotate the entire point cloud to align it with the z-axis, calculate the average depth of the point cloud data, and filter out the point cloud of irregular parts on the plane to obtain the contour;

[0011] S4. Based on the contour data obtained from the point cloud, the SAM model is used to segment the RGB visible light image to obtain the mask data of the irregular parts, the target pose of the irregular parts is calculated, and the robotic arm is used to grasp the object.

[0012] Preferably, in S3, after using an algorithm to segment the point cloud data of the largest planar part in the point cloud data, and estimating the normal vector of the segmented point cloud data, the normal vector is adjusted to point in the same direction of the z-axis, the average normal vector is calculated, the rotation matrix is ​​calculated, and the entire point cloud is rotated so that the average normal vector is aligned with the z-axis.

[0013] Further preferably, the RANSAC algorithm is used to segment the largest plane part data in the point cloud in S3.

[0014] Preferably, the KD tree of the K-nearest neighbor algorithm is used to estimate the normal vector in S3, specifically, the domain data of each point within the set radius search range that does not exceed the specified maximum number of neighbors is obtained, and the normal vector is calculated based on the data, and the formula is:

[0015]

[0016] C=VΛV T

[0017] Among them, {p i} is point cloud data, where each point is a three-dimensional coordinate p i =(x i ,y i ,z i ), for a target point p0, its neighborhood point set is {p1,p2,…,p k}, where k is the number of neighborhood points, c is the mean of the neighborhood points, C is the covariance matrix, V is the eigenvector matrix, Λ is the eigenvalue diagonal matrix, and the normal vector is the eigenvector corresponding to the smallest eigenvalue.

[0018] Further preferably, the formula for calculating the average normal vector in S3 is specifically:

[0019]

[0020] Among them, the normal vector of each point is n i =(n x,i ,n y,i , nz,i ), where i is the index of the point, n x,i , n y,i , n z,i are the x, y, and z components of the normal vector, N is the number of points in the point cloud, and n avg is the calculated average normal vector.

[0021] Preferably, the formula for calculating the rotation matrix in S3 is specifically:

[0022] v=m×t

[0023]

[0024] c=m·t

[0025]

[0026] Where m is the mean normal vector, t = [0,0,1] is the target vector, and v is the cross product of the two vectors. x , v y , v z are the x, y, and z components of vector v, s is the modulus of vector v, c is the dot product of the two vectors, skew_matrix represents the antisymmetric matrix of the rotation axis, skew_matrix 2 is the square of the antisymmetric matrix, I is the 3x3 identity matrix, and R is a calculated 3x3 rotation matrix.

[0027] Further preferably, in S4, the minimum circumscribed rectangle of the acquired contour data is calculated, and the object mask data is obtained by segmenting the RGB visible light image using the SAM model through a box prompt method.

[0028] The second aspect of the present invention provides an image processing device, which includes a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the target posture recognition method for robot grasping of parts on a conveyor belt described in the first aspect of the present invention.

[0029] A third aspect of the present invention provides an edge device, the edge device comprising the image processing device according to the second aspect of the present invention, and depth cameras respectively connected to the image processing device, the depth cameras being RealSence D435i depth cameras;

[0030] The depth camera is suitable for capturing visible light images and depth images, and the image processing device is used to reconstruct the captured object based on the depth image.

[0031] The fourth aspect of the present invention provides a computer-readable storage medium having program instructions that can be executed by a processor, and the program instructions are used to implement the target posture recognition method for robot grasping of parts on a conveyor belt described in the first aspect of the present invention.

[0032] The beneficial effects of the present invention are: through automatic segmentation, posture adjustment and point cloud analysis, the need for manual intervention is reduced, the degree of automation of the entire production process is improved, and it is suitable for efficient and large-scale production environments;

[0033] The RANSAC algorithm can effectively deal with noise and incomplete data. When processing irregular parts on a conveyor belt, even if the data contains more noise or there is occlusion, RANSAC can still robustly identify the largest plane part, thereby improving the reliability and robustness of target recognition. By calculating the normal vector and the rotation matrix aligned with the z-axis, this method can accurately adjust the three-dimensional posture information of the object, so that the robot can adapt to the arbitrary posture of irregular parts during the grasping process without the need for additional posture correction steps, thereby improving the efficiency and accuracy of grasping;

[0034] By using a depth camera to obtain RGB images and depth images and reconstructing 3D point clouds, highly accurate spatial information can be provided. By utilizing the powerful capabilities of the SAM model in image segmentation, the system can accurately identify and segment irregular parts to be grasped in complex backgrounds. The system can flexibly handle irregular parts of different shapes and sizes, and has good versatility and adaptability.

[0035] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0038] Figure 1 It is a flowchart of the steps of the target posture recognition method for robot grasping of parts on a conveyor belt of the present invention;

[0039] Figure 2 Reconstructing a point cloud data graph from a depth image captured by a camera of the target posture recognition method for robot grasping of parts on a conveyor belt of the present invention;

[0040] Figure 3 A point cloud data graph of a plane segmented by a point cloud data algorithm of a target posture recognition method for robot grasping of parts on a conveyor belt according to the present invention;

[0041] Figure 4 The point cloud data graph of the target irregular parts on the plane is obtained by filtering out the target posture recognition method for robot grasping of parts on a conveyor belt of the present invention;

[0042] Figure 5a-5b The target posture recognition method for robot grasping of parts on a conveyor belt of the present invention filters out the minimum circumscribed rectangle of the point cloud contour of the target irregular part on the plane and segmentation to obtain the mask data of the irregular part. DETAILED DESCRIPTION

[0043] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention. In the description of the present invention, it should be understood that, unless otherwise specified, the meaning of "multiple" is two or more.

[0044] See also Figure 1 , a target posture recognition method for robot grasping of parts on a conveyor belt according to a specific embodiment of the present invention comprises the following steps:

[0045] S1, obtain the camera stream images taken on both sides of the conveyor belt, and obtain RGB visible light image and D depth image data;

[0046] S2, reconstructing the depth image according to the depth image, reconstructing the three-dimensional point cloud of the captured depth image data to generate point cloud data;

[0047] S3, using an algorithm to segment the point cloud data of the largest plane part in the point cloud data, calculating the average depth of the point cloud data, and filtering out the contour data obtained from the point cloud data of the irregular parts on the plane;

[0048] S4. Based on the contour data obtained from the point cloud data, the SAM model is used to segment the RGB visible light image to obtain the mask data of the irregular parts, and the target pose of the irregular parts is calculated so that the robot arm can grasp the object.

[0049] Therefore, by using a depth camera to obtain RGB images and depth images and reconstructing a three-dimensional point cloud, highly accurate spatial information can be provided. At the same time, by using the powerful ability of the SAM model in image segmentation, the system can accurately identify and segment irregular parts to be grasped in complex backgrounds.

[0050] Moreover, the recognition method adopted by the system can flexibly adapt to irregular parts of different shapes and sizes, and has good versatility and adaptability. Therefore, the present invention reduces the need for manual intervention through automatic segmentation, posture adjustment and point cloud analysis, improves the automation level of the entire production process, and is suitable for efficient and large-scale production environments.

[0051] See also Figure 3 , the data of the largest plane part in the point cloud segmented in S3 is represented by the RANSAC algorithm. The point cloud data segmented by the RANSAC algorithm is fitted by randomly selecting 3 points in each iteration, and the distance threshold is set to 0.008 meters. If the distance from a point to the estimated plane is less than this threshold, it is considered that the point belongs to the plane. This point will be considered as an inner point of the plane. After 2000 iterations, a plane model with the most inner points is obtained, which is the point cloud data of the largest plane part. The RANSAC algorithm can effectively deal with noise and incomplete data. When processing irregular parts on the conveyor belt, even if the data contains more noise or there is occlusion, RANSAC can still robustly identify the largest plane part, thereby improving the reliability and robustness of target recognition.

[0052] Specifically, S3 implements the KD tree of the K-nearest algorithm to estimate the normal vector, specifically obtaining the domain data of each point within the set radius search range that does not exceed the specified maximum number of neighbors, where each point within the set radius of 0.05 meters does not exceed the specified maximum number of neighbors of 350 domain data, and calculates the normal vector based on the data, the formula is:

[0053]

[0054] C=VΛV T

[0055] Among them, {p i} is point cloud data, where each point is a three-dimensional coordinate p i =(x i ,y i ,z i ), for a target point p0, its neighborhood point set is {p1,p2,…,p k}, where k is the number of neighborhood points, c is the mean of the neighborhood points, C is the covariance matrix, V is the eigenvector matrix, Λ is the eigenvalue diagonal matrix, and the normal vector is the eigenvector corresponding to the smallest eigenvalue.

[0056] More specifically, in S3, after using the algorithm to segment the point cloud data of the largest plane part in the point cloud data, the normal vector is estimated for the segmented point cloud data, the normal vector is adjusted to point in the same direction of the z-axis, the average normal vector is calculated, the rotation matrix is ​​calculated, and the entire point cloud is rotated so that the average normal vector is aligned with the z-axis.

[0057] Among them, the formula for calculating the average normal vector is specifically:

[0058]

[0059] Among them, the normal vector of each point is n i =(n x,i ,n y,i , n z,i ), where i is the index of the point, n x,i , n y,i , n z,i are the x, y, and z components of the normal vector, N is the number of points in the point cloud, and n avg is the calculated average normal vector.

[0060] Among them, the formula for calculating the rotation matrix is ​​specifically:

[0061] v=m×t

[0062]

[0063] c=m·t

[0064]

[0065] Where m is the mean normal vector, t = [0,0,1] is the target vector, and v is the cross product of the two vectors. x , v y , v z are the x, y, and z components of vector v, s is the modulus of vector v, c is the dot product of the two vectors, skew_matrix represents the antisymmetric matrix of the rotation axis, skew_matrix 2 is the square of the antisymmetric matrix, I is the 3x3 identity matrix, and R is a calculated 3x3 rotation matrix.

[0066] Rotate the entire point cloud to align it with the z-axis, calculate the average depth of the point cloud data, and filter out the point cloud contours of irregular parts on the plane. For details, see Figure 4 , where the axes created are the x-axis, y-axis, and z-axis of the coordinate system. Therefore, by calculating the normal vector and aligning the rotation matrix with the z-axis, this method can accurately adjust the three-dimensional posture information of the object, so that the robot can adapt to any posture of irregular parts during the grasping process without the need for additional posture correction steps, thereby improving the efficiency and accuracy of grasping.

[0067] See also Figure 5a-5b In S4, the minimum enclosing rectangle of the acquired contour data is calculated, and the practical SAM model is used through the box prompt method and the FastSAM-s.pt weights to segment the RGB visible light image to obtain the irregular part mask data. When there are multiple point cloud contours, the minimum enclosing rectangle information of the point cloud contours of the irregular parts is input in turn, and the irregular part mask data is segmented and converted into three-dimensional coordinates through the camera coordinate system to obtain the target posture of the irregular part, and the robot arm is used to grasp the object.

[0068] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0069] The above description is based on the ideal embodiment of the present invention. Through the above description, relevant personnel can make various changes and modifications without departing from the technical concept of the present invention. The technical scope of the present invention is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A target posture recognition method for robot grasping of parts on a conveyor belt, characterized in that: The following steps are involved: S1, obtain the camera stream images taken on both sides of the conveyor belt, and obtain RGB visible light image and D depth image data; S2, reconstructing the depth image data into a three-dimensional point cloud to generate point cloud data; S3, using an algorithm to segment the point cloud data of the largest plane part in the point cloud data, calculating the average depth of the point cloud data, and filtering out the contour data obtained from the point cloud data of the irregular parts on the obtained plane; S4. Based on the contour data obtained from the point cloud data, the SAM model is used to segment the RGB visible light image to obtain the mask data of the irregular parts, and the target pose of the irregular parts is calculated so that the robot arm can grasp the object.

2. The target posture recognition method for robot grasping of parts on a conveyor belt according to claim 1 is characterized in that: In S3, after using the algorithm to segment the point cloud data of the largest plane part in the point cloud data, the normal vector is estimated for the segmented point cloud data, the normal vector is adjusted to point in the same direction of the z-axis, the average normal vector is calculated, the rotation matrix is ​​calculated, and the entire point cloud is rotated to align the average normal vector with the z-axis.

3. The target posture recognition method for robot grasping of parts on a conveyor belt according to claim 2 is characterized in that: The RANSAC algorithm is used to segment the largest plane data in the point cloud in S3.

4. The target posture recognition method for robot grasping of parts on a conveyor belt according to claim 3 is characterized in that: In S3, the KD tree of the K-nearest algorithm is used to estimate the normal vector. Specifically, the domain data of each point within the set radius search range that does not exceed the specified maximum number of neighbors is obtained, and the normal vector is calculated based on the data. The formula is: C=VΛV T Among them, {p i } is point cloud data, where each point is a three-dimensional coordinate p i =(x i ,y i ,z i ), for a target point p0, its neighborhood point set is {p1,p2,…,p k }, where k is the number of neighborhood points, c is the mean of the neighborhood points, C is the covariance matrix, V is the eigenvector matrix, Λ is the eigenvalue diagonal matrix, and the normal vector is the eigenvector corresponding to the smallest eigenvalue.

5. The target posture recognition method for robot grasping of parts on a conveyor belt according to claim 4 is characterized in that: The specific formula for calculating the average normal vector in S3 is: Among them, the normal vector of each point is n i =(n x,i ,n y,i , n z,i ), where i is the index of the point, n x,i , n y,i , n z,i are the x, y, and z components of the normal vector, N is the number of points in the point cloud, and n avg is the calculated average normal vector.

6. The target posture recognition method for robot grasping of parts on a conveyor belt according to claim 1 is characterized in that: The formula for calculating the rotation matrix in S3 is as follows: v=m×t c=m·t Where m is the mean normal vector, t = [0,0,1] is the target vector, v is the cross product of the two vectors, and v x , v y , v z are the x, y, and z components of vector v, s is the modulus of vector v, c is the dot product of the two vectors, skew_matrix represents the antisymmetric matrix of the rotation axis, skew_matrix 2 is the square of the antisymmetric matrix, I is the 3x3 identity matrix, and R is a calculated 3x3 rotation matrix.

7. The target posture recognition method for robot grasping of parts on a conveyor belt according to claim 1 is characterized in that: In S4, the minimum enclosing rectangle of the acquired contour data is calculated, and the SAM model is used to segment the RGB visible light image through the box prompt method to obtain the object mask data.

8. An image processing device, characterized in that: The device includes a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the target posture recognition method for robot grasping of parts on a conveyor belt as described in any one of claims 1 to 7.

9. An edge device, characterized in that: The edge device comprises the image processing device according to claim 8, and depth cameras respectively connected to the image processing device, wherein the depth cameras are RealSence D435i depth cameras; The depth camera is suitable for capturing visible light images and depth images, and the image processing device is used to reconstruct the captured object based on the depth image.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium has program instructions that can be executed by a processor, and the program instructions are used to implement the target posture recognition method for robot grasping of parts on a conveyor belt according to any one of claims 1 to 7.

Images