Multi-class stacked workpiece mechanical arm sorting method based on point cloud and deep learning

Abstract

The present application relates to a kind of based on point cloud and deep learning multi-class stacked workpiece mechanical arm sorting method, based on mechanical arm sorting system to multi-class stacked workpiece is sorted, comprising: using three-dimensional depth camera acquisition each category single workpiece different pose under point cloud data and pre-processing, obtain template point cloud and original point cloud data;Based on original point cloud data to the network model training of point cloud classification network based on deep learning;Using three-dimensional depth camera acquisition actual multi-class stacked workpiece point cloud and pre-processing and segmentation, and using point cloud classification network to each single workpiece point cloud is classified;Each single workpiece point cloud and predicted category template point cloud are registered, based on each single workpiece point cloud corresponding total coincidence degree determines the workpiece point cloud to be grabbed;Establish and calibrate the local coordinate system of workpiece point cloud to be grabbed and determine its space pose, guide mechanical arm to accurately sort it.This method is conducive to improve the accuracy of mechanical arm to multi-class stacked workpiece sorting.

Description

Technical Field

[0001] This invention relates to the field of industrial robotic arms, specifically to a method for sorting multi-category stacked workpieces using a robotic arm based on point cloud and deep learning. Background Technology

[0002] With the continuous upgrading of modern industrial systems, industrial robotic arms have been widely used in industrial production fields such as product sorting, handling, and palletizing due to their advantages of high production efficiency and low post-production costs. In the early days, to achieve batch and streamlined sorting, traditional robotic arm sorting methods were often used. This involved transporting the workpieces to be sorted one by one to a designated location on the production line, and then directly controlling the robotic arm to complete the pre-set sorting actions by recognizing specific markings on various workpieces. However, this overly streamlined sorting operation is actually difficult to handle the workpiece sorting problems in future specific production environments where multiple types of workpieces are stacked.

[0003] To achieve intelligent sorting of multi-category workpieces in stacked scenarios using robotic arms, it is necessary not only to accurately classify each workpiece in the scenario but also to simultaneously determine the actual pose of the most suitable workpiece for grasping, in order to guide the robotic arm in sorting. Currently, the mainstream pure two-dimensional image vision classification method used in industrial production can only obtain the planar contour and position information of the workpiece, but cannot obtain the depth information of the workpiece; in other words, while it is possible to classify workpieces using only two-dimensional images, it is often difficult to accurately obtain the spatial pose information of the workpiece. Compared with two-dimensional images, three-dimensional point cloud data can better and more intuitively describe the actual pose information of spatial objects, while also providing richer geometric information about the object's surface. In the field of industrial production, three-dimensional point clouds have been gradually used in reverse modeling, classification, precise positioning, defect detection, and other fields of various industrial products; however, due to the inherent disorder and unstructured nature of the three-dimensional point cloud data structure, the workpiece point cloud classification problem in multi-category stacked scenarios is quite challenging. To address this, some studies have combined two-dimensional image recognition technology with three-dimensional point cloud technology. The two-dimensional image is used to identify the outline of the workpiece first, and then the two-dimensional recognition result is matched to the three-dimensional point cloud to extract the corresponding workpiece point cloud. However, this approach is prone to problems such as conversion errors between data of different dimensions and loss of workpiece point cloud information. Summary of the Invention

[0004] The purpose of this invention is to provide a sorting method for multi-category stacked workpieces by a robotic arm based on point cloud and deep learning. This method is beneficial to improving the accuracy of the robotic arm in sorting multi-category stacked workpieces.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a robotic arm sorting method for multi-category stacked workpieces based on point cloud and deep learning, providing a robotic arm sorting system mainly composed of a robotic arm, a 3D depth camera, and a computer, wherein the computer is connected to the robotic arm and the 3D depth camera respectively and communicates bidirectionally; sorting multi-category stacked workpieces based on the robotic arm sorting system includes the following steps:

[0006] (1) Using a 3D depth camera, collect multiple sets of complete point cloud data of each type of single workpiece to be sorted under different placement poses and perform point cloud preprocessing to obtain template point clouds of each type of single workpiece and original point cloud data for subsequent point cloud classification network training.

[0007] (2) Based on the original point cloud data of each type of single workpiece, a deep learning-based point cloud classification network is trained to obtain a trained point cloud classification network.

[0008] (3) Use a 3D depth camera to collect point clouds of multi-category stacked workpieces in the actual scene, and preprocess and segment the collected point clouds of multi-category stacked workpieces to obtain multiple segmented single workpiece point clouds.

[0009] (4) Use the trained point cloud classification network to classify the segmented individual workpiece point clouds and obtain their corresponding category labels.

[0010] (5) Based on the registration of the point cloud of each individual workpiece with its predicted category template point cloud, the total overlap of the point cloud of each individual workpiece is calculated based on the bidirectional octree search method, and the point cloud of the workpiece with the highest total overlap is taken as the point cloud of the workpiece to be grasped by the robotic arm.

[0011] (6) Establish and calibrate the local coordinate system of the point cloud of the workpiece to be grasped during each round of grasping by the robotic arm, and determine the spatial pose of the point cloud of the workpiece to be grasped in order to guide the robotic arm to sort it accurately.

[0012] Furthermore, the implementation method of step (1) is as follows:

[0013] Using a 3D depth camera, multiple sets of complete point cloud data of each type of individual workpiece to be sorted are collected under different placement poses. The collected point cloud data of each type of individual workpiece contains at least three dimensions (x, y, z). Then, the collected point cloud data of each type of individual workpiece is preprocessed, including noise reduction and outlier removal, and the template point cloud of each type of individual workpiece and the original point cloud data for subsequent point cloud classification network training are determined.

[0014] Furthermore, in order to reduce the data processing volume of the subsequent point cloud classification network, while ensuring that the original structural features of the workpiece are not lost, the point clouds of each individual workpiece are uniformly downsampled before being input into the deep learning-based point cloud classification network for network model training.

[0015] Furthermore, the implementation method of step (2) is as follows:

[0016] In the point cloud classification network, firstly, a multilayer perceptron is used to appropriately expand the information dimension of the workpiece point cloud, gradually projecting the workpiece feature information contained in the point cloud into higher dimensions, and the internal weight structure of the multilayer perceptron is the same and the parameters are shared; secondly, after processing by the max pooling layer, the overall feature parameters of the workpiece point cloud information are output; finally, the multilayer perceptron classifies the pooled overall feature parameters to determine the corresponding category of the workpiece point cloud; the original point cloud data of individual workpieces of each category are used to train the network model of the point cloud classification network to obtain the optimal network parameters of each layer of the point cloud classification network, thereby obtaining a neural network model that can be used for workpiece point cloud classification.

[0017] Furthermore, the implementation method of step (3) is as follows:

[0018] A 3D depth camera was used to collect point clouds of multi-category stacked workpieces in a real scene; background filtering was performed on the point clouds of multi-category stacked workpieces, and then downsampling was performed on the point clouds after background filtering; in order to remove useless noise points in the point clouds caused by the camera itself or light interference, an outlier removal method based on Euclidean distance was used to denoise the downsampled point clouds.

[0019] Curvature calculation is performed on the preprocessed point cloud. The point with the smallest curvature is selected as the initial segmentation point, and the normals of the remaining points in its neighborhood are calculated. A smoothing threshold is preset. If the angle between the normal at the calculated neighborhood point and the normal at the segmentation point is less than the preset smoothing threshold, the neighborhood point and the segmentation point are determined to belong to the same segmentation body; otherwise, they are determined not to belong to the same segmentation body. Next, another curvature threshold is preset. If the curvature at the neighborhood points belonging to the same segmentation body is less than the preset curvature threshold, the current segmentation point is deleted, and these neighborhood points are regarded as new segmentation points for the next growth search. This process continues until no new segmentation points are found, thus completing the segmentation of a single workpiece point cloud. The above segmentation process is repeated to obtain the segmented point clouds of other single workpieces.

[0020] Furthermore, due to mutual occlusion between stacked workpieces, some incomplete small individual workpiece point clouds exist after segmentation; a point count threshold is set for the individual workpiece point cloud, and small workpiece point clouds containing fewer points than the threshold are filtered out, leaving only large individual workpiece point clouds containing more points.

[0021] Furthermore, the implementation method of step (4) is as follows:

[0022] To reduce the data processing volume of the point cloud classification network, on the premise of ensuring that the structural features of the workpiece point cloud are not lost, the point clouds of each individual workpiece selected in step (3) are subjected to uniform downsampling processing, and the downsampled point clouds of each individual workpiece are input into the trained point cloud classification network to obtain the predicted category of each individual workpiece.

[0023] Furthermore, the implementation method of step (5) is as follows:

[0024] The point clouds of each individual workpiece are registered with their corresponding predicted category template point clouds: First, the point pair relationship between points in the point cloud and their neighboring points is established using the fast point feature histogram method, and point pairs with similar feature values ​​are searched in the template point cloud and the individual workpiece point cloud for coarse matching; Second, a distance error threshold is set, and the distance error between the nearest point pair between each individual workpiece point cloud and the template point cloud after coarse matching is calculated using the iterative nearest point method. When the distance error is less than the threshold, the fine matching between the template point cloud and the individual workpiece point cloud is completed.

[0025] After matching the point cloud of each individual workpiece with its corresponding predicted category template point cloud, a bidirectional octree search method is used to calculate the overlap degree twice, in both positive and negative directions. The sum of the two overlap degrees is used as the final judgment standard. That is, the individual workpiece point cloud with the highest sum of overlap degrees is the one with the correct predicted category, is not occluded, and has the most complete point cloud information. It is regarded as the workpiece point cloud to be grasped by the robotic arm.

[0026] Furthermore, suppose that the point cloud of the individual workpiece obtained by segmentation has a total of The first one is calculated using the bidirectional octree search method. The specific method for determining the overlap ratio of individual workpieces is as follows:

[0027] Forward octree search: An octree mesh is constructed for the point cloud of the γth individual workpiece, and an appropriate mesh size is set; the total number of points in the corresponding category template point cloud is denoted as . Initialize the number of overlapping points Traversing points in a template point cloud Then, query whether the octree mesh containing the coordinates of point i contains a point from the point cloud of the workpiece to be captured. If it does, then consider point i as a coincident point and update the number of coincident points. If not included, the number of overlapping points will not be updated. in, This represents the number of overlapping points obtained after querying point i, and After the traversal is complete, use the final number of overlapping points. Calculate the overlap between the point cloud of a single workpiece and the point cloud of its corresponding category template:

[0028]

[0029] Reverse octree search: An octree mesh is constructed for the corresponding category template point cloud of the γth individual workpiece, and an appropriate mesh size is set; the total number of points in the point cloud of the γth individual workpiece is denoted as . Initialize the number of overlapping points Traversing the points in the point cloud of a single workpiece Then, query whether the octree grid contains a point from the template point cloud using the coordinates corresponding to point j. If it does, then treat point j as a coincident point and update the number of coincident points. If not included, the number of overlapping points will not be updated. in, This represents the number of overlapping points obtained after querying point j, and After the traversal is complete, use the final number of overlapping points. Calculate the overlap between the template point cloud and the individual workpiece point cloud:

[0030]

[0031] Therefore, the total overlap degree corresponding to the point cloud of the γth individual workpiece is:

[0032]

[0033] By comparing the total overlap of the point clouds of each individual workpiece, the point cloud of the workpiece with the highest total overlap is regarded as the point cloud of the workpiece to be captured.

[0034] Furthermore, the implementation method of step (6) is as follows:

[0035] To guide the robotic arm in grasping the workpiece, it is necessary to determine the centroid of the point cloud of the workpiece to be grasped, and establish a local coordinate system of the point cloud of the workpiece to be grasped with this centroid as the origin; let o w The centroid of the point cloud of the workpiece to be grasped is represented by the following formula, and its position coordinates are obtained by solving:

[0036]

[0037] Among them, (x k ,y k ,z k () represents the three-dimensional coordinates of point k (k = 1, 2, ..., N) in the point cloud of the workpiece to be captured in the camera coordinate system; N is the total number of points contained in the point cloud of the workpiece to be captured.

[0038] At the center of mass w Calculate the symmetric, positive definite covariance matrix of the workpiece's point cloud:

[0039]

[0040] in,

[0041] Find the covariance matrix C w The three eigenvalues ​​λ1, λ2, and λ3 and their corresponding three unit eigenvectors The lines containing these three feature vectors are defined as the x-coordinates of the local coordinate system of the point cloud of the workpiece to be captured. w axis, y w axis, z w The axes are set, and the directions of the three feature vectors are initially set as the positive directions of the three coordinate axes of the local coordinate system of the workpiece point cloud;

[0042] Considering that the workpiece to be grasped is different in each grasping cycle of the robotic arm, the corresponding local coordinate system z of the point cloud of the workpiece to be grasped is... w The axis orientations are also different; if the robotic arm uses its end effector coordinate system when grasping a workpiece... t -x t y t z t z t The axis and the local coordinate system of the point cloud of the workpiece to be grasped. w -x w y w z w z w The operation method involves aligning the axes before moving them close together for gripping;

[0043] To ensure consistency in the robotic arm's grasping operations, it is necessary to adjust the local coordinate system z of the point cloud of the workpiece to be grasped during each grasping cycle. w The positive direction of the axis is corrected in a timely manner to unify z. w The orientation of the positive axis; for this purpose, the camera coordinate system o c -x c y c z c z c With the positive axis as the reference, the local coordinate system z of the point cloud of the workpiece to be grasped during each gripping round of the robotic arm. w Correction is performed in the positive direction of the axis; assume For camera coordinate system z c The unit vector of the axis, if Indicates the local coordinate system z of the point cloud of the workpiece to be grasped. w Axis and camera coordinate system z c If the axes are roughly the same, then the initially defined local coordinate system x of the workpiece point cloud is maintained. w axis, y w axis, z w The positive direction of the axis remains unchanged; if This indicates that the local coordinate system z of the point cloud of the workpiece to be captured is... w Axis and camera coordinate system z cWith the axes oriented roughly opposite, the local coordinate system z of the point cloud of the workpiece to be grasped should be adjusted. w The positive correction of the axis is z. w The axis is initially determined to be in the opposite direction of the positive direction, while maintaining the local coordinate system y of the workpiece point cloud. w With the positive axis unchanged, the local coordinate system x of the point cloud of the workpiece to be grasped is recalibrated according to the right-hand rule of the coordinate system. w The positive direction of the axis refers to the local coordinate system x of the point cloud of the workpiece to be grasped. w The positive correction of the axis is x. w The axis is initially determined to be in the opposite direction to the positive direction;

[0044] After calibration, determine the local coordinate system o of the point cloud of the workpiece to be grasped. w -x w y w z w Relative to camera coordinate system o c -x c y c z c The rotation matrix R1∈R 3×3 And combined with the local coordinate system o w -x w y w z w In camera coordinate system o c -x c y c z c The position vector T1∈R 3×1 The homogeneous transformation matrix between the local coordinate system of the workpiece to be grasped and the camera coordinate system is obtained:

[0045]

[0046] Among them, 0 1×3 Represents a zero vector with 1 row and 3 columns;

[0047] Using the cone tip calibration method, the position of the standard cone on the robotic arm's worktable was adjusted multiple times, and the origin of the coordinate system of the robotic arm's end effector was controlled. t Align with the cone tip to obtain multiple sets of cone tips in the camera coordinate system o. c -x c y c z c and the robot arm coordinate system o r -x r y r z r The spatial coordinates are then used to calculate the rotation matrix R2∈R2 of the camera coordinate system relative to the robot arm coordinate system. 3×3 Combined with the position vector T2∈R of the camera coordinate system in the robot arm coordinate system 3×1The homogeneous transformation matrix between the camera coordinate system and the robot arm coordinate system is obtained as follows:

[0048]

[0049] Therefore, the local coordinate system o of the point cloud of the workpiece to be captured w -x w y w z w Relative to the robot arm coordinate system o r -x r y r z r The homogeneous transformation matrix is ​​finally expressed as:

[0050] M = M2·M1

[0051] By combining the point cloud information and homogeneous transformation matrix M of the workpiece to be grasped, the spatial pose of the workpiece to be grasped is calculated, thereby guiding the robotic arm to perform sorting operations on the workpiece to be grasped.

[0052] Compared with existing technologies, the present invention has the following advantages: The present invention directly starts from three-dimensional point clouds, and effectively segments the multi-category stacked workpiece point clouds in the original scene by adopting appropriate point cloud processing methods. It also uses deep learning technology to classify the segmented individual workpiece point clouds, thus maximizing the integrity of the spatial point cloud information of each workpiece and avoiding the data conversion errors and workpiece point cloud information loss that may occur in the traditional "two-dimensional image classification + three-dimensional point cloud extraction" method. In order to effectively filter out incomplete workpiece point clouds that are missing information due to workpiece stacking, do not meet the grasping conditions, and workpiece point clouds with incorrect category prediction, the present invention proposes to first register each individual workpiece point cloud with its predicted category standard template point cloud, and then calculate the total overlap degree of each individual workpiece point cloud based on the bidirectional octree search method. The workpiece point cloud with the highest total overlap degree is taken as the workpiece point cloud with the most complete point cloud information and the correct predicted category to be grasped. In addition, to ensure the consistency of the robotic arm's grasping operation, the present invention also actively corrects the local coordinate system of the point cloud of the workpiece to be grasped in each round of grasping by the robotic arm, and calculates the actual pose of the workpiece to be grasped on this basis, so as to guide the robotic arm to accurately sort it. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the robotic arm sorting system in an embodiment of the present invention;

[0054] In the image: 1. 3D depth camera; 2. Industrial robotic arm; 3. Stacked workpieces of various types; 4. Worktable; 5. Computer;

[0055] Figure 2 This is a flowchart illustrating the method implementation of an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the structure of a point cloud classification network based on deep learning in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the local coordinate system correction of the point cloud of the workpiece to be grasped in an embodiment of the present invention; wherein, (1) is the initially determined local coordinate system of the workpiece to be grasped, and (2) is the corrected local coordinate system of the workpiece to be grasped;

[0058] Figure 5 This is a schematic diagram illustrating the overlap calculation based on point cloud registration and bidirectional octree search in an embodiment of the present invention. Detailed Implementation

[0059] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0060] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0061] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0062] This embodiment provides a robotic arm sorting method for multi-category stacked workpieces based on point cloud and deep learning. It first provides a robotic arm sorting system, and then sorts multi-category stacked workpieces based on the robotic arm sorting system.

[0063] like Figure 1 As shown, the robotic arm sorting system consists of a 3D depth camera 1, an industrial robotic arm 2, a worktable 4, and a computer 5. The computer 5 is connected to the industrial robotic arm 2 and the 3D depth camera 1, enabling bidirectional communication. Multiple types of stacked workpieces 3 can be placed on the worktable 4.

[0064] like Figure 2 As shown, the multi-category stacked workpiece sorting method of the robotic arm includes the following steps:

[0065] (1) Using a 3D depth camera, collect multiple sets of complete point cloud data of each type of single workpiece to be sorted under different placement poses and perform point cloud preprocessing to obtain template point clouds of each type of single workpiece and original point cloud data for subsequent point cloud classification network training.

[0066] (2) Based on the original point cloud data of each type of single workpiece, a deep learning-based point cloud classification network is trained to obtain a trained point cloud classification network.

[0067] (3) Use a 3D depth camera to collect point clouds of multi-type stacked workpieces in the actual scene, and preprocess and segment the collected point clouds of multi-type stacked workpieces to obtain multiple segmented single workpiece point clouds.

[0068] (4) Use the trained point cloud classification network to classify the point clouds of each individual workpiece after segmentation and obtain their corresponding category labels.

[0069] (5) Based on the registration of the point cloud of each individual workpiece with its predicted category template point cloud, the total overlap of each individual workpiece point cloud is calculated based on the bidirectional octree search method, and the workpiece point cloud with the highest total overlap is taken as the workpiece point cloud to be grasped by the robotic arm.

[0070] (6) Establish and calibrate the local coordinate system of the point cloud of the workpiece to be grasped during each round of grasping by the robotic arm, and determine the spatial pose of the point cloud of the workpiece to be grasped in order to guide the robotic arm to sort it accurately.

[0071] The relevant content involved in this method will be further detailed below.

[0072] S1: In this invention, a three-dimensional depth camera is used to collect multiple sets of complete point cloud data of each type of single workpiece in the above system under different placement poses, and the collected point cloud data of each type of single workpiece should contain at least three dimensions (x, y, z); then, the collected point cloud data of each type of single workpiece is subjected to point cloud preprocessing such as noise reduction and outlier removal, and the template point cloud of each type of single workpiece and the original point cloud data for subsequent point cloud classification network training are determined.

[0073] To reduce the data processing load of the subsequent point cloud classification network, while ensuring that the original structural features of the workpiece are not lost, the collected point clouds of each individual workpiece are uniformly downsampled before being input into a network such as... Figure 3 The network model is trained in the deep learning-based point cloud classification network shown.

[0074] In the point cloud classification network, firstly, a multilayer perceptron is used to appropriately expand the information dimension of the workpiece point cloud, gradually projecting the workpiece feature information contained in the point cloud into higher dimensions. The internal weight structure of the multilayer perceptron is identical, and parameters are shared. Secondly, after processing by a max pooling layer, the overall feature parameters of the workpiece point cloud information are output. Finally, the multilayer perceptron classifies the pooled overall feature parameters to determine the corresponding category of the workpiece point cloud. By using the original point cloud data of individual workpieces of the aforementioned categories to train the point cloud classification network model, the optimal network parameters of each layer of the point cloud classification network can be obtained, thus yielding a neural network model that can be used for workpiece point cloud classification.

[0075] S2: Use a 3D depth camera to acquire point clouds of multi-category stacked workpieces in a real-world scene; perform background filtering on the point clouds of multi-category stacked workpieces, and then perform downsampling processing on the workpiece point clouds after background filtering. In order to remove useless noise points in the workpiece point clouds caused by camera interference or lighting, an outlier removal method based on Euclidean distance is used to denoise the point clouds.

[0076] S3: Calculate the curvature of the point cloud, select the point with the smallest curvature as the initial segmentation point, and calculate the normals of the remaining points within its neighborhood. By presetting a smoothing threshold, if the angle between the calculated normal at a neighboring point and the normal at the segmentation point is less than the given smoothing threshold, the neighboring point and the segmentation point are determined to belong to the same segmentation body; otherwise, they are determined not to belong to the same segmentation body. Next, preset another curvature threshold. If the curvature at the neighboring points belonging to the same segmentation body is less than the given curvature threshold, delete the current segmentation point and treat these neighboring points as new segmentation points for the next growth search, until no new segmentation points are found, thus completing the segmentation of a single workpiece point cloud. Repeat the above growth and segmentation process to obtain the segmented point clouds of other single workpieces.

[0077] Because stacked workpieces occlude each other, some incomplete small individual workpiece point clouds will inevitably exist after segmentation. By setting a threshold for the number of points in the individual workpiece point cloud, small workpiece point clouds containing fewer points than the threshold are filtered out, and only large individual workpiece point clouds containing more points are retained.

[0078] S4: To reduce the data processing volume of the point cloud classification network, while ensuring that the structural features of the workpiece point cloud are not lost, the point clouds of each individual workpiece selected above are subjected to uniform downsampling processing, and the downsampled point clouds of each individual workpiece are input into the trained point cloud classification network to obtain the predicted category of each individual workpiece.

[0079] S5: Register the point clouds of each individual workpiece with their corresponding predicted category template point clouds. First, the Fast Point Feature Histogram (FPFH) method is used to establish point pairs between points in the point cloud and their neighboring points, and point pairs with similar feature values ​​are searched in the template point cloud and the individual workpiece point cloud for coarse matching. Second, a distance error threshold is set, and the Iterative Closest Point (ICP) method is used to calculate the distance error between the nearest point pairs of each individual workpiece point cloud and the template point cloud after coarse matching. When the distance error is less than the threshold, the fine matching between the template point cloud and the individual workpiece point cloud is completed.

[0080] S6: Among the multiple individual workpiece point clouds obtained from the above segmentation, there are still some large residual point clouds retained due to mutual occlusion between workpieces. The workpieces corresponding to these residual point clouds may not actually meet the grasping conditions. Therefore, based on the coarse and fine matching of each individual workpiece point cloud and its template point cloud, the point cloud octree search method should be used to further calculate the overlap between each individual workpiece point cloud and its template point cloud. The individual workpiece point cloud with the highest overlap can be regarded as the unoccluded workpiece point cloud with the most complete point cloud information to be grasped. However, considering that there may be some similar shape features between some workpieces of different categories in the stacked scene, if the aforementioned retained residual point clouds happen to originate from the similar parts between these workpieces, when using the aforementioned point cloud classification network to classify and predict them, it is very likely to obtain incorrect classification results. At this time, after registering the individual workpiece point cloud with the incorrectly predicted template point cloud, if only the one-way octree search method is used to calculate the overlap between the two, it is very likely that the overlap will be too high, thus affecting the judgment result. To solve this problem, such as Figure 5 As shown, after completing the coarse and fine matching of the point clouds of each individual workpiece with their corresponding predicted category template point clouds, the bidirectional octree search method is used to calculate the overlap degree twice, and the sum of the two overlap degrees is used as the final judgment standard. That is, the point cloud of the individual workpiece with the highest sum of overlap degrees is the one with the correct predicted category, is not occluded, and has the most complete point cloud information. It is finally regarded as the point cloud of the workpiece to be grasped by the robotic arm.

[0081] Suppose that the point cloud of the individual workpiece obtained by the aforementioned segmentation has a total of The first one is calculated using the bidirectional octree search method. The specific process for verifying the overlap of individual workpieces is as follows:

[0082] Forward octree search: An octree mesh is constructed for the point cloud of the γth individual workpiece, and an appropriate mesh size is set; the total number of points in the corresponding category template point cloud is denoted as . Initialize the number of overlapping points Traversing points in a template point cloud Then, query whether the octree mesh containing the coordinates of point i contains a point from the point cloud of the workpiece to be captured. If it does, then consider point i as a coincident point and update the number of coincident points. If not included, the number of overlapping points will not be updated. in, This represents the number of overlapping points obtained after querying point i, and After the traversal is complete, use the final number of overlapping points. Calculate the overlap between the point cloud of a single workpiece and the point cloud of its corresponding category template:

[0083]

[0084] Reverse octree search: An octree mesh is constructed for the corresponding category template point cloud of the γth individual workpiece, and an appropriate mesh size is set; the total number of points in the point cloud of the γth individual workpiece is denoted as . Initialize the number of overlapping points Traversing the points in the point cloud of a single workpiece Then, query whether the octree grid contains a point from the template point cloud using the coordinates corresponding to point j. If it does, then treat point j as a coincident point and update the number of coincident points. If not included, the number of overlapping points will not be updated. in, This represents the number of overlapping points obtained after querying point j, and After the traversal is complete, use the final number of overlapping points. Calculate the overlap between the template point cloud and the individual workpiece point cloud:

[0085]

[0086] Therefore, the total overlap degree corresponding to the point cloud of the γth individual workpiece is:

[0087]

[0088] By comparing the total overlap of the point clouds of each individual workpiece, the point cloud of the workpiece with the highest total overlap is regarded as the point cloud of the workpiece to be captured.

[0089] S7: To guide the robotic arm in grasping the workpiece, the centroid of the point cloud of the workpiece to be grasped needs to be determined, and a local coordinate system of the point cloud of the workpiece to be grasped needs to be established with this centroid as the origin. Let o w The centroid of the point cloud of the workpiece to be grasped can be determined by the following formula:

[0090]

[0091] Among them, (x k ,y k ,z k) represents the three-dimensional coordinates of point k (k = 1, 2, ..., N) in the point cloud of the workpiece to be captured in the camera coordinate system; N is the total number of points contained in the point cloud of the workpiece to be captured.

[0092] At the center of mass w Calculate the symmetric, positive definite covariance matrix of the workpiece's point cloud:

[0093]

[0094] in,

[0095] Find the covariance matrix C w The three eigenvalues ​​λ1, λ2, and λ3 and their corresponding three unit eigenvectors The lines containing these three feature vectors are defined as the x-coordinates of the local coordinate system of the point cloud of the workpiece to be captured. w axis, y w axis, z w The axes are defined, and the directions of the three feature vectors are initially set to the positive directions of the three coordinate axes of the local coordinate system of the workpiece point cloud.

[0096] Considering that the workpiece to be grasped is different in each grasping cycle of the robotic arm, the corresponding local coordinate system z of the point cloud of the workpiece to be grasped is... w The axis orientations are also different; if the robotic arm uses its end effector coordinate system when grasping a workpiece... t -x t y t z t z t The axis and the local coordinate system of the point cloud of the workpiece to be grasped. w -x w y w z w z w The operation involves aligning the axes before moving closer together for gripping. To ensure consistency in the robotic arm's gripping operations, the local coordinate system z-axis of the workpiece point cloud needs to be adjusted for each gripping cycle. w The positive direction of the axis is corrected in a timely manner to unify z. w The orientation of the positive axis.

[0097] Therefore, such as Figure 4 As shown, with camera coordinate system o c -x c y c z c z c With the positive axis as the reference, the local coordinate system z of the point cloud of the workpiece to be grasped during each gripping round of the robotic arm. w Correction is performed in the positive direction of the axis. Let... For camera coordinate system z c The unit vector of the axis, if Indicates the local coordinate system z of the point cloud of the workpiece to be grasped. w Axis and camera coordinate system z c If the axes are roughly the same, then the initially defined local coordinate system x of the workpiece point cloud is maintained. w axis, y w axis, z w The positive direction of the axis remains unchanged; if This indicates that the local coordinate system z of the point cloud of the workpiece to be captured is... w Axis and camera coordinate system z c With the axes oriented roughly opposite, the local coordinate system z of the point cloud of the workpiece to be grasped should be adjusted. w The positive correction of the axis is z. w The axis is initially determined to be in the opposite direction of the positive direction, while maintaining the local coordinate system y of the workpiece point cloud. w With the positive axis unchanged, the local coordinate system x of the point cloud of the workpiece to be grasped is recalibrated according to the right-hand rule of the coordinate system. w The positive direction of the axis (i.e., the local coordinate system x of the point cloud of the workpiece to be grasped) w The positive correction of the axis is x. w The axis is initially determined to be in the opposite direction of the positive direction.

[0098] S8: After calibration is complete, the local coordinate system o of the point cloud of the workpiece to be grasped can be determined. w -x w y w z w Relative to camera coordinate system o c -x c y c z c The rotation matrix R1∈R 3×3 And combined with the local coordinate system o w -x w y w z w In camera coordinate system o c -x c y c z c The position vector T1∈R 3×1 The homogeneous transformation matrix between the local coordinate system of the workpiece to be grasped and the camera coordinate system is obtained:

[0099]

[0100] Among them, 0 1×3 This represents a zero vector with 1 row and 3 columns.

[0101] S9: Using the cone tip calibration method, the position of the standard cone on the robotic arm's worktable is adjusted multiple times, and the origin of the robotic arm's end effector coordinate system is controlled. t Align with the cone tip to obtain multiple sets of cone tips in the camera coordinate system o.c -x c y c z c and the robot arm coordinate system o r -x r y r z r The spatial coordinates are then used to calculate the rotation matrix R2∈R2 of the camera coordinate system relative to the robot arm coordinate system. 3×3 Combined with the position vector T2∈R of the camera coordinate system in the robot arm coordinate system 3×1 The homogeneous transformation matrix between the camera coordinate system and the robot arm coordinate system can be obtained:

[0102]

[0103] S10: Therefore, the local coordinate system o of the point cloud of the workpiece to be captured w -x w y w z w Relative to the robot arm coordinate system o r -x r y r z r The homogeneous transformation matrix can be finally expressed as:

[0104] M = M2·M1

[0105] By combining the point cloud information of the workpiece to be grasped and the homogeneous transformation matrix M, the spatial pose of the workpiece to be grasped can be calculated, thereby guiding the robotic arm to perform sorting operations on the workpiece to be grasped.

[0106] This embodiment also provides a multi-category stacked workpiece sorting system based on point cloud and deep learning, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-mentioned method steps.

[0107] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for sorting multi-category stacked workpieces using a robotic arm based on point cloud and deep learning, characterized in that, A robotic arm sorting system is provided, mainly composed of a robotic arm, a 3D depth camera, and a computer. The computer is connected to the robotic arm and the 3D depth camera and communicates bidirectionally. The robotic arm sorting system is used to sort multiple categories of stacked workpieces, including the following steps: (1) Using a 3D depth camera, collect multiple sets of complete point cloud data of each type of single workpiece to be sorted under different placement poses and perform point cloud preprocessing to obtain template point clouds of each type of single workpiece and original point cloud data for subsequent point cloud classification network training. (2) Based on the original point cloud data of each type of single workpiece, a deep learning-based point cloud classification network is trained to obtain a trained point cloud classification network. (3) Use a 3D depth camera to collect point clouds of multi-category stacked workpieces in the actual scene, and preprocess and segment the collected point clouds of multi-category stacked workpieces to obtain multiple segmented single workpiece point clouds. (4) Use the trained point cloud classification network to classify the segmented individual workpiece point clouds and obtain their corresponding category labels; (5) Based on the registration of the point cloud of each individual workpiece with its predicted category template point cloud, the total overlap of the point cloud of each individual workpiece is calculated based on the bidirectional octree search method, and the point cloud of the workpiece with the highest total overlap is taken as the point cloud of the workpiece to be grasped by the robotic arm. (6) Establish and calibrate the local coordinate system of the point cloud of the workpiece to be grasped during each round of grasping by the robotic arm, and determine the spatial pose of the point cloud of the workpiece to be grasped in order to guide the robotic arm to sort it accurately. The implementation method of step (5) is as follows: The point clouds of each individual workpiece are registered with their corresponding predicted category template point clouds: First, the point pair relationship between points in the point cloud and their neighboring points is established using the fast point feature histogram method, and point pairs with similar feature values ​​are searched in the template point cloud and the individual workpiece point cloud for coarse matching; Second, a distance error threshold is set, and the distance error between the nearest point pair between each individual workpiece point cloud and the template point cloud after coarse matching is calculated using the iterative nearest point method. When the distance error is less than the threshold, the fine matching between the template point cloud and the individual workpiece point cloud is completed. After matching the point cloud of each individual workpiece with its corresponding predicted category template point cloud, the bidirectional octree search method is used to calculate the overlap degree twice, and the sum of the two overlap degrees is used as the final judgment standard. That is, the individual workpiece point cloud with the highest overlap degree is the one with the correct predicted category, is not occluded, and has the most complete point cloud information. It is regarded as the workpiece point cloud to be grasped by the robotic arm. Suppose that the point cloud of the individual workpiece obtained by segmentation has a total of The first one is calculated using the bidirectional octree search method. The specific method for determining the overlap ratio of individual workpieces is as follows: Forward octree search: for the first An octree mesh is created for each individual workpiece point cloud, and an appropriate mesh size is set; the total number of points in the corresponding category template point cloud is denoted as . Initialize the number of overlapping points ; Traverse the points in the template point cloud and with points The corresponding coordinates are used to check if the octree mesh contains a point from the point cloud of the workpiece to be captured. If it does, the point is... Treat them as overlapping points and update the number of overlapping points. If not included, the number of overlapping points will not be updated. ;in, Indicates query point The number of overlapping points obtained later, and After the traversal is complete, use the final number of overlapping points. Calculate the overlap between the point cloud of a single workpiece and the point cloud of its corresponding category template: Reverse octree search: for the first An octree mesh is constructed from the corresponding category template point cloud of each individual workpiece, and an appropriate mesh size is set; [The last part is a fragment and doesn't need a direct translation.] The total number of points in the point cloud of a single workpiece is Initialize the number of overlapping points ; Traverse the points in the point cloud of a single workpiece and with points The corresponding coordinates are used to check if the octree grid contains points from the template point cloud. If it does, then the points are... Treat them as overlapping points and update the number of overlapping points. If not included, the number of overlapping points will not be updated. ;in, Indicates query point The number of overlapping points obtained later, and After the traversal is complete, use the final number of overlapping points. Calculate the overlap between the template point cloud and the individual workpiece point cloud: Therefore, the first The total overlap of the point cloud of each individual workpiece is: By comparing the total overlap of the point clouds of each individual workpiece, the point cloud of the workpiece with the highest total overlap is regarded as the point cloud of the workpiece to be captured.

2. The multi-category stacked workpiece sorting method based on point cloud and deep learning according to claim 1, characterized in that, The implementation method for step (1) is as follows: Using a 3D depth camera, multiple sets of complete point cloud data of each type of individual workpiece to be sorted are collected under different placement poses. The collected point cloud data of each type of individual workpiece must contain at least [data missing]. The three dimensions of information are then used to preprocess the point cloud data of each category of individual workpieces, including noise reduction and outlier removal, and to determine the template point cloud of each category of individual workpieces and the original point cloud data for subsequent point cloud classification network training.

3. The multi-category stacked workpiece sorting method based on point cloud and deep learning according to claim 2, characterized in that, To reduce the data processing volume of the subsequent point cloud classification network, while ensuring that the original structural features of the workpiece are not lost, the point clouds of each individual workpiece are uniformly downsampled before being input into the deep learning-based point cloud classification network for network model training.

4. The multi-category stacked workpiece sorting method based on point cloud and deep learning according to claim 1, characterized in that, The implementation method for step (2) is as follows: In the point cloud classification network, firstly, a multilayer perceptron is used to appropriately expand the information dimension of the workpiece point cloud, gradually projecting the workpiece feature information contained in the point cloud into higher dimensions, and the internal weight structure of the multilayer perceptron is the same and the parameters are shared; secondly, after processing by the max pooling layer, the overall feature parameters of the workpiece point cloud information are output; finally, the multilayer perceptron classifies the pooled overall feature parameters to determine the corresponding category of the workpiece point cloud; the original point cloud data of individual workpieces of each category are used to train the network model of the point cloud classification network to obtain the optimal network parameters of each layer of the point cloud classification network, thereby obtaining a neural network model that can be used for workpiece point cloud classification.

5. The multi-category stacked workpiece sorting method based on point cloud and deep learning according to claim 1, characterized in that, The implementation method for step (3) is as follows: A 3D depth camera was used to collect point clouds of multi-category stacked workpieces in a real scene; background filtering was performed on the point clouds of multi-category stacked workpieces, and then downsampling was performed on the point clouds after background filtering; in order to remove useless noise points in the point clouds caused by the camera itself or light interference, an outlier removal method based on Euclidean distance was used to denoise the downsampled point clouds. Curvature calculation is performed on the preprocessed point cloud. The point with the smallest curvature is selected as the initial segmentation point, and the normals of the remaining points in its neighborhood are calculated. A smoothing threshold is preset. If the angle between the normal at a neighboring point and the normal at the segmentation point is less than the preset smoothing threshold, the neighboring point and the segmentation point are determined to belong to the same segmentation body; otherwise, they are determined not to belong to the same segmentation body. Next, a curvature threshold is preset. If the curvature at a neighboring point belonging to the same segmentation body is less than the preset curvature threshold, the current segmentation point is deleted and these neighboring points are regarded as new segmentation points for the next growth search, until no new segmentation points are found, thus completing the segmentation of a single workpiece point cloud. The above segmentation process is repeated to obtain the segmented point clouds of other single workpieces.

6. The multi-category stacked workpiece sorting method based on point cloud and deep learning according to claim 5, characterized in that, Because the stacked workpieces occlude each other, there are some incomplete small individual workpiece point clouds after segmentation; a point number threshold is set for the individual workpiece point cloud, and small workpiece point clouds with fewer than the point number threshold are filtered out, and only large individual workpiece point clouds with more points are retained.

7. The multi-category stacked workpiece sorting method based on point cloud and deep learning according to claim 1, characterized in that, The implementation method for step (4) is as follows: To reduce the data processing volume of the point cloud classification network, on the premise of ensuring that the structural features of the workpiece point cloud are not lost, the point clouds of each individual workpiece selected in step (3) are subjected to uniform downsampling, and the downsampled point clouds of each individual workpiece are input into the trained point cloud classification network to obtain the predicted category of each individual workpiece.

8. The multi-category stacked workpiece sorting method based on point cloud and deep learning according to claim 1, characterized in that, The implementation method for step (6) is as follows: To guide the robotic arm in grasping the workpiece, it is necessary to determine the centroid of the point cloud of the workpiece to be grasped, and establish a local coordinate system of the point cloud of the workpiece to be grasped with this centroid as the origin; let... The centroid of the point cloud of the workpiece to be grasped is represented by the following formula, and its position coordinates are obtained by solving: in, Indicates the midpoint of the point cloud of the workpiece to be grabbed. Three-dimensional coordinates in camera coordinates; The total number of points contained in the point cloud of the workpiece to be captured; At the center of mass Calculate the symmetric, positive definite covariance matrix of the workpiece's point cloud: in, ; Find the covariance matrix The three eigenvalues , , and its three corresponding unit eigenvectors , , The lines containing these three feature vectors are defined as the local coordinate system of the point cloud of the workpiece to be captured. axis, axis, The axes are set, and the directions of the three feature vectors are initially set as the positive directions of the three coordinate axes of the local coordinate system of the workpiece point cloud; Considering that the workpiece to be grasped is different in each grasping round of the robotic arm, the corresponding point cloud local coordinate system of the workpiece to be grasped... The axis orientations are also different; if the robotic arm uses its end effector coordinate system when grasping a workpiece... of Axis and local coordinate system of point cloud of workpiece to be grasped of The operation method involves aligning the axes before moving them close together for gripping; To ensure consistency in the robotic arm's grasping operations, it is necessary to adjust the local coordinate system of the point cloud of the workpiece to be grasped during each grasping cycle. The positive direction of the shaft is corrected in a timely manner to ensure uniformity. The orientation of the positive axis; for this, the camera coordinate system... of With the positive axis as the reference, the local coordinate system of the point cloud of the workpiece to be grasped during each gripping round of the robotic arm. Correction is performed in the positive direction of the axis; assume Camera coordinate system The unit vector of the axis, if This indicates the local coordinate system of the point cloud of the workpiece to be captured. Axis and camera coordinate system If the axes are roughly the same, then the initially defined local coordinate system of the workpiece point cloud is maintained. axis, axis, The positive direction of the axis remains unchanged; if This indicates the local coordinate system of the point cloud of the workpiece to be captured. Axis and camera coordinate system With the axes oriented roughly opposite, the local coordinate system of the point cloud of the workpiece to be grasped should be adjusted. The positive correction of the axis is The axis is initially set in the opposite direction of the positive direction, while maintaining the local coordinate system of the workpiece point cloud. With the positive axis unchanged, the local coordinate system of the point cloud of the workpiece to be grasped is recalibrated according to the right-hand rule of the coordinate system. The positive axis represents the local coordinate system of the point cloud of the workpiece to be grasped. The positive correction of the axis is The axis is initially determined to be in the opposite direction to the positive direction; After calibration, determine the local coordinate system of the point cloud of the workpiece to be grasped. Relative to camera coordinate system rotation matrix and combined with the local coordinate system In the camera coordinate system Position vector in The homogeneous transformation matrix between the local coordinate system of the workpiece to be grasped and the camera coordinate system is obtained: in, Represents a zero vector with 1 row and 3 columns; Using the cone tip calibration method, the position of the standard cone on the robotic arm's worktable was adjusted multiple times, and the origin of the coordinate system of the robotic arm's end effector was controlled. Align with the cone tip to obtain multiple sets of cone tips in the camera coordinate system. and the robot arm coordinate system The spatial coordinates are then used to calculate the rotation matrix of the camera coordinate system relative to the robot arm coordinate system. Combine the position vector of the camera coordinate system in the robot arm coordinate system The homogeneous transformation matrix between the camera coordinate system and the robot arm coordinate system is obtained as follows: Therefore, the local coordinate system of the point cloud of the workpiece to be captured Relative to the robot arm coordinate system The homogeneous transformation matrix is ​​finally expressed as: Combining the point cloud information and homogeneous transformation matrix of the workpiece to be grasped The spatial pose of the workpiece to be grasped is calculated, and then the robotic arm is guided to perform sorting operations on the workpiece.

Images