Model-free object grasping pose generation method for unstructured work scenarios

Abstract

A model-free object grasping pose generation method for non-structured operation scenarios, after the collected point cloud data is generated into an original scene graph based on a neighborhood node algorithm, the original scene graph is input into a graph feature extraction network to obtain a graph embedding feature fused with multi-scale information, the graph embedding feature is sequentially input into an object surface point screening network and a high-value point screening network and a high-value point graph is obtained, the high-value point graph is input into a pose generation network to obtain a spatial six-degree-of-freedom grasping candidate pose, and then the best grasping is selected for implementation. The present application solves the problems of strong randomness of point selection, limited generalization performance, insufficient spatial feature extraction capability of the network model, and low grasping quality of the generated grasping in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a model-free object grasping pose generation method for non-structural operation scenes. BACKGROUND

[0002] The model-free grasping pose generation method generates a grasping pose and plans a grasping configuration of an end effector only by understanding the structure of a scene without paying attention to the category and number of grasping objects. Existing model-free grasping pose generation methods mainly fall into two categories: the first category of methods randomly samples grasping in a scene, and then uses a network model and an index to evaluate the grasping quality. This kind of method explicitly exists a grasping sampling link, and the core is to select a proper classification model to judge the effectiveness of the grasping pose. The second category of methods uses an end-to-end network model to directly generate a feasible six-degree-of-freedom grasping pose at a reasonable grasping position. This kind of method usually uses a classifier to solve the grasping parameters of each degree of freedom. This way of disassembling the position degree of freedom and the direction degree of freedom makes the model easier to learn. However, the above-mentioned methods still have some problems that are difficult to overcome and perform poorly in some challenging scenes. For example, the traditional point cloud processing network ignores the relative position relationship between scanning points, and it is difficult to widely explore the potential grasping domain and generate reliable grasping in a single-view incomplete point cloud scene. SUMMARY

[0003] The present application proposes a model-free object grasping pose generation method for non-structural operation scenes to solve the problems of strong randomness of point selection, limited generalization performance, insufficient spatial feature extraction capability of the network model, and low grasping quality of the generated grasping pose in the prior art.

[0004] The present application is implemented by the following technical solutions:

[0005] The present application relates to a model-free object grasping pose generation method for non-structural operation scenes. After generating an original scene graph based on the adjacent node algorithm from the collected point cloud data, the original scene graph is input into a graph feature extraction network to obtain graph embedding features fused with multi-scale information. The graph embedding features are sequentially input into an object surface point screening network and a high-value point screening network to obtain a high-value point graph. The high-value point graph is input into a pose generation network to obtain a spatial six-degree-of-freedom grasping candidate pose, and then the best grasping is selected for implementation.

[0006] The original scene graph is obtained by a depth sensor from the original point cloud of the grasping scene, and is obtained through preprocessing and based on the adjacent node algorithm.

[0007] The preprocessing includes scene region cropping and point cloud down-sampling.

[0008] The neighbor node algorithm specifically comprises: 1) taking each point in the preprocessed point cloud as a center point, calculating the Euclidean distance with other points in the scene; 2) finding K nearest neighbor points of the center point, constructing a directed edge from the center point to the neighbor points, and all points in the scene together forming an original scene graph; and 3) setting initial features of each graph node and edge in the original scene graph.

[0009] The graph feature extraction network comprises a graph relation learning module, a global pooling module and a position coding module, wherein: the graph relation learning module calculates a multi-level symmetric normalized adjacency matrix according to the initial features of each graph node in the original scene graph, then generates a multi-level nonlinear embedding feature matrix according to the multi-level symmetric normalized adjacency matrix by using a multi-layer perception machine, finally learns the nonlinear embedding feature weight of each level according to a channel attention mechanism, and obtains the graph node feature information of each node position by comprehensively processing the nonlinear embedding feature matrix; the global pooling module obtains the graph node feature information by using the multi-layer perception machine and the normalized exponential function to process each graph node feature according to the graph node feature information obtained by the graph relation learning module, and obtains the global feature information by summing the above results; and the position coding module obtains the embedded graph node position information, i.e. the graph embedding feature, according to the position feature of each graph node in the original scene graph by using the multi-layer perception machine.

[0010] The object surface point screening network uses a multi-layer graph network composed of a graph convolution operator and a binary classifier to determine whether each graph node in the graph embedding feature belongs to the surface of the object to be grasped, samples M1 graph nodes from the surface of the object based on the farthest point sampling algorithm, and generates a target point graph by using the neighbor node algorithm.

[0011] The high-value point screening network uses a multi-layer network composed of a graph convolution operator and a multi-classifier to predict the grasping value classification of each graph node in the target point graph and the graph embedding feature, optimizes M2 graph nodes with the highest value classification, and generates a high-value point graph by using the neighbor algorithm.

[0012] The optimal grasping is obtained by the following methods:

[0013] i) determining the optimal approaching direction of grasping at the position by using a multi-classifier according to the graph node feature and the spatial coordinates in the high-value point graph;

[0014] ii) obtaining the plane rotation angle and the grasping depth by using a multi-classifier under the selected approaching direction, and regressing to obtain a grasping evaluation score;

[0015] iii) selecting the grasping implementation with the highest grasping evaluation score in the scene.

[0016] This invention relates to a system for implementing the above method, comprising: a graph feature extraction network, an object surface point selection network, a high-value point selection network, and a grasping pose generation network, wherein: the graph feature extraction network obtains graph embedding features by fusing the output features of a graph relationship learning module, a global pooling module, and a position encoding module based on the initial features of each graph node in the original scene graph; the object surface point selection network samples M1 graph nodes belonging to the object surface using a farthest point sampling algorithm based on the graph embedding features; the high-value point selection network selects the M2 graph nodes with the highest value grading as grasping points based on the graph embedding features; and the grasping pose generation network processes the grasping point positions and their graph embedding features using a neural network to obtain a six-degree-of-freedom grasping pose.

[0017] Technical effect

[0018] This invention introduces a channel attention mechanism to enhance graph node features and constructs a two-stage point selection strategy to gradually approach the ideal grasping position. At the same time, it combines the channel attention mechanism with traditional graph convolutional networks and adopts a learned point selection strategy to focus the grasping on the ideal position. Compared with the prior art, this invention achieves a hierarchical understanding of scattered stacked spatial scenes and improves the grasping accuracy of unknown multi-targets in unstructured environments. Attached Figure Description

[0019] Figure 1 This is a flowchart of the present invention;

[0020] Figure 2 This is a schematic diagram of the system of the present invention;

[0021] Figure 3 This is a schematic diagram of the graph relationship learning module;

[0022] Figure 4 This is a graph showing the changes in loss function values ​​and accuracy during the training process.

[0023] Figure 5 A schematic diagram illustrating part of the results generated for capturing pose;

[0024] Figure 6 This is a verification diagram for a physical scenario. Detailed Implementation

[0025] like Figure 1 As shown in the figure, this embodiment relates to a method for generating grasping poses in unstructured task scenarios, including:

[0026] Step 1: Obtain the original point cloud of the scene to be captured using a depth sensor, specifically including:

[0027] Step 1.1: Randomly place the object to be grabbed on the workbench.

[0028] The object to be grabbed can be a new object that does not exist in the training set, and there can be occlusion and coverage between multiple objects. The method understands the geometric features of the scene in a model-free manner and generates reliable grasping poses.

[0029] Step 1.2, fix the depth sensor above the workbench so that the depth sensor can completely capture the entire scene within the workbench range and there is no occlusion or interference between the depth sensor and the object to be grabbed.

[0030] The Photoneo PhoXi three-dimensional scanner is selected as the depth sensor.

[0031] Step 1.3, capture the depth image, and only keep the pixels with a depth greater than 0 in the depth image to obtain the filtered depth image.

[0032] Step 1.4, convert the filtered depth image into a point cloud using a point cloud generation algorithm, and save it in the corresponding point cloud format.

[0033] Step 2, pre-process the original point cloud obtained in step 1, and generate an original scene graph based on the adjacent node algorithm, specifically including:

[0034] Step 2.1, scene region cutting: before training the model, the workbench region range is obtained by manual labeling according to the relative position relationship between the depth sensor and the workbench region; during training and actual testing, only the point cloud within the workbench region range is kept.

[0035] Since the relative position relationship between the depth sensor and the workbench region remains unchanged, it only needs to be manually labeled once.

[0036] Step 2.2, point cloud downsampling: to balance the grasping pose detection effect and hardware computing power, the cut point cloud is randomly downsampled to N points, and the obtained point cloud scene is wherein: is the point cloud scene, p i is the coordinate of each point in the point cloud.

[0037] The N is 12000.

[0038] Step 2.3, generate an original scene graph based on the adjacent node algorithm: take each point in the pre-processed point cloud as a center point Calculate the Euclidean distance with other points in the scene; find the K nearest neighbors of the center point Construct a directed edge from the center point to the adjacent point, and all directed edges form an edge set All points in the scene together form an original scene graph

[0039] The initial feature of each node in the original scene graph is set as its position coordinate p. i The edge feature is empty, and K is 32.

[0040] Step 3: Based on the original scene graph, construct a graph feature extraction network including a graph relationship learning module, a global pooling module, and a position encoding module, and fuse multi-scale information to obtain graph embedding features, specifically including:

[0041] Step 3.1: Obtain graph node feature information from the graph relationship learning module: such as... Figure 3 As shown, based on the original scene diagram Generate adjacency matrix Where: the elements w of the adjacency matrix W ij The symmetric normalized adjacency matrix A of level p is greater than 0 if and only if there is an edge connection between the i-th node and the j-th node. p =A p , in: For the angle matrix D, the elements d ii For graph node V i The degree.

[0042] Specifically, the symmetric normalized adjacency matrices at each level are uniquely determined by the original scene graph, and A is calculated. p Since no learning parameters are involved, they can be calculated in the preprocessing step.

[0043] The p-th level nonlinear embedding feature matrix is ​​generated using a multilayer perceptron based on the p-th level symmetric normalized adjacency matrix. in: For a multilayer perceptron, X is the initial feature matrix, which is a matrix composed of the coordinates of the feature positions.

[0044] The nonlinear embedding feature weights at each level are learned based on the channel attention mechanism, and the graph node feature information H at each node position is obtained by synthesizing the nonlinear embedding feature matrix. local Specifically:

[0045] Wherein: H cat ψ is the concatenation matrix of p-level nonlinear embedding feature matrices, ψ is the multilayer perceptron, max is the max pooling, and avg is the average pooling.

[0046] The ψ m With ψ a The structures are the same, p is 4, H local The feature dimension of each node in the graph is 64.

[0047] Step 3.2: Obtain global information of the current scene from the global pooling module: based on the graph node feature information matrix H localobtain a global feature vector of the current scene with a feature dimension of 64

[0048] wherein: θ is a multi-layer perception, θ f with the same structure as θ g .

[0049] Step 3.3, obtain geometric information H at the position of the graph node with a feature dimension of 64 by the position coding module pos =σ(X), wherein: σ is a multi-layer perception.

[0050] Step 3.4, splice the feature information H of each graph node and the geometric information H local , and append the global information h pos to the feature of each graph node to obtain a graph embedding feature H with a feature dimension of 256. global

[0051] Step 4, according to the graph embedding feature, construct an object surface point screening network and a high-value point screening network, and generate a target point graph and a high-value point graph based on a selection strategy, specifically including:

[0052] Step 4.1, construct an object surface point screening network to generate a target point graph:

[0053] a) replace the feature of each graph node in the original scene graph with the graph embedding feature H, process the graph embedding feature using a binary classifier to obtain a classification result, so as to distinguish whether each graph node belongs to the surface of the object to be grasped.

[0054] The binary classifier is implemented by using a multi-layer perception with a size of (256, 128, 64, 16, 2).

[0055] b) according to the classification result of the binary classifier, use the farthest point sampling algorithm to down-sample the graph nodes belonging to the surface of the object to be grasped into M1 points, and use a method similar to step 22 to construct a target point graph wherein: is a binary classifier, and FPS is a farthest point sampling algorithm.

[0056] The M1 is 2048.

[0057] Step 4.2, construct a high-value point screening network to generate a high-value point graph:

[0058] i) replace the feature of each graph node in the target point graph with the graph embedding feature H, process the graph embedding feature using a multi-layer residual network composed of an EdgeConv graph convolution operator, and process the output result of the multi-layer residual network using a multi-classifier to obtain the grasping value classification of each graph node.​

[0059] The multi-classifier is implemented by a multi-layer perceptron with a size of (256, 64, 32, 10), and a multi-layer residual network composed of EdgeConv graph convolution operators Wherein: is the feature of the l-th layer graph is the feature of the l-th layer graph i is the feature of the l-th layer graph r is the feature of the l-th layer graph c is a multi-layer perceptron, and * is obj or val.

[0060] ii) According to the classification result of the multi-classifier, M2 points with the highest value classification are selected, and a high-value point graph is constructed Wherein: is a multi-layer residual network, and ζ val is a multi-classifier, and Top-k is a value classification sorting and selection algorithm.

[0061] The value classification is sorted from high to low according to the value classification, if the number of remaining optional points is greater than the number of points in the current value classification, the points in the current value classification are selected; otherwise, the points in the current value classification are randomly sampled.

[0062] The M2 is 512.

[0063] In the training stage, the expected value classification of each graph node in the target point graph needs to be specified, since each graph node position may contain multiple grabbing annotations, and each grabbing annotation can represent its quality by the grabbing score, therefore, the calculation method of the expected value classification is: Wherein:

[0064] S(·) is a grabbing quality score function, and the grabbing score is usually provided by the data set; g i is all grabbing annotations at the position of the V i graph node; is a set composed of all in the target point graph; N obj is the number of graph nodes in the target point graph; and DoV is the value classification.

[0065] Step 5, according to the high-value point graph, the spatial six-degree-of-freedom grabbing candidate pose is obtained through the pose generation network, and then the best grabbing is selected for implementation, which specifically includes:

[0066] Step 5.1, processing the graph embedding features H by a multi-layer residual network composed of EdgeConv graph convolution operators, the residual network is three layers, and the size of the EdgeConv operator in each layer is (256, 256).

[0067] Step 5.2, according to the graph node features and spatial coordinates in the high-value point graph, the best approaching direction of grasping at this position is determined by a multi-classifier.

[0068] The best approaching direction is determined by dividing the continuous grasping approaching space into 300 approaching directions, and determining the best approaching direction at the current graph node position by classification.

[0069] The multi-classifier is implemented by a multi-layer perceptron with a size of (256, 256, 300, 300).

[0070] Step 5.3, in the selected approaching direction, the plane rotation angle and grasping depth are obtained by a multi-classifier, and the grasping evaluation score is obtained by regression.

[0071] Step 5.4, select the grasping implementation with the highest grasping evaluation score in the scene, and part of the results are shown in Figure 4

[0072] Through specific actual experiments, using PyTorch as the underlying deep learning framework, using DGL as the graph network framework, using NVIDIA RTX 3080 GPU (10 GB) hardware environment, using Adam optimizer, learning rate parameter 0.001, batch size 2 to run the above method, the experimental results are as follows:

[0073] For test set category one, when grasping objects that have appeared in the training set, the average accuracy under each friction coefficient is 43.33, which is improved by 15.77 compared with the technology described in Fang H S, Wang C, Gou M, et al. GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping [C]. 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 11444-11453 as a baseline; for test set category two, when grasping objects that have not appeared in the training set but are similar in shape, the average accuracy under each friction coefficient is 39.98, which is improved by 13.87 compared with the baseline method; for test set category three, when grasping completely new objects that have not appeared in the training set, the average accuracy under each friction coefficient is 14.90, which is improved by 4.35 compared with the baseline method.​

[0074] In summary, compared with the prior art, the two-stage selection strategy of the present application improves the average precision of the three categories of test sets by 10.28, 10.58 and 2.96, respectively.

[0075] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, and each implementation scheme within the scope is subject to the present application.

Claims

1. A method for generating poses of model-free objects for unstructured task scenarios, characterized in that, After generating the original scene map based on the neighboring node algorithm from the collected point cloud data, the original scene map is input into the graph feature extraction network to obtain graph embedding features that integrate multi-scale information. The graph embedding features are then input into the object surface point filtering network and the high-value point filtering network to obtain a high-value point map. The high-value point map is then input into the pose generation network to obtain spatial six-degree-of-freedom grasping candidate poses, and then the best grasping is selected for implementation. The graph feature extraction network comprises a graph relation learning module, a global pooling module, and a position encoding module. Specifically: the graph relation learning module calculates a multi-level symmetric normalized adjacency matrix based on the initial features of each graph node in the original scene graph; then, it uses a multilayer perceptron to generate a multi-level nonlinear embedding feature matrix based on the multi-level symmetric normalized adjacency matrix; finally, it learns the weights of each level of nonlinear embedding features using a channel attention mechanism, and synthesizes the nonlinear embedding feature matrix to obtain the graph node feature information of each node position; the global pooling module processes the graph node features obtained from the graph relation learning module using a multilayer perceptron and a normalized exponential function, and sums the results to obtain global feature information; the position encoding module obtains the embedded graph node position information, i.e., graph embedding features, based on the position features of each graph node in the original scene graph using a multilayer perceptron. The object surface point filtering network described above uses a multi-layer graph network composed of graph convolution operators and binary classifiers to determine whether each graph node in the graph embedding feature belongs to the surface of the object to be captured. Based on the farthest point sampling algorithm, M1 graph nodes belonging to the object surface are sampled, and the target point map is generated using the nearest neighbor algorithm. The high-value point screening network described above uses a multi-layer network composed of graph convolution operators and multiple classifiers to predict the target point map and capture the value classification of each graph node in the graph embedding features. It selects the M2 graph nodes with the highest value classification and uses the nearest neighbor algorithm to generate a high-value point map.

2. The method for generating poses of model-free objects for unstructured work scenarios according to claim 1, characterized in that, The original scene map is obtained by using a depth sensor to acquire the original point cloud of the scene to be captured, which is then preprocessed and based on a neighboring node algorithm.

3. The method for generating poses of model-free objects for unstructured work scenarios according to claim 2, characterized in that, The preprocessing includes scene region cropping and point cloud downsampling; The neighboring node algorithm specifically includes: 1) taking each point in the preprocessed point cloud as the center point and calculating the Euclidean distance to other points in the scene; 2) finding the K nearest neighboring points with the Euclidean distance to the center point, constructing directed edges from the center point to the neighboring points, and all points in the scene together constitute the original scene graph; 3) setting the initial features of each graph node and edge in the original scene graph.

4. The method for generating poses of model-free object grasping in unstructured work scenarios according to claim 1, characterized in that, The target point map is obtained in the following way: a) Original scene image The graph node features in the image are replaced with graph embedding features. A binary classifier is used to process the graph embedding features and obtain the classification results, thereby identifying whether each graph node belongs to the surface of the object to be grasped. b) Based on the classification results of the binary classifier, the graph nodes belonging to the surface of the object to be grasped are downsampled into M1 points using the farthest point sampling algorithm to construct the target point map. ,in: , It is a binary classifier, and FPS is the farthest point sampling algorithm; The binary classifier is implemented using a multilayer perceptron of size (256, 128, 64, 16, 2).

5. The method for generating poses of model-less objects for unstructured work scenarios according to claim 1, characterized in that, The high-value point map mentioned above is obtained through the following method: i) Map the target points The graph node features in the image are replaced with graph embedding features. The graph embedding features are processed by a multi-layer residual network constructed using the EdgeConv graph convolution operator. The output of the multi-layer residual network is processed by a multi-classifier to obtain the grading of the grasping value of each graph node. ii) Based on the classification results of the multi-classifier, select the M2 points with the highest value grading and construct a high-value point map. It is a multi-layer residual network. For multi-classifier, To capture value-based ranking and selection algorithms; The multi-classifier is implemented using a multilayer perceptron of size (256, 64, 32, 10) and a multilayer residual network composed of EdgeConv graph convolution operators. in: For the first Layer diagram Chinese map nodes Features It is a multilayer perceptron. For obj or val; The value classification is achieved by sorting the points from high to low according to the crawling value classification. If the number of remaining selectable points is greater than the number of points contained in the current crawling value classification, then all points in the current crawling value classification will be selected; otherwise, random sampling will be performed on the points in the current crawling value classification.

6. The method for generating poses of model-free object grasping in unstructured work scenarios according to claim 1, characterized in that, The optimal crawling method is obtained as follows: i) Based on the graph node features and spatial coordinates in the high-value point map, a multi-classifier is used to determine the best approximation direction for capturing at that location. ii) Under the selected approximation direction, the planar rotation angle and grasping depth are obtained through a multi-classifier, and the grasping evaluation score is obtained through regression. iii) Select the crawling implementation with the highest crawling score in the scenario.

7. A system for implementing the model-less object grasping pose generation method for unstructured work scenarios as described in any one of claims 1-6, characterized in that, include: The network consists of a graph feature extraction network, an object surface point filtering network, a high-value point filtering network, and a pose generation network. Among them, the graph feature extraction network obtains graph embedding features by fusing the output features of the graph relationship learning module, the global pooling module, and the position encoding module based on the initial features of each graph node in the original scene graph. The object surface point selection network uses the farthest point sampling algorithm to sample M1 graph nodes belonging to the object surface based on graph embedding features; the high-value point selection network selects the M2 graph nodes with the highest value level as grasping points based on graph embedding features; the grasping pose generation network uses neural network processing to obtain a six-degree-of-freedom grasping pose based on the grasping point position and its graph embedding features.

Images