Multi-category stacked workpiece point cloud segmentation method based on local point cloud feature enhancement mechanism

Abstract

The invention discloses a multi-category stacked workpiece point cloud segmentation method based on a local point cloud feature enhancement mechanism. Comprising the following steps: collecting point clouds of multiple types of stacked workpieces by using a visual camera, and properly preprocessing the point cloud data; marking unshielded complete workpieces and shielded workpieces in the collected point clouds in a distinguishing manner; a local point cloud feature enhancement mechanism and a double attention mechanism are introduced to improve a feature extraction module in the Point Net + + point cloud segmentation network, and a trained segmentation network is obtained; segmenting the actual multi-category stacked workpiece scene point cloud by using the trained point cloud segmentation network to obtain an unshielded workpiece point cloud; and finally, performing secondary segmentation on the unshielded workpiece point cloud output by the network by using a region growing segmentation method to obtain an unshielded single workpiece point cloud. The invention aims to provide a point cloud segmentation method based on a local point cloud feature enhancement mechanism so as to better solve the problem of point cloud segmentation of unshielded workpieces in a multi-category stacking scene.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning point cloud segmentation, and in particular to a multi-category stacked workpiece point cloud segmentation method based on a local point cloud feature enhancement mechanism. Background Art

[0002] Sorting operation is an important part of industrial production. In the early days, major manufacturers often used manual sorting to achieve orderly sorting of industrial products. Later, in order to further improve sorting efficiency and accuracy, industrial robotic arms were introduced to replace manual work. However, it is worth noting that the existing robotic arms that have been put into use are more likely to perform pre-programmed sorting of single-category, non-overlapping workpieces in specific positions, which is difficult to effectively cope with the flexible sorting needs of multi-category stacked workpieces in the future. In recent years, artificial intelligence detection technology has developed rapidly, and such technology will provide great possibilities and convenience for the intelligent and flexible sorting of industrial products in the future. Today, more and more researchers have begun to try to combine various intelligent target detection algorithms with industrial sorting, and use this to achieve accurate identification and positioning of industrial products, thereby guiding the robotic arms to perform accurate sorting. In order to realize the intelligent detection of target workpieces in multi-category stacked workpiece scenes, most companies currently adopt the method of first segmenting the two-dimensional image and then extracting the three-dimensional point cloud. That is, first use the two-dimensional target detection algorithm (such as Faster RCNN, Mask RCNN and other deep learning networks) to segment the target workpiece in the two-dimensional image of the scene, and then use the segmented image mask to intercept the target workpiece point cloud in the scene; however, the deep learning network used in this method cannot achieve pixel-level segmentation, so the interception accuracy of the target workpiece point cloud obtained needs to be further improved. As one of the important branches of the target detection algorithm, the three-dimensional point cloud segmentation method has always attracted people's attention. At present, there are two main types of common point cloud segmentation methods: point cloud segmentation methods based on pure mathematical models and geometric reasoning technology, and point cloud segmentation methods based on feature descriptors and deep learning technology. The former covers various segmentation methods based on boundaries and region growth, and mainly uses geometric information such as the boundaries or neighborhoods of point clouds to distinguish different point clouds; the latter covers segmentation methods such as PointNet, PointNet++, and RandLA-Net, and mainly uses feature descriptors to directly extract point cloud features from point clouds, and uses deep learning networks to distinguish the categories of each point in the point cloud. For multi-category stacked workpiece scene point clouds, if the first type of method is used to segment them, it often requires a large amount of repeated empirical algorithm parameter adjustment, which is time-consuming and laborious; if the second type of segmentation method is used directly, its point cloud segmentation effect is largely limited by the feature extraction method of the deep learning network. However, compared with the first type of segmentation method, as long as the feature extraction method of the deep learning network is properly designed, the second type of point cloud segmentation method will be more suitable for target detection in various complex scenes and have broader application prospects. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a point cloud segmentation method for multi-category stacked workpieces based on a local point cloud feature enhancement mechanism, which effectively solves the problem of fast and accurate detection of unobstructed target workpieces in multi-category stacked workpiece scenes.

[0004] To achieve the above object, the present invention adopts the following technical solution: a multi-category stacked workpiece point cloud segmentation method based on a local point cloud feature enhancement mechanism, comprising the following steps:

[0005] Step S1: using a visual camera to collect scene point clouds of multiple categories of stacked workpieces;

[0006] Step S2: performing appropriate point cloud preprocessing on the scene point cloud;

[0007] Step S3: Producing a point cloud segmentation dataset of multi-category stacked workpieces;

[0008] Step S4: design an improved point cloud segmentation network based on local point cloud feature enhancement mechanism and dual attention mechanism;

[0009] Step S5: putting the scene point clouds of multiple types of stacked workpieces collected by the visual camera into the improved point cloud segmentation network for reasoning;

[0010] Step S6: Use region growing segmentation to perform secondary segmentation on the multiple unobstructed workpiece point clouds output by the segmentation network to obtain unobstructed single workpiece point clouds.

[0011] In a preferred embodiment, in step S1, a visual camera is used to collect scene point clouds of multiple categories of stacked workpieces in different placement postures and stacking orders multiple times, and these scene point clouds should contain the three-dimensional spatial coordinate information of each point in the point cloud, namely X, Y, Z and three color information, namely R, B, G.

[0012] In a preferred embodiment, the specific implementation process of step S2 is: on the premise of ensuring that the characteristics and structure of the scene point cloud obtained in step S1 are not lost, the scene point cloud is preprocessed by using threshold segmentation, noise reduction, downsampling and other methods.

[0013] In a preferred embodiment, the specific implementation process of step S3 is:

[0014] Select each unobstructed complete workpiece point cloud in the scene point cloud obtained in step S2 one by one, and use different numbers to distinguish and label the individual workpiece point clouds of different categories in these workpiece point clouds, that is, the unobstructed first-category workpiece is labeled as "1", the second-category workpiece is labeled as "2", and so on; all the obstructed incomplete workpiece point clouds in the scene point cloud obtained in step S2 are also labeled as "-1" to distinguish the different categories of workpiece point clouds and all the obstructed incomplete workpiece point clouds in the unobstructed workpiece point cloud in the scene point cloud; uniformly downsample the aforementioned preprocessed scene point cloud to N points to obtain a group of downsampled point clouds, and at the same time remove these N points from the original scene point cloud to obtain a scene point cloud after removing N points; then repeat the aforementioned similar uniform downsampling and elimination operations on the scene point cloud after removing N points, and finally obtain multiple groups of downsampled point clouds, which are used as training data sets for the deep learning network.

[0015] In a preferred embodiment, the specific implementation process of step S4 is:

[0016] Step S41: Reconstruct a 4-layer feature extraction module for the PointNet++ point cloud segmentation network;

[0017] Step S42: Introduce the self-attention mechanism and channel attention mechanism into SA-k to make the deep learning network pay more attention to the local key features of the point cloud; at the same time, add a residual structure to the self-attention mechanism to reduce the risk of network overfitting;

[0018] Step S43: retain the spatial information decoding module structure of the PointNet++ point cloud segmentation network, and design a 4-layer spatial information decoding module FP to improve the segmentation network; restore the density of the point cloud by using the KNN-based inverse distance interpolation average method, and the interpolation feature of each point is obtained by interpolating the features of its neighboring points; then, concatenate these interpolation features with the output features of the feature extraction module, and finally use MLP for dimensionality reduction;

[0019] Step S44: construct a network structure between the 4-layer feature extraction module and the 4-layer spatial information decoding layer.

[0020] In a preferred embodiment, the feature extraction method of the kth, k=1, 2, 3, 4th layer feature extraction module SA-k is designed as follows:

[0021] Step S411: The input of the kth, k=1, 2, 3, 4th layer feature extraction module SA-k is the point cloud coordinate set X k ={ k x 11 , k x 12 ,…, k x 1W} and its point cloud feature set in, k x 1i , Represents the X, Y, Z three-dimensional coordinate information of the i-th, i=1,2,3,...,W-th point in the input point cloud and its corresponding k u-dimensional feature information, W represents the total number of points contained in the input point cloud;

[0022] Step S412: Use the farthest point sampling method FPS to sample the input value X k Sample w, w<<W points, and use the point cloud coordinate set corresponding to these w points as the output point cloud coordinate set of SA-k. This point cloud coordinate set will also be used as the input point cloud coordinate set of the k+1th layer feature extraction module, denoted by X k+1 ={ k+1 x 21 , k+1 x 22 ,…, k+1 x 2w};in, k+1 x 2i Represents the X, Y, Z three-dimensional coordinate information of the i-th, i=1, 2, 3, ..., w-th point in the point cloud;

[0023] Step S413: For each of the w points obtained in step S412, the K-nearest neighbor algorithm KNN is used to select the 32 points closest to the neighborhood radius s, and the point cloud coordinates of the obtained 32×w points are set to k F 0 ={ k x 31 , k x 32 ,…, k x 3w} as the neighborhood feature of SA-k; among them, k x 3i It means that the i, i=1,2,3,...,w points of the point cloud are taken as the center point of the algorithm KNN, and the three-dimensional coordinate information of the 32 points within the neighborhood radius s is obtained; at the same time, the 32×w points in X' k The corresponding feature in is used as another neighborhood feature of SA-k in It means that the i-th point in the point cloud, i=1,2,3,...,w, is the center point of the algorithm KNN, and the 32 points within its neighborhood radius s k u-dimensional feature information;

[0024] Step S414: Subtract the coordinates of the center point from the point cloud coordinates of the 32×w points obtained in step S413 to obtain the enhanced local feature set in the SA-k neighborhood k F 2 ={k x 41 , k x 42 ,…, k x 4w};in k x 4i It means that the coordinate difference information is obtained by subtracting the coordinates of the center point from the coordinates of the 32 points in the neighborhood radius s, with the i-th point of the point cloud, i=1,2,3,...,w, as the center point of the algorithm KNN;

[0025] Step S415: The above neighborhood feature set k F 0 , k F 1 and enhanced local feature set k F 2 Reasonable splicing is performed, and the total features of SA-k are obtained through deep learning network multi-layer perceptron MLP dimensionality upgrade. k r, that is

[0026]

[0027] in, It is a feature splicing symbol.

[0028] In a preferred embodiment, in step 42,

[0029] Step 421: Introduce self-attention mechanism: k r After the self-attention mechanism and channel attention mechanism MLP, the query matrix of the self-attention mechanism will be obtained respectively k Q, key matrix k K and value matrix k V; then the query matrix k Q and the bond matrix k The correlation matrix between features is calculated by multiplying the transpose of K. Then the correlation matrix is ​​normalized by the softmax function. Finally, the normalized correlation matrix is ​​combined with the value matrix k V is multiplied, and a residual structure is used in the mechanism to obtain the output feature of the self-attention mechanism k s, that is

[0030] k s= k V×softmax( k Q×( k K) T )+ k r

[0031] Step 422: Introduce a channel attention mechanism; in the channel attention mechanism, the maximum pooling operation and the average pooling operation are organically combined to obtain the output feature set X' of the kth, k=1, 2, 3, 4th layer feature extraction module k+1 ,Right now

[0032] X' k+1 = k T max + k T avg

[0033] in, k T max =MLP(MAXPOOL( k s))、 k T avg =MLP(AVGPOOL( k s)) are the features after maximum pooling and average pooling in SA-k, respectively; MAXPOOL is the maximum pooling operation, and AVGPOOL is the average pooling operation;

[0034] In a preferred embodiment, step 44 includes the following steps:

[0035] Step 441: For the first-layer feature extraction module SA-1, input the point cloud dataset X after downsampling in step S3 1 ={ 1 x 11 , 1 x 12 ,…, 1 x 1N}and in, 1 x 1i , Respectively represent the 3D coordinate information of the i-th, i=1,2,3,...,N points in the point cloud, namely X, Y, Z and 1 u = 6-dimensional feature information, namely X, Y, Z, R, B, G; after passing through the feature extraction module SA-1, we get X 2 ={ 2 x 11 , 2 x 12 ,…, 2 x 1n}and in, 2 x 1i , It means that the X, Y, Z three-dimensional coordinate information of the i-th, i=1,2,3,...,n-th point in the point cloud is obtained by FPS in SA-1 and its dimension is obtained after MLP in SA-1. 2 High-dimensional features with u=64;

[0036] Step 442: For the second-layer feature extraction module SA-2, input X 2 and X 2 ', after the feature extraction module SA-2 obtains X 3 ={ 3 x 11 , 3 x 12 ,…, 3 x 1α}and in, 3 x 1i , It means that the X, Y, Z three-dimensional coordinate information of the i-th (i=1, 2, 3, …, α) point in the point cloud is obtained by FPS in SA-2 and its dimension is obtained by MLP in SA-2. 3 High-dimensional features with u=128;

[0037] Step 443: For the third layer feature extraction module SA-3, input X 3 and X 3 ', after the feature extraction module SA-3, we get X 4 ={ 4 x 11 , 4 x 12 ,…, 4 x 1β}and in, 4 x 1i , It means that the X, Y, Z three-dimensional coordinate information of the i-th point (i=1, 2, 3, ..., β) in the point cloud is obtained by FPS in SA-3 and its dimension is obtained by MLP in SA-3. 4 High-dimensional features with u=256;

[0038] Step 444: For the 4th layer feature extraction module SA-4, input X 4 and X' 4 , obtained through feature extraction module SA-4 5 ={ a x 11 , a x 12 ,…, a x 1γ}and in, a x 1i , Indicates the X, Y, Z three-dimensional coordinate information of the i-th (i=1, 2, 3, ..., γ) point in the point cloud obtained by FPS in SA-4 and its high-dimensional features of 512 dimensions obtained by MLP in SA-4;

[0039] Step 445: For the first spatial information decoding layer block FP-1, input X' 4 and X 5 ', after the inverse distance interpolation averaging method and jump connection in the spatial information decoding layer FP-1, and dimensionality reduction is obtained in, It means that the i-th point in the point cloud (i=1,2,3,…,β) is reduced in dimension by MLP in FP-1 to obtain a high-dimensional feature of 256 dimensions;

[0040] Step 446: For the second spatial information decoding layer block FP-2, input X 3 ' and P 1 ', after the inverse distance interpolation averaging method and jump connection in the spatial information decoding layer FP-2, and dimensionality reduction is obtained in, It means that the i-th point in the point cloud (i = 1, 2, 3, ..., α) is reduced to a high-dimensional feature of 256 dimensions by MLP in FP-2;

[0041] Step 447: For the third spatial information decoding layer block FP-3, input X 2 ' and P 2 ', after the inverse distance interpolation averaging method and jump connection in the spatial information decoding layer FP-3, and dimensionality reduction is obtained in, It means that the i-th point in the point cloud (i=1,2,3,…,n) is reduced to a high-dimensional feature of 128 dimensions by MLP in FP-3;

[0042] Step 448: For the 4th spatial information decoding layer block FP-4, input P 3 ', obtained by the inverse distance interpolation averaging method in the spatial information decoding layer FP-4 in, It means that the i-th point in the point cloud, i=1,2,3,…,N, is passed through the MLP in FP-4 to obtain a high-dimensional feature of 128 dimensions;

[0043] Step 449: Use the fully connected layer to 4 'Perform dimensionality reduction so that the final dimension obtained by dimensionality reduction is equal to the number of segmentation categories, and give the segmentation result in, It means that the i-th point in the point cloud (i=1,2,3,…,N) passes through the fully connected layer to obtain the category prediction value of the point.

[0044] In a preferred embodiment, the specific implementation process of step S5 is: putting the segmentation data set produced in step S3 into the point cloud segmentation network based on the enhanced local point cloud feature enhancement mechanism in step S4 for training to obtain an inference model; then, using a visual camera to collect point clouds of multiple types of stacked workpieces in a real scene, and inputting them into the trained inference model for inference, thereby obtaining an unobstructed workpiece point cloud.

[0045] In a preferred embodiment, the specific steps of step S6 are: using the region growing segmentation method to perform a secondary simple segmentation on the multiple unobstructed workpieces output by the network in step S5; using the region growing segmentation method to perform a secondary segmentation on the unobstructed workpiece point cloud in multiple types of stacked workpieces, so as to easily obtain the point cloud of a single workpiece.

[0046] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a method for point cloud segmentation of multi-category stacked workpieces based on a local point cloud feature enhancement mechanism for scenarios of multi-category stacked workpieces in industry. Compared with point cloud segmentation methods based on pure mathematical models and geometric reasoning techniques, this method avoids a large number of repeated empirical algorithm parameter adjustments. It only needs to make the stacked workpieces into a data set and perform network training, and it can be applied to various complex scenarios. Compared with the method based on two-dimensional image segmentation and using its segmentation mask to intercept three-dimensional point clouds, the point cloud data itself contains richer information than the two-dimensional image. Therefore, the segmentation of stacked workpieces can achieve pixel-level accuracy, which improves the accuracy of point cloud segmentation of stacked workpieces. In addition, in addition to introducing a local point cloud feature enhancement mechanism based on the PointNet++ point cloud segmentation network, the present invention effectively improves the category recognition accuracy of unobstructed workpiece point clouds, and also uses a dual attention mechanism to enable the network to focus on local key features in the point cloud, further enhancing the point cloud segmentation performance of the present invention in scenarios of multi-category stacked workpieces. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a stacked workpiece point cloud segmentation process provided by a preferred embodiment of the present invention;

[0048] Figure 2 A schematic diagram of the improved point cloud segmentation network feature extraction process provided by the preferred embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the dual attention mechanism structure of the improved point cloud segmentation network provided by the preferred embodiment of the present invention;

[0050] Figure 4 A schematic diagram of an improved point cloud segmentation network structure provided by a preferred embodiment of the present invention;

[0051] Figure 5A data annotation model diagram of a scene point cloud provided by a preferred embodiment of the present invention;

[0052] Figure 6 This is a comparison diagram of the effects of the original scene point cloud, PointNet++ network and improved point cloud segmentation network used for testing provided in the preferred embodiment of the present invention, where a is the preprocessed scene point cloud image, b is the PointNet++ point cloud segmentation effect image, and c is the improved point cloud segmentation effect image. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0054] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0055] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0056] The present invention provides a multi-category stacked workpiece point cloud segmentation method based on a local point cloud feature enhancement mechanism, such as Figure 1 The figure is a schematic diagram of the stacked workpiece point cloud segmentation process provided by an embodiment of the present invention, comprising the following steps:

[0057] Step S1: Use a visual camera to collect scene point clouds of multiple categories of stacked workpieces.

[0058] Step S2: Perform appropriate point cloud preprocessing on the scene point cloud.

[0059] Step S3: Produce a point cloud segmentation dataset of multi-category stacked artifacts.

[0060] Step S4: Design an improved point cloud segmentation network based on local point cloud feature enhancement mechanism and dual attention mechanism.

[0061] Step S5: Put the scene point cloud of multiple types of stacked workpieces collected by the visual camera into the improved point cloud segmentation network for reasoning.

[0062] Step S6: Use region growing segmentation to perform secondary segmentation on the multiple unobstructed workpieces output by the segmentation network to obtain an unobstructed single workpiece point cloud.

[0063] In one embodiment of the present invention, in step S1, a visual camera is used to collect scene point clouds of multiple categories of stacked workpieces in different placement postures and stacking orders for multiple times, and these scene point clouds should contain three-dimensional spatial coordinate information (i.e., X, Y, Z) and three color information (i.e., R, B, G) of each point in the point cloud.

[0064] In one embodiment of the present invention, the specific implementation process of step S2 is:

[0065] In order to improve the segmentation efficiency and accuracy of the subsequent point cloud segmentation network, on the premise of ensuring that the characteristics and structure of the scene point cloud obtained in step 1 are not lost, the scene point cloud is preprocessed using threshold segmentation, denoising, downsampling and other methods.

[0066] In one embodiment of the present invention, the specific implementation process of step S3 is:

[0067] Select each unobstructed complete workpiece point cloud in the scene point cloud obtained in step 2 one by one, and use different numbers to distinguish and label the individual workpiece point clouds of different categories in these workpiece point clouds (such as the unobstructed first category of workpieces are labeled as "1", the second category of workpieces are labeled as "2", and so on); in addition, all the obstructed incomplete point cloud workpieces in the scene point cloud obtained in step 2 need to be labeled as "-1" to distinguish the different categories of point cloud workpieces in the unobstructed point cloud workpieces in the scene point cloud and all the obstructed incomplete point cloud workpieces (such as Figure 5 The figure shows an example of a scene point cloud that has been labeled. The occluded incomplete point cloud is labeled as "-1" and is represented by blue; the unoccluded complete circular workpiece point cloud is labeled as "1" and is represented by red; the unoccluded complete star-shaped workpiece point cloud is labeled as "2" and is represented by purple; the unoccluded complete square workpiece point cloud is labeled as "3" and is represented by yellow). In order to improve the training speed of the subsequent deep learning network, the scene point cloud after the above preprocessing is uniformly downsampled to 8192 points to obtain a group of downsampled point clouds, and at the same time, these 8192 points are removed from the original scene point cloud to obtain the scene point cloud after removing 8192 points; the above-mentioned similar uniform downsampling and elimination operations are continuously repeated on the scene point cloud after removing 8192 points, and finally multiple groups of downsampled point clouds are obtained, which are used as the training data set for the deep learning network.

[0068] In one embodiment of the present invention, the specific implementation process of step S4 is:

[0069] S4.1: Reconstruct a 4-layer feature extraction module for the PointNet++ point cloud segmentation network, where the feature extraction method of the k-th (k = 1, 2, 3, 4) layer feature extraction module SA-k is as follows Figure 2As shown, it is a schematic diagram of the improved point cloud segmentation network feature extraction process provided by an embodiment of the present invention, and the specific steps are designed as follows:

[0070] S4.1.1: The input of the k-th (k=1, 2, 3, 4) layer feature extraction module SA-k is the point cloud coordinate set X k ={ k x 11 , k x 12 ,…, k x 1W} and its point cloud feature set in, k x 1i , Represents the (X, Y, Z) three-dimensional coordinate information of the i-th (i = 1, 2, 3, ..., W) point in the input point cloud and its corresponding k u-dimensional feature information, W represents the total number of points contained in the input point cloud.

[0071] S4.1.2: Use the farthest point sampling method (FPS) to sample the input quantity X k Sample w (w < < W) points, and use the point cloud coordinate set corresponding to these w points as the output point cloud coordinate set of SA-k. This point cloud coordinate set will also be used as the input point cloud coordinate set of the k+1th layer feature extraction module, denoted by X k+1 ={ k+1 x 21 , k+1 x 22 ,…, k+1 x 2w}.in, k+1 x 2i Represents the (X, Y, Z) three-dimensional coordinate information of the i-th (i=1, 2, 3, …, w) point in the point cloud.

[0072] S4.1.3: For each of the w points obtained in S4.1.2, use the K-nearest neighbor (KNN) algorithm to select the 32 points closest to it within its neighborhood radius s, and then put the point cloud coordinates of the resulting 32×w points into a set k F 0 ={ k x 31 , k x 32 ,…, k x 3w} as the neighborhood feature of SA-k; among them, k x 3i Indicates that the i-th (i=1,2,3,…,w) point of the point cloud is taken as the center point of the KNN algorithm, and the three-dimensional coordinate information of the 32 points within its neighborhood radius s is obtained. At the same time, the 32×w points in X' kThe corresponding feature in is used as another neighborhood feature of SA-k in It means that the i-th point (i=1,2,3,…,w) of the point cloud is the center point of the KNN algorithm, and the 32 points within its neighborhood radius s k u-dimensional feature information.

[0073] S4.1.4: In order to further enhance the local features of the segmentation network and improve the accuracy of the network in segmenting different types of workpieces, the point cloud coordinates of the 32×w points obtained in S4.1.3 are subtracted from the coordinates of their center points to obtain the enhanced local feature set in the SA-k neighborhood. k F 2 ={ k x 41 , k x 42 ,…, k x 4w}.in k x 4i It means that the i-th (i=1, 2, 3, …, w) point of the point cloud is taken as the center point of the KNN algorithm, and the coordinate difference information is obtained by subtracting the coordinates of the center point from the coordinates of the 32 points within the neighborhood radius s.

[0074] S4.1.5: The above neighborhood feature set k F 0 , k F 1 and enhanced local feature set k F 2 After reasonable splicing and dimensionality upgrading through the deep learning network multi-layer perceptron (MLP), the total features of SA-k are obtained. k r, that is

[0075]

[0076] in, It is a feature splicing symbol.

[0077] S4.2: Introduce self-attention mechanism and channel attention mechanism in SA-k to make the deep learning network pay more attention to the local key features of the point cloud; at the same time, add a residual structure to the self-attention mechanism to reduce the risk of network overfitting. Figure 3 As shown, it is a schematic diagram of the dual attention mechanism structure of the improved point cloud segmentation network provided by an embodiment of the present invention, and the specific steps are as follows:

[0078] S4.2.1: Introducing the self-attention mechanism: k r passes through MLP (in the self-attention mechanism and channel attention mechanism, MLP does not perform dimension increase or decrease operations, but only plays a role in nonlinear transformation of features) to obtain the query matrix of the self-attention mechanism respectivelyk Q, key matrix k K and value matrix k V; then the query matrix k Q and the bond matrix k The correlation matrix between features is calculated by multiplying the transpose of K. Then the correlation matrix is ​​normalized by the softmax function. Finally, the normalized correlation matrix is ​​combined with the value matrix k V and use the residual structure in the self-attention mechanism to obtain the output features of the self-attention mechanism k s, that is

[0079] k s= k V×softmax( k Q×( k K) T )+ k r

[0080] S4.2.2: Introducing channel attention mechanism: Since the PointNet++ network only uses maximum pooling to aggregate the features of the point cloud into global features, this will inevitably cause a certain loss of the local features of the point cloud; for this reason, the present invention organically combines the maximum pooling operation and the average pooling operation in the channel attention mechanism to better retain the local features of the point cloud during the maximum pooling aggregation, and finally obtains the output feature set X' of the kth (k = 1, 2, 3, 4) layer feature extraction module k+1 ,Right now

[0081] X' k+1 = k T max + k T avg

[0082] in, k T max =MLP(MAXPOOL( k s))、 k T avg =MLP(AVGPOOL( k s)) are the features of maximum pooling and average pooling in SA-k respectively; MAXPOOL is the maximum pooling operation, and AVGPOOL is the average pooling operation.

[0083] S4.3: Retain and use the spatial information decoding layer of the PointNet++ point cloud segmentation network. In the PointNet++ point cloud segmentation network, the spatial information decoding layer passes the features learned by the feature extraction module (FP) back to the original point cloud. In this process, distance-based interpolation algorithms and feature jump connections are mainly used. Specifically, the density of the point cloud is restored by the KNN-based inverse distance interpolation averaging method, and the interpolated features of each point are obtained by interpolating the features of its neighboring points. Then, these interpolated features are concatenated with the output features of the feature extraction module, and finally MLP is used for dimensionality reduction.

[0084] S4.4: Construct a specific network structure between the 4-layer feature extraction module and the 4-layer spatial information decoding layer. Figure 4 As shown, it is a schematic diagram of the improved point cloud segmentation network structure provided by an embodiment of the present invention, and the specific steps are as follows:

[0085] S4.4.1: The first-layer feature extraction module SA-1 takes as input the point cloud dataset X after downsampling in step S3 1 ={ 1 x 11 , 1 x 12 ,…, 1 x 1N}and Among them, N = 8192, 1 x 1i , Represents the 3D coordinate information (i.e., X, Y, Z) and 6D feature information (i.e., X, Y, Z, R, B, G) of the i-th (i=1, 2, 3, ..., 8192) point in the point cloud. After the feature extraction module SA-1, X 2 ={ 2 x 11 , 2 x 12 ,…, 2 x 1n}and Where n = 1024, 2 x 1i , It means that the (X, Y, Z) three-dimensional coordinate information of the i-th (i=1, 2, 3, ..., 1024) point in the point cloud is obtained by the FPS in SA-1 and its dimension is obtained by the MLP in SA-1. 1 High dimensional features with u=64.

[0086] S4.4.2: Layer 2 feature extraction module SA-2, input X 2 and X 2 'Get X through the feature extraction module 3 ={3 x 11 , 3 x 12 ,…, 3 x 1α}and Among them, α=256, 3 x 1i , It represents the (X, Y, Z) three-dimensional coordinate information of the i-th (i=1, 2, 3, ..., 256) point in the point cloud obtained by FPS in SA-2 and its dimension obtained by MLP in SA-2 is 2 High-dimensional features with u=128.

[0087] S4.4.3: Layer 3 feature extraction module SA-3, input X 3 and X 3 ', after the feature extraction module, we get X 4 ={ 4 x 11 , 4 x 12 ,…, 4 x 1β}and Among them, β = 64, 4 x 1i , It represents the (X, Y, Z) three-dimensional coordinate information of the i-th (i=1, 2, 3, ..., 64) point in the point cloud obtained by FPS in SA-3 and its dimension obtained by MLP in SA-3 is 3 High-dimensional features with u=256.

[0088] S4.4.4: Layer 4 feature extraction module SA-4, input X 4 and X' 4 , obtained through the feature extraction module 5 ={ a x 11 , a x 12 ,…, a x 1γ}and Among them, γ=16, a x 1i , It represents the (X, Y, Z) three-dimensional coordinate information of the i-th (i=1, 2, 3, ..., 16) point in the point cloud obtained by FPS in SA-4 and its dimension obtained by MLP in SA-4 is 4 High dimensional features with u=512.

[0089] S4.4.5: 1st layer spatial information decoding layer block FP-1, input X'4 and X 5 ', after the inverse distance interpolation averaging method and skip connection in the spatial information decoding layer, and dimensionality reduction, we get Among them, β=64, It means that the i-th (i=1, 2, 3, ..., 64) point in the point cloud is reduced in dimension by MLP in FP-1 to obtain a high-dimensional feature of 256 dimensions.

[0090] S4.4.6: Layer 2 spatial information decoding layer block FP-2, input X 3 ' and P 1 ', after the inverse distance interpolation averaging method and skip connection in the spatial information decoding layer, and dimensionality reduction, we get Among them, α=256, It means that the i-th (i=1,2,3,…,256) point in the point cloud is reduced in dimension by MLP in FP-2 to obtain a high-dimensional feature of 256 dimensions.

[0091] S4.4.7: Layer 3 spatial information decoding block FP-3, input X 2 ' and P 2 ', after the inverse distance interpolation averaging method and skip connection in the spatial information decoding layer, and dimensionality reduction, we get Where n = 1024, It means that the i-th point (i=1, 2, 3, ..., 1024) in the point cloud is reduced in dimension by MLP in FP-3 to obtain a high-dimensional feature of 128 dimensions.

[0092] S4.4.8: FP-4, input for the 4th spatial information decoding layer block The inverse distance interpolation averaging method in the spatial information decoding layer is used to obtain Among them, N = 8192, It means that the i-th (i=1, 2, 3, ..., 8192) point in the point cloud passes through the MLP in FP-4 to obtain a high-dimensional feature of 128 dimensions.

[0093] S4.4.9: Use fully connected layers to 4 'Perform dimensionality reduction so that the final dimension obtained by dimensionality reduction is equal to the number of segmentation categories, and give the segmentation result Among them, N = 8192, It means that the i-th (i=1,2,3,…,8192) point in the point cloud passes through the fully connected layer to obtain the category prediction value of the point.

[0094] In one embodiment of the present invention, the specific implementation process of step S5 is as follows:

[0095] The segmentation data set prepared in step S3 is put into the point cloud segmentation network based on the enhanced local point cloud feature enhancement mechanism in step S4 for training to obtain an inference model. Then, a visual camera is used to collect point clouds of multiple types of stacked workpieces in real scenes, and the point clouds are input into the trained inference model for inference to obtain unobstructed workpiece point clouds. Figure 6 As shown, it is a comparison diagram of the segmentation effects of the original scene point cloud used for testing provided in an embodiment of the present invention, the PointNet++ network-based point cloud segmentation network and the improved point cloud segmentation network (a is the preprocessed point cloud image, b is the PointNet++ point cloud segmentation effect image, and c is the improved point cloud segmentation effect image).

[0096] In one embodiment of the present invention, the specific steps of step S6 are:

[0097] The region growing segmentation method is used to perform a secondary simple segmentation on the multiple unobstructed workpieces output by the network. Although the unobstructed workpiece point cloud in multiple stacked workpieces can be obtained through the above deep learning segmentation network, only one of the unobstructed workpiece point clouds is needed in the actual robot arm grasping. Therefore, the region growing segmentation method is used to perform a secondary segmentation on the unobstructed workpiece point cloud in multiple types of stacked workpieces, so as to obtain the point cloud of a single workpiece.

[0098] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A multi-category stacked workpiece point cloud segmentation method based on local point cloud feature enhancement mechanism, characterized in that: The steps include: Step S1: using a visual camera to collect scene point clouds of multiple categories of stacked workpieces; Step S2: performing appropriate point cloud preprocessing on the scene point cloud; Step S3: Producing a point cloud segmentation dataset of multi-category stacked workpieces; Step S4: design an improved point cloud segmentation network based on local point cloud feature enhancement mechanism and dual attention mechanism; Step S5: putting the scene point clouds of multiple types of stacked workpieces collected by the visual camera into the improved point cloud segmentation network for reasoning; Step S6: Use region growing segmentation to perform secondary segmentation on the multiple unobstructed workpiece point clouds output by the segmentation network to obtain unobstructed single workpiece point clouds.

2. According to the method of claim 1, a multi-category stacked workpiece point cloud segmentation method based on a local point cloud feature enhancement mechanism is characterized in that: In step S1, a visual camera is used to collect scene point clouds of multiple categories of stacked workpieces in different placement postures and stacking orders for multiple times, and these scene point clouds should contain the three-dimensional spatial coordinate information of each point in the point cloud, namely X, Y, Z and three color information, namely R, B, G.

3. The multi-category stacked workpiece point cloud segmentation method based on local point cloud feature enhancement mechanism according to claim 1 is characterized in that: The specific implementation process of step S2 is: on the premise of ensuring that the features and structures of the scene point cloud obtained in step S1 are not lost, the scene point cloud is preprocessed by using threshold segmentation, noise reduction, downsampling and other methods.

4. The multi-category stacked workpiece point cloud segmentation method based on local point cloud feature enhancement mechanism according to claim 1, characterized in that: The specific implementation process of step S3 is: Select each unobstructed complete workpiece point cloud in the scene point cloud obtained in step S2 one by one, and use different numbers to distinguish and label the different categories of individual workpiece point clouds in these workpiece point clouds, that is, the first category of unobstructed workpieces are labeled as "1", the second category of workpieces are labeled as "2", and so on; all the obstructed incomplete workpiece point clouds in the scene point cloud obtained in step S2 are also labeled as "-1" to distinguish different categories of workpiece point clouds in the unobstructed workpiece point cloud in the scene point cloud and all obstructed incomplete workpiece point clouds; The scene point cloud after the above preprocessing is uniformly downsampled to N points to obtain a group of downsampled point clouds, and at the same time, these N points are removed from the original scene point cloud to obtain the scene point cloud after removing N points; then the above-mentioned similar uniform downsampling and elimination operations are continuously repeated on the scene point cloud after removing N points, and finally multiple groups of downsampled point clouds are obtained, which are used as training data sets for the deep learning network.

5. The multi-category stacked workpiece point cloud segmentation method based on local point cloud feature enhancement mechanism according to claim 1, characterized in that: The specific implementation process of step S4 is as follows: Step S41: Reconstruct a 4-layer feature extraction module for the PointNet++ point cloud segmentation network; Step S42: Introduce the self-attention mechanism and channel attention mechanism into SA-k to make the deep learning network pay more attention to the local key features of the point cloud; at the same time, add a residual structure to the self-attention mechanism to reduce the risk of network overfitting; Step S43: retain the spatial information decoding module structure of the PointNet++ point cloud segmentation network, and design a 4-layer spatial information decoding module FP to improve the segmentation network; restore the density of the point cloud by using the KNN-based inverse distance interpolation average method, and the interpolation feature of each point is obtained by interpolating the features of its neighboring points; Then, these interpolated features are concatenated with the output features of the feature extraction module, and finally MLP is used for dimensionality reduction; Step S44: construct a network structure between the 4-layer feature extraction module and the 4-layer spatial information decoding layer.

6. The method for segmenting multi-category stacked workpiece point clouds based on a local point cloud feature enhancement mechanism according to claim 5, characterized in that: In step 41, the feature extraction method of the kth, k=1, 2, 3, 4th layer feature extraction module SA-k is designed as follows: Step S411: The input of the kth, k=1, 2, 3, 4th layer feature extraction module SA-k is the point cloud coordinate set X k ={ k x 11 , k x 12 ,…, k x 1W } and its point cloud feature set in, k x 1i , Represents the X, Y, Z three-dimensional coordinate information of the i-th, i=1,2,3,...,W-th point in the input point cloud and its corresponding k u-dimensional feature information, W represents the total number of points contained in the input point cloud; Step S412: Use the farthest point sampling method FPS to sample the input value X k Sample w, w<<W points, and use the point cloud coordinate set corresponding to these w points as the output point cloud coordinate set of SA-k. This point cloud coordinate set will also be used as the input point cloud coordinate set of the k+1th layer feature extraction module, denoted by X k+1 ={ k+1 x 21 , k+1 x 22 ,…, k+1 x 2w };in, k+1 x 2i Represents the X, Y, Z three-dimensional coordinate information of the i-th, i=1, 2, 3, ..., w-th point in the point cloud; Step S413: For each of the w points obtained in step S412, the K-nearest neighbor algorithm KNN is used to select the 32 points closest to the neighborhood radius s, and the point cloud coordinates of the obtained 32×w points are set to k F0={ k x 31 , k x 32 ,…, k x 3w } as the neighborhood feature of SA-k; among them, k x 3i It means that the i, i=1,2,3,...,w points of the point cloud are taken as the center point of the algorithm KNN, and the three-dimensional coordinate information of the 32 points within the neighborhood radius s is obtained; at the same time, the 32×w points in X' k The corresponding feature in is used as another neighborhood feature of SA-k in It means that the i-th point in the point cloud, i=1,2,3,...,w, is the center point of the algorithm KNN, and the 32 points within its neighborhood radius s k u-dimensional feature information; Step S414: Subtract the coordinates of the center point from the point cloud coordinates of the 32×w points obtained in step S413 to obtain the enhanced local feature set in the SA-k neighborhood k F2={ k x 41 , k x 42 ,…, k x 4w };in k x 4i It means that the coordinate difference information is obtained by subtracting the coordinates of the center point from the coordinates of the 32 points in the neighborhood radius s, with the i-th point of the point cloud, i=1,2,3,...,w, as the center point of the algorithm KNN; Step S415: The above neighborhood feature set k F0, k F1 and enhanced local feature set k F2 is reasonably spliced ​​and dimensionalized by the deep learning network multi-layer perceptron MLP to obtain the total features of SA-k k r, that is in, It is a feature splicing symbol.

7. The method for segmenting multi-category stacked workpiece point clouds based on a local point cloud feature enhancement mechanism according to claim 5, characterized in that: In step 42, Step 421: Introduce self-attention mechanism: k r After the self-attention mechanism and channel attention mechanism MLP, the query matrix of the self-attention mechanism will be obtained respectively k Q, key matrix k K and value matrix k V; then the query matrix k Q and the bond matrix k The correlation matrix between features is calculated by multiplying the transpose of K. Then the correlation matrix is ​​normalized by the softmax function. Finally, the normalized correlation matrix is ​​combined with the value matrix k V is multiplied, and a residual structure is used in the mechanism to obtain the output feature of the self-attention mechanism k s, that is k s= k V×softmax( k Q×( k K) T )+ k r Step 422: Introduce a channel attention mechanism; in the channel attention mechanism, the maximum pooling operation and the average pooling operation are organically combined to obtain the output feature set X' of the kth, k=1, 2, 3, 4th layer feature extraction module k+1 ,Right now X' k+1 = k T max + k T avg in, k T max =MLP(MAXPOOL( k s))、 k T avg =MLP(AVGPOOL( k s)) are the features after maximum pooling and average pooling in SA-k, respectively; MAXPOOL is the maximum pooling operation, and AVGPOOL is the average pooling operation.

8. The method for segmenting multi-category stacked workpiece point clouds based on a local point cloud feature enhancement mechanism according to claim 5, characterized in that: Step 44 includes the following steps: Step 441: For the first-layer feature extraction module SA-1, input the point cloud data set X1 after downsampling in step S3 = { 1 x 11 , 1 x 12 ,…, 1 x 1N }and in, 1 x 1i , Respectively represent the 3D coordinate information of the i-th, i=1,2,3,...,N points in the point cloud, namely X, Y, Z and 1 u = 6-dimensional feature information, namely X, Y, Z, R, B, G; after the feature extraction module SA-1, X2 = { 2 x 11 , 2 x 12 ,…, 2 x 1n }and in, 2 x 1i , It means that the X, Y, Z three-dimensional coordinate information of the i-th, i=1,2,3,...,n-th point in the point cloud is obtained by FPS in SA-1 and its dimension is obtained after MLP in SA-1. 2 High-dimensional features with u=64; Step 442: For the second layer feature extraction module SA-2, input X2 and X2', and obtain X3={ 3 x 11 , 3 x 12 ,…, 3 x 1α }and in, 3 x 1i , It means that the X, Y, Z three-dimensional coordinate information of the i-th (i=1, 2, 3, …, α) point in the point cloud is obtained by FPS in SA-2 and its dimension is obtained by MLP in SA-2. 3 High-dimensional features with u=128; Step 443: X3 and X3' are input to the third-layer feature extraction module SA-3, and X4 is obtained through the feature extraction module SA-3. 4 x 11 , 4 x 12 ,…, 4 x 1β }and in, 4 x 1i , It means that the X, Y, Z three-dimensional coordinate information of the i-th point (i=1, 2, 3, ..., β) in the point cloud is obtained by FPS in SA-3 and its dimension is obtained by MLP in SA-3. 4 High-dimensional features with u=256; Step 444: For the 4th layer feature extraction module SA-4, X4 and X'4 are input, and X5={ a x 11 , a x 12 ,…, a x 1γ }and in, a x 1i , Indicates the X, Y, Z three-dimensional coordinate information of the i-th (i=1, 2, 3, ..., γ) point in the point cloud obtained by FPS in SA-4 and its high-dimensional features of 512 dimensions obtained by MLP in SA-4; Step 445: For the first spatial information decoding layer block FP-1, input X'4 and X5', and pass the inverse distance interpolation averaging method and jump connection in the spatial information decoding layer FP-1, and reduce the dimension to obtain in, It means that the i-th point in the point cloud (i=1,2,3,…,β) is reduced in dimension by MLP in FP-1 to obtain a high-dimensional feature of 256 dimensions; Step 446: For the second spatial information decoding layer block FP-2, input X3' and P1', and pass the inverse distance interpolation averaging method and jump connection in the spatial information decoding layer FP-2, and reduce the dimension to obtain in, It means that the i-th point in the point cloud (i = 1, 2, 3, ..., α) is reduced to a high-dimensional feature of 256 dimensions by MLP in FP-2; Step 447: For the third spatial information decoding layer block FP-3, input X2' and P2', and pass through the inverse distance interpolation averaging method and jump connection in the spatial information decoding layer FP-3, and reduce the dimension to obtain in, It means that the i-th point in the point cloud (i=1,2,3,…,n) is reduced to a high-dimensional feature of 128 dimensions by MLP in FP-3; Step 448: For the fourth spatial information decoding layer block FP-4, input P3' and obtain P3' through the inverse distance interpolation averaging method in the spatial information decoding layer FP-4. in, It means that the i-th point in the point cloud, i=1,2,3,…,N, is passed through the MLP in FP-4 to obtain a high-dimensional feature of 128 dimensions; Step 449: Use the fully connected layer to reduce the dimension of P4' so that the final dimension is equal to the number of segmentation categories, and give the segmentation result in, It means that the i-th point in the point cloud (i=1,2,3,…,N) passes through the fully connected layer to obtain the category prediction value of the point.

9. The method for segmenting multi-category stacked workpiece point clouds based on a local point cloud feature enhancement mechanism according to claim 1, characterized in that: The specific implementation process of step S5 is: putting the segmentation data set produced in step S3 into the point cloud segmentation network based on the enhanced local point cloud feature enhancement mechanism in step S4 for training to obtain an inference model; then, using a visual camera to collect point clouds of multiple types of stacked workpieces in a real scene, and inputting them into the trained inference model for inference, thereby obtaining unobstructed workpiece point clouds.

10. The method for segmenting multi-category stacked workpiece point clouds based on a local point cloud feature enhancement mechanism according to claim 1, characterized in that: The specific steps of step S6 are: using the region growing segmentation method to perform a secondary simple segmentation on the multiple unobstructed workpieces output by the network in step S5; using the region growing segmentation method to perform a secondary segmentation on the unobstructed workpiece point cloud in multiple types of stacked workpieces, so as to easily obtain the point cloud of a single workpiece.

Images