3D point cloud representation learning method based on enhanced graph convolution

Abstract

The application provides a 3D point cloud representation learning method based on enhanced graph convolution, which has the following characteristics, including the following steps: step S1, obtaining k neighborhood points of an unordered point according to a knn algorithm of pair distance calculation; step S2, constructing a local neighborhood graph of the unordered point according to the unordered point and the k neighborhood points; step S3, respectively performing global feature enhancement on point coordinates and point features to obtain point coordinate global feature enhanced representation and point feature global feature enhanced representation; step S4, calculating a vector Query, a vector Key and a vector Value of an attention mechanism, and position encoding; and step S5, calculating a local neighborhood graph feature according to the point feature, the vector Query, the vector Key, the vector Value and the position encoding. In summary, the method can improve the accuracy and accuracy of point cloud segmentation and classification.

Description

TECHNICAL FIELD

[0001] The present application relates to a point cloud representation learning method, in particular to a 3D point cloud representation learning method based on enhanced graph convolution. BACKGROUND

[0002] 3D sensors can scan and generate 3D point cloud data corresponding to real-world objects such as objects, terrain, etc. In particular, for advanced 3D object acquisition systems such as airborne laser scanners (ALS), mobile laser scanners (MLS), and terrestrial laser scanners (TLS), they can obtain the geometric information of 3D objects in real time and model them with high precision in the form of point clouds. Compared with classic 2D remote sensing images, 3D point clouds have unique spatial information that can more completely describe the geometric information of objects. In recent years, with the continuous development of remote sensing technology and the rapid iteration and upgrading of sensor equipment, 3D sensors such as lidar are becoming more and more available and inexpensive. Therefore, 3D point cloud data has been widely used, and its scale has become increasingly large. How to learn representation information from these point cloud data to solve various problems related to point cloud processing such as point cloud segmentation, classification, etc., so as to better realize 3D environment perception and serve remote sensing tasks such as urban 3D modeling, forestry monitoring, road segmentation, and simultaneous localization and mapping has become an important research hotspot.

[0003] Unlike 2D images arranged in a regular pixel grid, 3D point clouds are unordered and unstructured point sets in three-dimensional space, which greatly limits the direct application of mature 2D deep neural networks on 3D point clouds. In order to cope with this major challenge, in recent years, various point cloud representation learning methods based on deep learning have emerged. Existing point cloud representation learning methods include:

[0004] Irregular point clouds are converted into regular representations of voxels, and then 3D CNN is used to complete point cloud feature learning as if using CNN to learn pixel features in 2D images. Although this method solves the problem of unstructured point cloud data, the voxelization process inevitably loses geometric details, which are indispensable for point cloud feature learning, which will cause its point cloud segmentation and classification performance to remain at a low level.

[0005] Based on the idea of dimensionality reduction, 3D point clouds are rendered into multiple 2D images using different viewpoint cameras, and then mature 2D CNN is used to extract features from multiple views. Finally, the multi-view low-level local features are aggregated into 3D global descriptors to complete point cloud feature learning. Although this method obtains 2D images from different angles, it still cannot capture all the geometric and spatial information of the point cloud, which will affect the accuracy of subsequent point cloud segmentation and classification tasks.

[0006] Methods of directly processing 3D point cloud data, including pointwise MLP methods, convolution-based methods, graph-based methods and transformer-based methods, can effectively capture global and local feature information of point clouds, but these methods either only focus on how to efficiently learn local features of point clouds or capture long-range dependencies of point clouds but lose fine-grained features of local neighborhoods of point clouds. In addition, these methods also have the problem of not being able to efficiently utilize the spatial geometric information of point cloud objects.

[0007] In summary, the existing methods are difficult to further improve the accuracy of point cloud segmentation and classification. SUMMARY

[0008] The present application is to solve the above problems, and aims to provide a 3D point cloud representation learning method based on enhanced graph convolution.

[0009] The present application provides a 3D point cloud representation learning method based on enhanced graph convolution, which is used to extract local neighborhood graph features of unordered points x a in a point cloud set X containing n unordered points Has the characteristics, including the following steps: step S1, according to the knn algorithm of pair distance calculation, select k unordered points from the point cloud set X as the neighborhood points of unordered point x a Step S2, according to the unordered point x a And the k neighborhood points, the local neighborhood graph g of the unordered point x a Is constructed, the feature information contained in the local neighborhood graph g includes point coordinates And point features Step S3, the point coordinates And point features Respectively, the global feature enhancement is carried out, and the point coordinate global feature enhancement representation And the point feature global feature enhancement representation Step S4, according to the point coordinates Point features Point coordinate global feature enhancement representation And point feature global feature enhancement representation The vector Query, the vector Key and the vector Value of the attention mechanism and the position encoding Step S5, according to the point features Vector Query, vector Key, vector Value and position encoding The local neighborhood graph feature

[0010] In the 3D point cloud representation learning method based on enhanced graph convolution provided by the application, the step S1 can further include the following sub-steps: step S1-1, calculating the pair distance between the unordered point x a and the remaining n-1 unordered points in the point cloud set X respectively; step S1-2, sorting all the pair distances from small to large, and selecting the unordered points corresponding to the first k pair distances as the neighborhood points of the unordered point x a .

[0011] In the 3D point cloud representation learning method based on enhanced graph convolution provided by the application, the step S2 can further include the following sub-steps: step S2-1, calculating the local neighborhood graph g of the unordered point x , and the formula of the point coordinate is as follows: , wherein p a is the three-dimensional point coordinate of the unordered point x a , p is the three-dimensional point coordinate of the first neighborhood point, p is the three-dimensional point coordinate of the jth neighborhood point, and p is the three-dimensional point coordinate of the kth neighborhood point. , wherein p a is the point feature of the unordered point x a , p is the point feature of the first neighborhood point, p is the point feature of the jth neighborhood point, and p is the point feature of the kth neighborhood point.

[0012] In the 3D point cloud representation learning method based on enhanced graph convolution provided by the application, the step S3 can further include the following sub-steps: step S3-1, calculating the point coordinate global feature enhanced representation , and the formula of the point coordinate global feature enhanced representation is as follows: , wherein p is the point feature global feature enhanced representation. , wherein is a global feature aggregation function.

[0013] In the 3D point cloud representation learning method based on enhanced graph convolution provided by the application, the step S4 can further include the following sub-steps: step S4-1, calculating the vector Query, and the formula of the vector Query is as follows: , wherein p is the vector Key, and p is the vector Value. ​ where LRBN is processed by leaky_relu activation function and batch normalization, and φ is 1×1 convolution.

[0014] In the 3D point cloud representation learning method based on enhanced graph convolution provided by the application, the step S5 can further include the following sub-steps: in step S5-1, the position encoding vector Query and vector Key are used to calculate the feature map In step S5-2, the vector Value and the feature map are used to calculate the self-attention calculation result In step S5-3, the self-attention calculation result and the point feature are connected in residual, to obtain the edge feature set In step S5-4, the edge feature set is aggregated to obtain the local neighborhood graph feature

[0015] In the 3D point cloud representation learning method based on enhanced graph convolution provided by the application, the feature map can have the following formula: The formula of the self-attention calculation result is as follows: The formula of the edge feature set is as follows: The formula of the local neighborhood graph feature is as follows: where d k is the dimension of the vector Key, CoBN is processed by 1×1 convolution and batch normalization, LR is leaky_relu activation function, is the feature aggregation function.

[0016] Effects of the application

[0017] According to the 3D point cloud representation learning method based on enhanced graph convolution related to the present application, on the one hand, the local neighborhood graph is enhanced by the EFR method of the present application, so that the local features, global features and enhanced global features of the local neighborhood graph can be obtained, thereby more compactly representing the features of the local neighborhood graph; on the other hand, the local neighborhood graph features generated by the EFR method are sequentially subjected to attention weight dynamic allocation, convolution extraction of refined features and adaptive learning of edge features by the GCT method of the present application, so that the generated edge features can more significantly represent the local geometric information of the point cloud object. By constructing the point cloud partial segmentation model and the point cloud classification model with multiple EFR-GCT layers through the EFR method and the GCT method, the point cloud partial segmentation accuracy and the point cloud classification accuracy of the present application are better than those of the existing models. Therefore, the 3D point cloud representation learning method based on enhanced graph convolution of the present application can improve the accuracy and precision of point cloud segmentation and classification. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 FIG. 1 is a flowchart of the 3D point cloud representation learning method based on enhanced graph convolution in an embodiment of the present application;

[0019] Figure 2 FIG. 2 is a schematic diagram of the local neighborhood graph g of the unordered point x a in an embodiment of the present application;

[0020] Figure 3 FIG. 3 is a visual comparison diagram of four point feature representation methods in an embodiment of the present application;

[0021] Figure 4 FIG. 4 is a structural diagram of the GCT method in an embodiment of the present application;

[0022] Figure 5 FIG. 5 is a structural diagram of the point cloud partial segmentation model in an embodiment of the present application;

[0023] Figure 6 FIG. 6 is a partial segmentation result diagram of the point cloud partial segmentation model of the present application in an embodiment of the present application;

[0024] Figure 7 FIG. 7 is an error comparison diagram of the point cloud partial segmentation model of the present application and the AGC o nv model in an embodiment of the present application;

[0025] Figure 8 FIG. 8 is a structural diagram of the point cloud classification model in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the technical means, creative features, purposes and effects of the present application easy to understand, the following embodiments will be specifically described based on the drawings of the enhanced graph convolutional 3D point cloud representation learning method of the present application.

[0027] Figure 1 is a flowchart of the enhanced graph convolutional 3D point cloud representation learning method in the embodiment of the present application.

[0028] As shown in Figure 1 , the enhanced graph convolutional 3D point cloud representation learning method of the present embodiment is used to extract the local neighborhood graph features of the unordered point x a in the point cloud set X containing n unordered points. The method comprises the following steps:

[0029] Step S1, according to the knn algorithm of pair-wise distance calculation, select k unordered points from the point cloud set X as the neighborhood points of the unordered point x a .

[0030] Wherein, step S1 comprises the following sub-steps:

[0031] Step S1-1, calculate the pair-wise distance between the unordered point x a and the remaining n-1 unordered points in the point cloud set X.

[0032] Step S1-2, sort all pair-wise distances from small to large, and select the unordered points corresponding to the first k pair-wise distances as the neighborhood points of the unordered point x a .

[0033] Step S2, according to the unordered point x a and the k neighborhood points, construct the local neighborhood graph g of the unordered point x a , the local neighborhood graph g contains feature information including point coordinates and point features

[0034] Figure 2 is a schematic diagram of the local neighborhood graph g of the unordered point x a in the embodiment of the present application.

[0035] As shown in Figure 2 , the local neighborhood graph g is composed of the unordered point x a , i.e. the center point, and the k neighborhood points, i.e. ,

[0036] Wherein, the formula of the local neighborhood graph g is:

[0037]

[0038] The formula of the point coordinates is:

[0039]

[0040] where p a is the 3D point coordinate of the unordered point x a , is the 3D point coordinate of the 1st neighbor point, is the 3D point coordinate of the jth neighbor point, is the 3D point coordinate of the kth neighbor point.

[0041] Point feature is calculated as:

[0042]

[0043] where p a is the point feature of the unordered point x a , is the point feature of the 1st neighbor point, is the point feature of the jth neighbor point, is the point feature of the kth neighbor point, which contains both 3D point coordinate and surface normal.

[0044] Local neighborhood graph g a spreads the center point x and all its neighbor points, which contains both local feature relationship between points and global feature information represented by the center point and its neighbor points, but it contains not compact enough features, which needs further processing to compactly represent the features of the local neighborhood graph g.

[0045] Figure 3 is the visualization comparison diagram of four point feature representation methods in the embodiments of the present application.

[0046] As Figure 3 shown, (a) is the visualization diagram of the existing point feature representation method of PointNet, (b) is the visualization diagram of the existing point feature representation method of PointNet++, (c) is the visualization diagram of the existing point feature representation method of DGCNN, and (d) is the visualization diagram of the enhanced point feature representation method EFR proposed by the 3D point cloud feature learning method based on enhanced graph convolution of the present application, and different feature representations obtained by processing the local feature graph via the four methods are as follows:

[0047] The PointNet method only encodes the global feature information of the local point cloud x a Without considering the characteristic connections of its local neighborhood This method is used for point x a The formula for the feature representation R of the local feature map is: R = h θ (x a ).

[0048] The PointNet++ method regards the feature information of the local point cloud as the center point x a and neighboring points The point pair features between Encoding without losing the global geometry x a , the characteristic representation R of this method is:

[0049] The DGCNN method transforms the center point x a The global feature information represented by the neighborhood point pair and the local neighborhood information represented by the neighborhood point pair Combined to represent the local neighborhood graph of the point cloud, but all neighborhood points are lost Separate feature information, the formula for the feature representation R of this method is:

[0050] The EFR method uses the center point x a and neighboring points The relative distance To represent the local features of the neighborhood graph g, use the center point x a The feature information represents the global features of the neighborhood graph g, for all neighborhood points Apply global feature aggregation functions To further enhance the representation of global features, the feature representation R of this method is: Compared with the other three methods, it can more compactly represent the global and local features of the neighborhood graph g.

[0051] Step S3, point coordinates and point features Perform global feature enhancement separately to obtain the point coordinate global feature enhancement representation and point feature global feature enhanced representation

[0052] Among them, point coordinate global feature enhancement representation The formula is:

[0053]

[0054]

[0055] Point feature global feature enhanced representation The formula is:

[0056]

[0057]

[0058] In the formula is a global feature aggregation function. In this embodiment, the global feature aggregation function includes average pooling.

[0059] The 3D point cloud representation learning method based on enhanced graph convolution of the present invention proposes a Graph Convolution Transformer method, namely the GCT method, for extracting deeper point cloud features at different scales of the point cloud object. In this embodiment, the GCT method is embodied in steps S4 and S5.

[0060] Figure 4 Schematic diagram of the structure of the GCT method in an embodiment of the present invention.

[0061] like Figure 4 As shown, (a) Input embedding construction is the structure of input embedding construction, which is the processing process of step S4 in this embodiment, used to obtain the vector Query, vector Key and vector Value of the attention mechanism, as well as the position encoding (b) Point cloud feature learning is the structure of point cloud feature learning, which is the processing process of step S5 in this embodiment, used to obtain local neighborhood graph features. Combine Figure 4 Steps S4 and S5 are described as follows:

[0062] Step S4, according to the point coordinates Point Features Point coordinate global feature enhanced representation and point feature global feature enhanced representation Calculate the attention mechanism's vector Query, vector Key, vector Value, and position encoding

[0063] In step S4, the formula of the vector Query is:

[0064]

[0065] The formula for vector Key is:

[0066]

[0067] The formula for the vector Value is:

[0068]

[0069] Positional encoding The formula is:

[0070]

[0071] Where LRBN is processed by leaky_relu activation function and batch_normalization, and φ is 1×1 convolution, that is, Figure 4 Conv in.

[0072] By using the EFR method to simultaneously map point coordinates and point features Embedding Query, Key and Value into the input makes it possible to fully utilize the geometric information of the 3D point cloud object, and the constructed input embedding can contain rich feature relationships between different point pairs.

[0073] The spatial coordinates of each point in the unordered point set can uniquely represent its own spatial position information, so the point coordinates are converted to Map them to the same dimension, then perform point-to-point subtraction on them to get the relative distance between different points, and then process the calculated relative distance through 1×1 convolution to construct the position code

[0074] Step S5: Based on point features Vector Query, Vector Key, Vector Value, and Positional Encoding Calculate the local neighborhood graph features

[0075] Wherein, step S5 includes the following sub-steps:

[0076] Step S5-1, according to the position code Vector Query and vector Key, calculate the feature map

[0077] Feature Map The formula is:

[0078]

[0079] Where d k The dimension of the vector Key.

[0080] The attention score is calculated by the subtraction operation between the vector Query and the vector Key, which represents the relative distance between points, can make full use of the characteristics of the three-dimensional coordinates of points, and is more robust using relative distance.

[0081] The position encoding is added to the self-attention calculation, which is beneficial to learn the feature information of the proximity of points belonging to the same semantic part of the point cloud object and the distinctive spatial position information of points belonging to different semantic parts.

[0082] Step S5-2, the vector Value and the feature map are calculated to obtain the self-attention calculation result

[0083] The formula of the self-attention calculation result is as follows:

[0084]

[0085] In the formula, CoBN is processed by 1x1 convolution and batch normalization.

[0086] Although the self-attention calculation is very suitable for capturing the global features of the local neighborhood of the point cloud, it will lose the equally important fine-grained features, so a 1x1 convolution processing is added to the self-attention calculation to strengthen the extraction of local fine-grained features.

[0087] Step S5-3, the self-attention calculation result and the point feature are performed once residual connection to obtain the edge feature set for relieving the problem of gradient disappearance.

[0088] The formula of the edge feature set is as follows:

[0089]

[0090]

[0091] In the formula, LR is a leaky_relu activation function.

[0092] Step S5-4, the edge feature set is aggregated to obtain the local neighborhood graph feature

[0093] The formula of the local neighborhood graph feature is as follows:

[0094]

[0095] In the formula is a feature aggregation function, and in the embodiment, the feature aggregation function is a max pooling Max Pool.

[0096] In the embodiment, the 3D point cloud representation learning method based on enhanced graph convolution of the application is applied to point cloud partial segmentation, and a point cloud partial segmentation model is constructed as the point cloud partial segmentation model of the application, which is used to process an input point cloud set according to a pre-constructed category vector to obtain a point cloud partial segmentation result.

[0097] Figure 5 is a structural schematic diagram of the point cloud partial segmentation model in the embodiment of the application.

[0098] As shown in Figure 5 , the point cloud partial segmentation model includes an STN layer, a first EFR layer, a second EFR layer, a third EFR layer, a fourth EFR layer, a fifth EFR layer, a sixth EFR layer, a seventh EFR layer, an eighth EFR layer, a first GCT layer, a second GCT layer, a third GCT layer, a fourth GCT layer, a 1x1 convolution layer, a first multi-layer perceptron, a second multi-layer perceptron, a splicing module, and a shared multi-layer perceptron.

[0099] The STN layer is used to perform STN normalization processing on the input point cloud to obtain first point coordinates and first point features.

[0100] The first EFR layer is used to perform feature enhancement on the input first point features according to the EFR method to obtain a first point feature global feature enhanced representation.

[0101] The second EFR layer is used to perform feature enhancement on the input first point coordinates according to the EFR method to obtain a first point coordinate global feature enhanced representation.

[0102] The first GCT layer is used to calculate the first local neighborhood graph feature according to the GCT method based on the input first point coordinates, first point features, first point feature global feature enhanced representation, and first point coordinate global feature enhanced representation.

[0103] The third EFR layer is used to perform feature enhancement on the point features in the input first local neighborhood graph feature as second point features according to the EFR method to obtain a second point feature global feature enhanced representation.

[0104] The fourth EFR layer is used to perform feature enhancement on the input first point coordinates according to the EFR method to obtain a second point coordinate global feature enhanced representation.

[0105] The second GCT layer is used to calculate the second local neighborhood graph feature according to the GCT method based on the input first point coordinates, second point features, second point feature global feature enhanced representation, and second point coordinate global feature enhanced representation.

[0106] The fifth EFR layer is configured to perform feature enhancement on the point features in the second local neighborhood map feature after the FPS processing as third point features according to the EFR method, to obtain a third point feature global feature enhanced representation.

[0107] The sixth EFR layer is configured to perform feature enhancement on the spliced results of the first point coordinates and the point coordinates in the second local neighborhood map feature, i.e., the third point coordinates, after the FPS processing according to the EFR method, to obtain a third point coordinate global feature enhanced representation.

[0108] The third GCT layer is configured to perform calculation on the input third point coordinates, third point features, third point feature global feature enhanced representation and third point coordinate global feature enhanced representation according to the GCT method, to obtain a third local neighborhood map feature.

[0109] The seventh EFR layer is configured to perform feature enhancement on the point features in the third local neighborhood map feature after the FPS processing as fourth point features according to the EFR method, to obtain a fourth point feature global feature enhanced representation.

[0110] The eighth EFR layer is configured to perform feature enhancement on the spliced results of the third point coordinates and the point coordinates in the third local neighborhood map feature, i.e., the fourth point coordinates, after the FPS processing according to the EFR method, to obtain a fourth point coordinate global feature enhanced representation.

[0111] The fourth GCT layer is configured to perform calculation on the input fourth point coordinates, fourth point features, fourth point feature global feature enhanced representation and fourth point coordinate global feature enhanced representation according to the GCT method, to obtain a fourth local neighborhood map feature.

[0112] The 1*1 convolution layer is configured to perform convolution on the point features in the fourth local neighborhood map feature after the FPS processing, i.e., the fourth point features, and the spliced results of the fourth point coordinates and the point coordinates in the fourth local neighborhood map feature after the FPS processing, i.e., the fifth point coordinates, to obtain a convolution result.

[0113] The first multi-layer perception is configured to process the fifth point coordinates and the convolution result, to obtain fifth point features.

[0114] The second multi-layer perception is configured to process the self-stacked fifth point features, to obtain sixth point features.

[0115] The splicing module is configured to splice the first point features, the second point features, the third point features, the fourth point features, the fifth point features, the sixth point features and the category vector, to obtain multi-scale point features.

[0116] The shared multi-layer perception is configured to process the multi-scale point features, to obtain a point cloud partial segmentation result.

[0117] In this embodiment, the FPS processing is the farthest sampling point operation.

[0118] In this embodiment, 14006 samples of the existing ShapeNetPart dataset are used as training samples, the number of neighborhood points k is set to 30, the total batch number of training is 200, 30 samples are selected for training in each batch, the SGD algorithm is used to optimize the point cloud partial segmentation model, the momentum is set to 0.9, the initial learning rate is 0.1, and the cosine annealing is used to reduce the learning rate to 0.001 in the training process, and then the trained point cloud partial segmentation model is obtained.

[0119] Figure 6 is a partial segmentation result schematic diagram of the point cloud partial segmentation model in the embodiment of the application.

[0120] As shown in Figure 6 , the black box represents the partial segmentation result of the point cloud partial segmentation model of the application on the “airplane” sample in the ShapeNetPart dataset, and the model divides the “airplane” sample into four different parts, namely the engine, wing, fuselage and tail. In this embodiment, the model is used to perform partial segmentation on other samples in the ShapeNetPart dataset, and the samples can also be accurately segmented into 2-6 parts.

[0121] The point cloud partial segmentation model of the application and the AGConv model constructed according to the existing AGConv method are compared in terms of partial segmentation error on the ShapeNetPart dataset.

[0122] Figure 7 is an error comparison schematic diagram of the point cloud partial segmentation model of the application and the AGConv model in the embodiment of the application.

[0123] As shown in Figure 7 , the left side is an error schematic diagram of the point cloud partial segmentation model of the application, and the right side is an error schematic diagram of the AGConv model. The black circles represent points that are incorrectly segmented when the model is used to perform partial segmentation on the sample. The size of the circle represents the degree of incorrect segmentation. The larger the circle, the greater the degree of incorrect segmentation. As can be seen, the point cloud partial segmentation model of the application has better partial segmentation accuracy for points at the boundary position compared to the AGConv model on this sample. As can be seen, the point cloud partial segmentation model of the application effectively represents the local neighborhood graph features at the boundary of different semantic parts of the point cloud object through the EFR layer, and then extracts deep features of different scales through multiple GCT layers. The model not only learns the fine-grained features of points belonging to the same semantic part in the local region, but also captures the distinctive long-range dependency relationship between points belonging to different semantic parts in the local region, thereby achieving more effective partial segmentation of point clouds.

[0124] According to the existing point-based PointNet method and PointNet++ method, attention-based DGANet method and AGNet method, graph-based DGCNN method, PAConv method and AGConv method, and transformer-based PCT method, GiGT method, PT method and ST method, the corresponding point cloud partial segmentation models are respectively constructed, and the obtained PointNet model, PointNet++ model, DGANet model, AGNet model, DGCNN model, PAConv model, AGConv model, PCT model, GiGT model, PT model and ST model are compared with the point cloud partial segmentation model of the application in the point cloud partial segmentation effect on the existing ShapeNetPart data set, and the segmentation indexes mIoU of each model are calculated as shown in the following table:

[0125]

[0126]

[0127] The first column in the above table is the name of each model, and the second column is the mIoU corresponding to each model, for example, the cell in the second column of the second row indicates that the segmentation index mIoU of the point cloud partial segmentation of the PointNet model on the ShapeNetPart data set is 83.7%, and from the above table, the mIoU of the point cloud partial segmentation model of the application is 86.8%, which is higher than that of each model, indicating that the point cloud partial segmentation model of the application has the best point cloud partial segmentation accuracy on the ShapeNetPart data set compared with other models.

[0128] In this embodiment, the STN layer of the point cloud partial segmentation model of the application is subjected to ablation verification, the point cloud partial segmentation model of the application without the STN layer is taken as the no-STN model, and the mIoU calculated on the ShapeNetPart data set is shown in the following table:

[0129] Model name mIoU (%) STN-free model 86.6 Point cloud partial segmentation model of the application 86.8

[0130] The first column in the above table is the name of each model, and the second column is the mIoU corresponding to each model, for example, the cell in the second column of the second row indicates that the segmentation index mIoU of the point cloud partial segmentation of the PointNet model on the ShapeNetPart data set is 83.7%, and from the above table, the mIoU of the point cloud partial segmentation model of the application is 86.8%, which is higher than that of each model, indicating that the point cloud partial segmentation model of the application has the best point cloud partial segmentation accuracy on the ShapeNetPart data set compared with other models.

[0131] In this embodiment, the 3D point cloud representation learning method based on enhanced graph convolution of the application is applied to point cloud classification, and a point cloud classification model is constructed as the point cloud classification model of the application, which is used for processing an input point cloud set to obtain a point cloud classification result.

[0132] Figure 8 FIG. 1 is a structural schematic diagram of the point cloud classification model in an embodiment of the application.

[0133] As shown in FIG. 1, the point cloud classification model includes a first EFR layer, a second EFR layer, a third EFR layer, a fourth EFR layer, a first GCT layer, a second GCT layer, a first 1x1 convolution layer, a second 1x1 convolution layer, a splicing module, a spatial attention (SA) layer, and a shared multi-layer perception. Figure 8 The first EFR layer is configured to perform feature enhancement on point coordinates in the input point cloud set as first point coordinates according to the EFR method, to obtain a first point coordinate global feature enhanced representation.

[0134] The second EFR layer is configured to perform feature enhancement on point coordinates in the input point cloud set as second point coordinates according to the EFR method, to obtain a second point coordinate global feature enhanced representation.

[0135] The first GCT layer is configured to perform calculation on the input first point coordinates, second point coordinates, first point coordinate global feature enhanced representation, and second point coordinate global feature enhanced representation according to the GCT method, to obtain a first local neighborhood graph feature.

[0136] The third EFR layer is configured to perform feature enhancement on point features in the first local neighborhood graph feature as first point features according to the EFR method, to obtain a first point feature global feature enhanced representation.

[0137] The fourth EFR layer is configured to perform feature enhancement on point coordinates in the input point cloud set as third point coordinates according to the EFR method, to obtain a third point coordinate global feature enhanced representation.

[0138] The second GCT layer is configured to perform calculation on the input third point coordinates, first point features, first point feature global feature enhanced representation, and third point coordinate global feature enhanced representation according to the GCT method, to obtain a second local neighborhood graph feature.

[0139] The first 1x1 convolution layer is configured to perform convolution calculation on point features in the second local neighborhood graph feature as second point features, to obtain third point features.

[0140] The second 1x1 convolution layer is configured to perform convolution calculation on the third point features, to obtain fourth point features.

[0141]

[0142] ​The splicing module is configured to splice the first point feature, the second point feature, the third point feature and the fourth point feature to obtain a multi-scale point feature.

[0143] The spatial attention SA layer is configured to update the multi-scale point feature to obtain an updated point feature.

[0144] The shared multi-layer perceptron is configured to process the updated point feature to obtain a point cloud classification result.

[0145] In the embodiment, the first EFR layer, the second EFR layer and the first GCT layer are taken as a first EFR-GCT layer, the third EFR layer, the fourth EFR layer and the second GCT layer are taken as a second EFR-GCT layer, and the EFR-GCT stack layer is stacked by the first EFR-GCT layer and the second EFR-GCT layer, and is configured to obtain a plurality of point features of different scales, and the output of each EFR-GCT layer is used as the input of the next EFR-GCT layer.

[0146] In the embodiment, 9843 samples of an existing ModelNet40 dataset are taken as training samples, the number of neighborhood points k is set to 20, the total batch number of training is set to 500, 30 samples are selected for training in each batch, and the point cloud classification model is optimized by using an SGD algorithm, wherein the momentum is set to 0.9, the initial learning rate is set to 0.1, and the cosine annealing is used to reduce the learning rate to 0.001 in the training process, and thus the trained point cloud classification model is obtained.

[0147] In the embodiment, the corresponding point cloud classification models are respectively constructed according to the existing DGANet method, the AGNet method, the DGCNN method and the PT method, the obtained DGANet model, the AGNet model, the DGCNN model and the PT model are compared with the point cloud classification model in the point cloud classification effect on the ModelNet40 dataset, and the average accuracy AA of each model is calculated and shown in the following table:

[0148] Model name AA (%) DGANet model 89.4 AGNet model 90.7 DGCNN model 90.2 PT model 90.6 Point cloud classification model of the application 91.2

[0149] In the above table, the first column is the name of each model, and the second column is the AA corresponding to each model, for example, the cell in the second column of the sixth row indicates that the average accuracy AA of the point cloud classification of the point cloud classification model on the ModelNet40 dataset is 91.2%, and it can be seen from the above table that the point cloud classification model has the best average accuracy of point cloud classification on the ModelNet40 dataset compared with other models.

[0150] Effects of the embodiment

[0151] According to the 3D point cloud representation learning method based on enhanced graph convolutional network related to the embodiment, on the one hand, the local neighborhood graph is enhanced by the EFR method, and the local features, global features and enhanced global features of the local neighborhood graph are obtained, so that the features of the local neighborhood graph are more compactly represented; on the other hand, the local neighborhood graph features generated by the EFR method are sequentially subjected to attention weight dynamic distribution, convolution extraction of refined features and adaptive learning of edge features by the GCT method, so that the generated edge features can more significantly represent the local geometric information of the point cloud object. The point cloud partial segmentation model and the point cloud classification model with multiple EFR-GCT layers are constructed by the EFR method and the GCT method, and compared with the existing model, the point cloud partial segmentation accuracy and the point cloud classification accuracy are better. In summary, the method can improve the accuracy and accuracy of point cloud segmentation and classification.

[0152] The above embodiment is a preferred case of the present application and does not limit the protection scope of the present application.

Claims

1. A 3D point cloud representation learning method based on enhanced graph convolution for extracting unordered points x from a point cloud set X containing n unordered points a Local neighborhood graph features It is characterized in that The following steps are involved: Step S1, according to the knn algorithm of pairwise distance calculation, select k unordered points from the point cloud set X as the unordered points x a Neighborhood points of Step S2, according to the disordered point x a and k neighborhood points to construct the unordered point x a The local neighborhood graph g contains feature information including point coordinates and point features Step S3, coordinate the point and the point features Perform global feature enhancement separately to obtain the point coordinate global feature enhancement representation and point feature global feature enhanced representation Step S4, according to the point coordinates The point features The point coordinate global feature enhancement representation And the point feature global feature enhanced representation Calculate the attention mechanism's vector Query, vector Key, vector Value, and position encoding Step S5: Based on the point features The vector Query, the vector Key, the vector Value and the position code Calculate the local neighborhood graph features Wherein, the step S5 includes the following sub-steps: Step S5-1, according to the position code The vector Query and the vector Key are used to calculate the feature map Step S5-2, based on the vector Value and the feature map Calculate the self-attention calculation results Step S5-3, calculate the self-attention result and the point features Perform a residual connection to obtain the edge feature set Step S5-4, the edge feature set Perform aggregation to obtain the local neighborhood graph features The feature map The formula is: The self-attention calculation result The formula is: The edge feature set The formula is: The local neighborhood graph features The formula is: Where, d k is the dimension of the vector Key, CoBN is processed by 1×1 convolution and batch_normalization, LR is the leaky_relu activation function, is the feature aggregation function.

2. The 3D point cloud representation learning method based on enhanced graph convolution according to claim 1, characterized in that: in, The step S1 includes the following sub-steps: Step S1-1, calculate the disordered point x a The pairwise distances between each of the remaining n-1 unordered points in the point cloud set X; Step S1-2: sort all the pairwise distances from small to large, and select the unordered points corresponding to the first k pairwise distances as the unordered points x a The neighborhood points.

3. The 3D point cloud representation learning method based on enhanced graph convolution according to claim 1, characterized in that: in, In step S2, the formula of the local neighborhood graph g is: The point coordinates The formula is: Where p a is the disordered point x a The three-dimensional coordinates of the point, is the three-dimensional point coordinate of the first neighborhood point, is the three-dimensional point coordinate of the j-th neighborhood point, is the three-dimensional point coordinate of the kth neighborhood point, The point features The formula is: Where p a is the disordered point x a The point features, is the point feature of the first neighborhood point, is the point feature of the j-th neighborhood point, is the point feature of the kth neighborhood point, where the point feature includes the three-dimensional point coordinates and the surface normal.

4. The 3D point cloud representation learning method based on enhanced graph convolution according to claim 1, characterized in that: in, In step S3, the point coordinate global feature enhancement representation The formula is: The point feature global feature enhanced representation The formula is: In the formula is the global feature aggregation function.

5. The 3D point cloud representation learning method based on enhanced graph convolution according to claim 1, characterized in that: in, In step S4, the formula of the vector Query is: The formula of the vector Key is: The formula of the vector Value is: The position code The formula is: Where LRBN is processed by leaky_relu activation function and batch_normalization, and φ is 1×1 convolution.

Images