A graph convolutional network for point cloud semantic segmentation with enhanced local and global semantics

Abstract

A convolutional network for enhancing local and global semantic segmentation of point clouds is proposed. This network utilizes a Local Geometry Augmentation (LGA) module to make the network rotationally invariant about the Z-axis, thereby strengthening the representation of local point cloud information. A Global Geometry Feature (GGF) module is employed to calculate the ratio of local to global sphere volumes and connect it to the coordinate feature X, preserving global geometric information. Multiple symmetric aggregation operations are used to retain local information from various perspectives. Each layer in the network uses residual operations to pass information from the previous layer to the next, and a Reversed Residual MLP (RevResMLP) module is used to mine deeper semantic information.

Description

Technical Field

[0001] This invention relates to a point cloud segmentation method, specifically a point cloud semantic segmentation graph convolutional network that enhances both local and global semantic segmentation. Background Technology

[0002] With the rapid development of 3D sensors, acquiring laser point clouds has become increasingly convenient. Utilizing laser point clouds to perceive the real world plays a crucial role in fields such as home intelligent robots, autonomous driving, and AR / VR. However, point clouds differ from images; images are regularized data, while point clouds are unstructured and disordered data, making it difficult to apply grid-based convolutional neural networks (CNNs).

[0003] In recent years, many deep learning-based methods have been proposed to address this problem. These methods can be broadly categorized into three types: projection-based methods, voxel-based methods, and point-based methods. However, projection-based methods lead to significant information loss, voxel-based methods consume substantial computational resources, while point-based methods do not require preprocessing of point cloud data, thus avoiding information loss and high computational costs. Among point-based methods, graph convolution-based methods have a strong ability to extract information from contextual features. They construct graph data from points based on the similarity between point features and apply 2D convolution to laser point clouds, achieving higher accuracy than point-by-point methods. However, this method does not enhance the local representation capabilities of the point cloud and ignores global geometric information. Furthermore, it uses max pooling to aggregate local information, resulting in the retention of only the maximum response value within a local area.

[0004] To address the above problems, this invention proposes the GRes-Net network, such as... Figure 3 As shown, the Graph Residual (GRes) module enhances the local representation of the point cloud while preserving global geometric information. The GRes module includes LocalGeometryAugment (LGA) and GlobalGeometry Feature (GGF) modules. The LGA module utilizes the relative spherical coordinates between local points and the centroid to ensure rotational invariance of the network with respect to the Z-axis of the point cloud, thereby enhancing the local representation. The GGF module obtains the volume ratio between the local and global points and concatenates it with the point cloud coordinate features to preserve global geometric information. Multiple aggregation operations are used to retain feature information other than the maximum response value. An inverse residual module is added to mine deeper feature information.

[0005] While deep learning has made significant progress in NLP and image processing, its application in laser point clouds has not been well explored. This invention will review previous research, including projection-based methods, voxel-based methods, point-based methods, and graph-based methods.

[0006] Projection- and voxel-based methods. Both methods require converting point clouds into regularized data and performing convolution operations. The former converts point clouds into images, and the latter into voxels. MVCNN extracts features from each view and aggregates them into a global descriptor using max pooling. GVCNN models and groups the visual descriptors extracted from each view. VoxNet converts point clouds into voxels and extracts features using 3D convolution. OctNet uses an imbalanced octree to partition the space, reducing the number of empty voxels and improving memory and computational efficiency.

[0007] Point-based methods: PointNet was the first network to apply deep learning to point cloud processing. It proposed applying a point-by-point MLP and then using max pooling to aggregate global features, but it did not mine local information in the point cloud. PointNet++ uses an Encoder-Decoder structure and focuses on local feature information in the point cloud. KPConv uses kernel functions to calculate the weight matrix for each local point. Point2Sequence learns information from different local regions through a self-attention mechanism. PCT uses a global self-attention mechanism to ensure that each point pays attention to each other's information.

[0008] Graph-based methods treat points as nodes in a graph and establish edges based on the feature relationships between points. DGCNN obtains the neighboring points of each point in the feature space and performs predictions through MLP and aggregation operations. 3D-GCN proposes a deformable kernel, which improves model performance by aggregating at multiple scales. GAC assigns different weights to different graph nodes.

[0009] In summary, directly manipulating point clouds solves the problem of needing preprocessing. However, the graph-based methods mentioned above do not enhance the local representation capabilities of point clouds, ignore global geometric information, and the aggregation operation only retains the local maximum response value. Therefore, this invention proposes the GRes-Net network, which can effectively enhance local representation while preserving global geometric information, thereby improving semantic segmentation accuracy. Summary of the Invention

[0010] To address the issues of current graph convolutional point cloud processing networks failing to enhance local features, neglecting global geometric information, and losing local submaximal information, this invention designs the GRes-Net network structure, providing a graph convolutional network for point cloud semantic segmentation that enhances both local and global features.

[0011] The innovative approach adopted in this invention is:

[0012] A convolutional network for point cloud semantic segmentation, enhancing both local and global semantic features, employs an encoder-decoder architecture. The Dynamic Graph Convolutional Neural Network (DGCNN) uses point cloud coordinates and color information as features, performs a k-nearest neighbor search on each point, calculates the difference between each point's features and those of its neighbors, and concatenates the center point's features with this difference to construct a new feature map. A Multilayer Perceptron (MLP) and max pooling are then used to learn the local features of this new feature map. The network includes a Local Geometry Enhancement (LGA) module, a Global Geometry Feature (GGF) module, multiple aggregation operations, and a RevResMLP module.

[0013] The LGA module is used to enhance the local feature representation of the point cloud;

[0014] The GGF module is used to retain global geometric information;

[0015] The multiple aggregation operation modules are used to retain the local maximum response value and other information;

[0016] The RevResMLP module is used to mine deeper information.

[0017] The LGA module obtains a relative polar coordinate representation by calculating the relative angle between each point in the local area and the local centroid.

[0018] The GGF module obtains the volume ratio of the local to the global point cloud and connects it with the point cloud coordinate features.

[0019] The multiple aggregation operations concatenate the results of max pooling and summation pooling.

[0020] The inverse residual module RevResMLP first increases the dimensionality of the input features and then reduces it, and then adds the output result to the input data.

[0021] The working method of the convolutional network includes the following steps:

[0022] Step 1: Input point cloud data and corresponding hyperparameters;

[0023] Step 2: Perform point cloud semantic segmentation. The sampled points are fed into the encoder to encode the point cloud.

[0024] Step 3: Enhance local point cloud geometric information and preserve global point cloud geometric information in the GRes module. The LGA module enhances local feature representations of the point cloud, while the GGF module preserves global geometric information. Details are as follows:

[0025] The steps for building an LGA module are as follows:

[0026] 1) Calculate the distances between all points within the local region. With angle

[0027] As shown in formulas (1) to (3). For point p i With point The distance between them

[0028] For point p i With point The angle between them. Definition: For point p i Let i be a local region and its k nearest neighbors.

[0029]

[0030]

[0031]

[0032] in, It is point p i Coordinates in the Cartesian coordinate system.

[0033] 2) Calculate the angle of the local incenter of mass. The centroid is calculated using the coordinates of points in a local region. The angle is calculated using formulas (2) and (3).

[0034] 3) Calculate the relative angles between the centroid and all points within the local area.

[0035] As shown in formulas (4) and (5).

[0036]

[0037]

[0038] The steps for using the GGF module are as follows:

[0039] The volume of the global point cloud and the volume of the local point cloud are calculated according to formula (6), and then the ratio of the volume of the local point cloud to the volume of the global point cloud is calculated according to formula (7).

[0040]

[0041]

[0042] Step 4: Apply multiple aggregation operations to preserve the local maximum response value and other information, as shown in formulas (8), (9) and (10);

[0043] Info max =max(input)(8)

[0044] Info sum =sum(input)(9)

[0045] Info = Concat(Info) max ,Info sum (10)

[0046] Where, input is the input feature data; max and sum are the max pooling and summation operations, respectively; and the Concat operation is used to convert Info... max With Info sum Then, the parts are assembled.

[0047] Step 5: Explore deeper information in the RevResMLP module, as shown in formulas (11), (12) and (13);

[0048] input knn =KNN(input)(11)

[0049]

[0050]

[0051] Wherein, input is the input feature data; input knn Local data obtained after performing a K-nearest neighbor search on the input; It is a multilayer perceptron.

[0052] Step 6: Use the furthest sampling point to perform downsampling operation at FPS;

[0053] Step 7: Use the output point cloud as input to the decoder for decoding;

[0054] Step 8: Upsample the input point cloud and then connect it to the output of the corresponding layer in the encoder;

[0055] Step 9: Pass the results through a two-layer multilayer perceptron and output them.

[0056] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0057] This invention proposes the GRes-Net network, which enhances local point cloud information and preserves global geometric information during graph convolution by introducing LGA and GGF modules. It employs multiple aggregation operations to retain the maximum response value and other information within the local point cloud, while utilizing the RevResMLP module to mine deeper semantic information.

[0058] The proposed network was evaluated on S3DIS dataset 5, achieving a segmentation accuracy of 61%. Compared to DGCNN, DeepGCN, and PointWeb, its semantic segmentation performance was improved by 14%, 8.5%, and 0.7%, respectively. Ablation experiments on the LGA, GGF, and RevResMLP modules show that these three modules effectively improve the model's segmentation performance. The LGA module enhances the local feature representation of the point cloud. The GGF module preserves global geometric information. Multiple aggregation operations preserve local maximum response values ​​and other information. A deeper semantic information is mined through the inverse residual module. The GRes-Net network outperforms multiple baselines on the S3DIS dataset. Attached Figure Description

[0059] Figure 1 Constructing new feature maps

[0060] Figure 2 It is a graph residual module (GraphResidual, Gres) module

[0061] Figure 3 It is the GRes-Net network structure

[0062] Figure 4 It is the Local Geometry Augment (LGA) module.

[0063] Figure 5 It is the Global GeometryFeature (GGF) module.

[0064] Figure 6 It is the reverse residual module (ReversedResidualMLP, RevResMLP) module.

[0065] Figure 7 Semantic segmentation visualization on S3DISArea5

[0066] Figure 2 In the diagram: B - batch size, N - number of points, C - coordinate and color feature channel size of the points, 3 - coordinate feature channel size of the points, k - number of neighboring points, out - output channel size, LGA - local geometry enhancement module, GGF - global geometry feature module.

[0067] Figure 3 In Chinese: GRes - Graph Residual, FP - Feature Propagation, RevResMLP - Reversed Residual MLP, FPS - Farthest Point Sample.

[0068] Figure 4 In the middle: Figure (2) shows the calculation of p i and Angle between Figure (3) shows the calculation of p i and Angle between Figures (4), (5), and (6) show the calculations. as well as The difference between them. Detailed Implementation

[0069] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0070] like Figure 1-7 As shown, a convolutional network for point cloud semantic segmentation is proposed to enhance both local and global semantic segmentation.

[0071] The graph residual GRes-Net network proposed in this invention employs an encoder-decoder architecture. The Dynamic Graph Convolutional Neural Network (DGCNN) uses point cloud coordinates and color information as features, and performs a k-nearest neighbor search on each point using these features. It then calculates the difference between the features of each point and its neighbors, concatenates the center point features with this difference, and thus constructs a new feature map, such as... Figure 1 As shown, a new feature map local feature information is learned using a multilayer perceptron (MLP) and max pooling.

[0072] The network consists of three key modules. The first is the Local Geometry Augmentation (LGA) module, which enhances the representation of local information in the point cloud. The second is the Global Geometry Feature (GGF) module, which aims to preserve global geometric information. The third is the Reversed Residual MLP (RevResMLP) module, which mines deeper semantic information from the point cloud. B represents the batch size, N represents the number of points, C represents the size of the RGB and other feature channels of a point, 3 represents the size of the coordinate feature channels of a point, k represents the number of neighboring points, and out represents the output channel size.

[0073] Local Geometry Augmentation Module

[0074] In large-scale point cloud semantic segmentation, different orientations of objects of the same category can lead to differences in the semantic segmentation results. This can affect the overall performance of scene semantic segmentation to some extent. To enhance robustness to orientation, this invention utilizes spherical coordinates to strengthen the local geometric representation. By calculating the relative angle between each point in the local area and its local centroid, which is rotationally invariant about the Z-axis, such as... Figure 4 As shown.

[0075] Calculate distance and angle. Distance is calculated by applying the point coordinates in the Cartesian coordinate system to equations (1), (2), and (3). With angle For point p i With point The distance between them For point p i With point The angle between them. Definition: For point p i Let i be a local region and its k nearest neighbors.

[0076]

[0077]

[0078]

[0079] in, It is point p i Coordinates in the Cartesian coordinate system.

[0080] Calculate the angle of the centroid. The centroid is calculated using the coordinates of points within a local region. This invention defines from p i arrive The direction is the local direction. This definition has two advantages: 1) the centroid can reflect the overall characteristics of the local area; 2) by using the centroid, the randomness generated by downsampling can be reduced. The angle of the centroid is calculated according to equations (2) and (3). Calculate the relative angles. Subtract the angle between each point in the region and the centroid to obtain the relative angles.

[0081]

[0082]

[0083] (1) Calculate p i and Angle between (2) Calculate p i and Angle between Calculate (3), (4), and (5) and as well as and The difference between

[0084] Global GeometryFeature module

[0085] Point cloud density, as an intrinsic geometric property of point clouds, reflects the variation in the spacing within the point cloud. Dramatic changes in density typically indicate variations in the object to which that point belongs. This characteristic can add additional geometric information to point cloud features, thereby improving segmentation performance.

[0086] Will Defined as the ratio of the local to the global volume of a sphere, such as Figure 5 As shown. Where, v k Point p i The volume of the local sphere formed by the k nearest neighbors, v g This is the volume of a sphere containing the input point cloud. The volume will be determined locally at a distance p from point p. i The distance from the farthest point is used as the local sphere radius. To calculate the global sphere radius, the centroid of the input point cloud needs to be calculated, and the distance from the farthest point within the input point cloud to the centroid is used as the global sphere radius. The local and global sphere volumes can be obtained using equation (7).

[0087]

[0088] Reversed Residual MLP

[0089] This invention utilizes the RevResMLP module to mine deeper semantic information. Traditional residual structures first reduce the dimensionality of input features and then increase it, while inverse residual structures increase the dimensionality of input features and then reduce it. By increasing the dimensionality of input features, richer feature information is learned. This module is added after the GRes module to mine deeper semantic information.

[0090] Dataset

[0091] This invention validates the effectiveness of the GRes-Net network on the S3DIS dataset. S3DIS is a high-quality indoor scene dataset, approximately 14.2GB in size, containing 6 different indoor areas, covering a total area of ​​approximately 6020m². 2The dataset contains 272 rooms. It has 13 categories (walls, floors, chairs, etc.). A 1m × 1m area is randomly selected based on the plane formed by the X and Y axes.

[0092] 4096 points were randomly selected from this region as input data. The experiment was conducted on a server equipped with an NVIDIA GeForce RTX 3080 GPU, using the PyTorch framework.

[0093] Experimental parameter settings

[0094] The network was trained for a total of 100 epochs. This invention uses the AdamW optimizer with an initial learning rate of 0.01 and employs a cosine annealing algorithm to gradually decay the learning rate. The number of neighbor points was set to 16, i.e., k = 16. During training, the batch size was set to 16; during testing, the batch size was also set to 16.

[0095] Experimental results

[0096] As shown in Table 1, the GRes-Net network achieves the following segmentation metrics on S3DISArea5: mIOU 61%, OA 86.7%, and mACC 68.7%. Compared to previous methods, especially to graph convolutional algorithms DGCNN and DeepGCN, the mIOU metrics are improved by 14% and 8.5%, respectively, validating the effectiveness of the proposed method. Furthermore, to visually represent the segmentation results, this invention visualizes some rooms within the S3DISArea5 dataset, such as... Figure 7 As shown.

[0097] Table 1. Results of different methods on S3DISArea5.

[0098]

[0099] From left to right: input point cloud data, ground truth semantics, DGCNN predicted semantics, and GRes-Net predicted semantics.

[0100] ablation experiment

[0101] To verify the effectiveness of each module in the network of this invention, ablation experiments were conducted on the S3DISArea5 dataset. As shown in Table 2, removing all modules resulted in a 7.6% decrease in segmentation performance. The first to third rows illustrate that adding the LGA and GGF modules can improve model performance. Compared to the results without the RevResMLP module (i.e., compared to the third and fourth rows), adding the RevResMLP module resulted in a 5.4% increase in segmentation performance, indicating that this module can significantly improve model performance.

[0102] Table 2 shows the ablation experiment results on S3DISArea5.

[0103] LGA GGF RevResMLP Result (mIOU%) 53.4 √ 54.9 √ √ 55.6 √ √ √ 61.0

[0104] To address the problems of weak local representation capabilities, lack of global geometric information, and retention of only the maximum response value in local aggregation of existing graph convolutional point cloud segmentation networks, this invention proposes the GRes-Net network. By introducing LGA and GGF modules, it enhances local information of the point cloud and preserves global geometric information during graph convolution. It employs multiple aggregation operations to retain the maximum response value and other information within the local area of ​​the point cloud, while utilizing the RevResMLP module to mine deeper semantic information.

[0105] The proposed network was evaluated on S3DISArea5, achieving a segmentation accuracy of 61%. Compared to DGCNN, DeepGCN, and PointWeb, its semantic segmentation performance was improved by 14%, 8.5%, and 0.7%, respectively. Ablation experiments on the LGA, GGF, and RevResMLP modules showed that these three modules effectively improved the model's segmentation performance.

[0106] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions conceived without inventive effort should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A convolutional network for point cloud semantic segmentation that enhances both local and global semantic segmentation, characterized in that: An encoder-decoder architecture is adopted; the Dynamic Graph Convolutional Neural Network (DGCNN) uses point cloud coordinates and color information as features, and performs a k-nearest neighbor search on each point using these features. Then, it calculates the difference between the features of each point and its neighbors, concatenates the center point features with this difference, and constructs a new feature map. A Multilayer Perceptron (MLP) and max pooling are used to learn the local feature information of the new feature map; this includes a Local Geometry Enhancement (LGA) module, a Global Geometry Feature (GGF) module, multiple aggregation operations, and a RevResMLP module. The LGA module is used to enhance the local feature representation of the point cloud; The GGF module is used to retain global geometric information; The multiple aggregation operation modules are used to retain the local maximum response value and other information; The RevResMLP module is used to mine deeper information. The LGA module obtains a relative polar coordinate representation by calculating the relative angle between each point in the local area and the local centroid. The GGF module obtains the volume ratio of the local to the global point cloud and connects it with the point cloud coordinate features; The multiple aggregation operations connect the results of max pooling and summation pooling; The inverse residual module RevResMLP first increases the dimensionality of the input features and then reduces it, and then adds the output result to the input data.

2. A method for operating a convolutional network as described in claim 1, characterized in that, Includes the following steps: Step 1: Input point cloud data and corresponding hyperparameters; Step 2: Perform point cloud semantic segmentation. The sampled points are fed into the encoder to encode the point cloud. Step 3: Enhance local point cloud geometric information and preserve global point cloud geometric information in the GRes module. The LGA module enhances local feature representations of the point cloud, while the GGF module preserves global geometric information. Details are as follows: The steps for building an LGA module are as follows: 1) Calculate the distances between all points within the local region. With angle As shown in formulas (1) to (3); For point p i With point The distance between them For point p i With point The angle between them; definition: For point p i The k nearest neighbors, where i represents a local region; in, It is point p i Coordinates in the Cartesian coordinate system; 2) Calculate the angle of the local incenter of mass. The centroid is calculated using the coordinates of points in a local region. Calculate its angle using formulas (2) and (3); 3) Calculate the relative angles between the centroid and all points within the local area. As shown in formulas (4) and (5); The steps for using the GGF module are as follows: The volume of the global point cloud and the volume of the local point cloud are calculated according to formula (6), and then the ratio of the volume of the local point cloud to the volume of the global point cloud is calculated according to formula (7). Step 4: Apply multiple aggregation operations to preserve the local maximum response value and other information, as shown in formulas (8), (9) and (10); Info max =max(input)(8) Info sum =sum(input)(9) Info=Concat(Info max ,Info sum )(10) Where, input is the input feature data; max and sum are the max pooling and summation operations, respectively; and the Concat operation is used to convert Info... max With Info sum To splice; Step 5: Explore deeper information in the RevResMLP module, as shown in formulas (11), (12) and (13); input knn =KNN(input) (11) Wherein, input is the input feature data; input knn Local data obtained after performing a K-nearest neighbor search on the input; It is a multilayer perceptron; Step 6: Use the furthest sampling point to perform downsampling operation at FPS; Step 7: Use the output point cloud as input to the decoder for decoding; Step 8: Upsample the input point cloud and then connect it to the output of the corresponding layer in the encoder; Step 9: Pass the results through a two-layer multilayer perceptron and output them.

Images