A semantic segmentation method for 3D point clouds based on coordinate offset and feature enhancement

Abstract

This invention discloses a 3D point cloud semantic segmentation method based on coordinate offset and feature enhancement. To overcome the problems of outliers and overlapping points being collected during the neighborhood construction process of point clouds, and the lack of directionality and insufficient global representation in the features extracted by existing encoders, this invention proposes a 3D point cloud semantic segmentation method based on coordinate offset and feature enhancement. Based on the RandLA-Net network framework, this invention uses 3D point clouds as input data, offsets the geometric coordinates based on additional features, and then uses the offset geometric coordinates to achieve the offset of additional features, thereby mitigating the influence of outliers and sampling points. In the encoder, polar coordinate representation is introduced, establishing polar coordinate direction representations for each point and its global neighborhood, thus representing the directionality within the point cloud. After extracting local features, the ratio of the local neighborhood to the global neighborhood is calculated, and this ratio is multiplied by the features to complete the representation of global information.

Description

Technical Field

[0001] This invention relates to the field of semantic segmentation of 3D point clouds in computers, and is based on deep learning methods to achieve semantic segmentation of 3D point clouds. Background Technology

[0002] In recent years, with the development of various 3D sensors (such as LiDAR, RGB-D cameras, etc.), 3D data has become easier to acquire. Furthermore, with the rise of autonomous driving, understanding real-world 3D scenes has become paramount, and semantic segmentation of 3D point clouds is an indispensable step in this understanding. Traditional point cloud semantic segmentation methods can be categorized into multi-view based, projection-based, and voxel-based methods. However, these methods inevitably lead to information loss during data conversion. Therefore, point-based methods are increasingly being adopted by researchers. This invention focuses on a point cloud semantic segmentation method based on Multiprocessor Language (MLP). Existing MLP-based techniques often use fixed metrics (such as Euclidean distance) to represent neighborhood points, inevitably resulting in outliers and overlapping points. Furthermore, previous point cloud semantic segmentation work uses geometric features and additional features (such as color and normal vector information) as input. Geometric features typically only use Euclidean distance, which cannot represent the orientation within the point cloud. In real-world 3D point cloud scenes, orientation is crucial information; the same object can lead to different segmentation results depending on its orientation. Currently popular encoder-decoder frameworks primarily extract local features from the encoder, failing to adequately represent global features. While some works employ residual networks to combine the original feature information with the local features output by the encoder, this representation is insufficient and lacks a comprehensive characterization of global information. Summary of the Invention

[0003] To overcome the problems of outliers and overlapping points being collected during the neighborhood construction process of point clouds, and the lack of directionality and insufficient global representation of features extracted by existing encoders, this invention proposes a 3D point cloud semantic segmentation method based on coordinate offset and feature enhancement. This method uses the RandLA-Net network as the backbone network, and the network structure of RandLA-Net is as follows: Figure 1As shown, this method uses 3D point clouds as input data. After constructing neighborhood points using KNN indexing, it offsets the geometric coordinates based on additional features. The offset geometric coordinates are then used to achieve the offset of additional features, thereby mitigating the influence of outliers and sampling points. In the encoder, to address the lack of directional features, polar coordinate representation is introduced. Polar coordinate directional representations are established for each point and its global neighborhood to represent the directionality within the point cloud. After extracting local features, the ratio of the local neighborhood to the global feature is calculated, and this ratio is multiplied point-by-point with the original features to complete the representation of global information. The framework diagram of point cloud semantic segmentation technology based on coordinate offset and feature enhancement is shown below. Figure 2 As shown. Experiments were conducted on the LASDU dataset based on the method proposed in this invention, and the quantitative results obtained are as follows. Figure 3 As shown, the qualitative results are as follows: Figure 4 As shown.

[0004] The technical solution adopted in this invention is based on coordinate offset and feature enhancement for 3D point cloud semantic segmentation. The method includes:

[0005] Step 1: Find the neighborhood points of the sampling point and obtain their geometric coordinates and additional features. Then, reduce the dimensionality of the additional features to the same dimension as the geometric coordinates to complete the coordinate offset. Then, apply the offset coordinates to the additional features to complete the feature offset, thereby reducing the impact of outliers and overlapping points.

[0006] Step 2: Input the offset coordinates and features, and the original coordinates and features into the encoder. Use the information of both as the geometric information in Euclidean space. Then, for each neighboring point, calculate its polar coordinates to represent the orientation information of each point. For each neighborhood space, use the sampling points to calculate the orientation information of the neighborhood as a whole. Use the information obtained by subtracting the two as the local orientation information.

[0007] Step 3: In order to represent global information, the sampling point coordinates of the corresponding layer input of the encoder are treated as a whole. The volume ratio of the neighborhood sphere to the global sphere of each neighborhood is calculated. This ratio is then used as the ratio of each neighborhood. The ratio is then multiplied point by point with the coordinates and features, and the residual is connected with the original coordinates and features to represent the global information.

[0008] Compared with the prior art, the beneficial effects of the present invention are:

[0009] (1) Previous point cloud semantic segmentation algorithms did not take into account the influence of outliers and overlapping points in the neighborhood. This invention uses features to offset coordinates, and then uses the offset coordinates to offset features, thereby reducing the influence of outliers and overlapping points.

[0010] (2) Previous point cloud semantic segmentation algorithms lacked global representation and insufficient representation of directional information in local areas. This invention uses polar coordinate representation in the encoder to characterize global information; after extracting local features, it calculates the ratio of local neighborhood to global features to aggregate global features and performs residual connection with the original features, thus retaining more useful information. Attached Figure Description

[0011] Appendix Figure 1 : Network framework diagram of RandLA-Net.

[0012] Appendix Figure 2 The improved network framework diagram of this invention.

[0013] Appendix Figure 3 Results of RandLA-Net and the method proposed in this invention implemented on the LASDU dataset.

[0014] Appendix Figure 4 The results of visualizing the method proposed in this invention on the LASDU dataset. Detailed Implementation

[0015] The invention will now be further described with reference to the accompanying drawings.

[0016] Step 1: Using p j f represents the geometric information of neighboring points j p represents the semantic information of neighboring points. i f i Let the geometric and semantic information of the center point be respectively represented. And by offsetting the geometric coordinate information accordingly, we obtain... MLP stands for shared multilayer perceptron;

[0017] Step 2: Based on the offset geometric coordinate information obtained in Step 1, use a shared Multilayer Perceptron (MLP) to... The dimension is increased to f j Same dimension, i.e.

[0018] Step 3: To represent local direction information, we use the relative coordinates of each point. Let the subscript i be the coordinates of the center point and j be the global coordinates. Therefore, the relative coordinates are (x... j -x i ,y j -y i ,z j -z i Therefore, its polar coordinates are represented as

[0019] Step 4: After obtaining the local direction representation, use the coordinates of the centroid of the neighborhood sphere to calculate the direction of the global neighborhood. The centroid is... Using the centroid, calculate the direction θ of the global neighborhood, and then express the direction of the neighborhood as θ' ​​= θ - θ i ;

[0020] Step 5: Based on the above information, the geometric features are as follows: Therefore, the local representation in the encoder is:

[0021] Step 6: To characterize the ratio of each neighborhood sphere in the overall point cloud space, calculate the maximum volume ratio of each neighborhood sphere to the overall point cloud space, and calculate the radius of the neighborhood sphere using relative coordinates. Calculate the Euclidean distance of the overall point cloud using the coordinates of the sampling points, i.e. Therefore, the ratio is

[0022] Step 7: After calculating the ratio of each neighborhood sphere to the overall point cloud, multiply it by the original point cloud features to obtain v. f =MLP(v i ·f i );

[0023] Step 8: Combining the information from the local encoder, the residual information from the original features, and the global information, the final point cloud features are encoded as G = concat(G i ,f i ,v f );

[0024] Step 9: Use the output of Step 8 as the input of the next layer, and then repeat Steps 1-8 in the encoding of the next layer.

Claims

1. A 3D point cloud semantic segmentation method based on coordinate offset and feature enhancement, characterized in that, Includes the following steps: Step 1: Using p j f represents the geometric information of neighboring points j p represents the semantic information of neighboring points. i f i Let the geometric and semantic information of the center point be respectively represented. And by offsetting the geometric coordinate information accordingly, we obtain... MLP stands for shared multilayer perceptron; Step 2: Based on the offset geometric coordinate information obtained in Step 1, use a shared Multilayer Perceptron (MLP) to... The dimension is increased to f j Same dimension, i.e. Step 3: To represent local direction information, we use the relative coordinates of each point. Let the subscript i be the coordinates of the center point and j be the global coordinates. Therefore, the relative coordinates are (x... j -x i ,y j -y i ,z j -z i Therefore, its polar coordinates are represented as Step 4: After obtaining the local direction representation, use the coordinates of the centroid of the neighborhood sphere to calculate the direction of the global neighborhood. The centroid is... Using the centroid, calculate the direction θ of the global neighborhood, and then express the direction of the neighborhood as θ' ​​= θ - θ i ; Step 5: Based on the above information, the geometric features are as follows: Therefore, the local representation in the encoder is: Step 6: To characterize the ratio of each neighborhood sphere in the overall point cloud space, calculate the maximum volume ratio of each neighborhood sphere to the overall point cloud space, and calculate the radius of the neighborhood sphere using relative coordinates. Calculate the Euclidean distance of the overall point cloud using the coordinates of the sampling points, i.e. Therefore, the ratio is Step 7: After calculating the ratio of each neighborhood sphere to the overall point cloud, multiply it by the original point cloud features to obtain v. f =MLP(v i ·f i ); Step 8: Combining the information from the local encoder, the residual information from the original features, and the global information, the final point cloud features are encoded as G = concat(G i ,f i ,v f ).

2. The method as described in claim 1, characterized in that, The direction of the neighborhood obtained in step 4 is the difference between the overall direction of the neighborhood and the direction of each point within the neighborhood.

3. The method as described in claim 1, characterized in that, The additional features combined in step 5 are the features obtained after offsetting.

4. The method as described in claim 1, characterized in that, The ratio obtained in step 6 is derived from the maximum radius within the neighborhood.

Images