Three-dimensional point cloud semantic segmentation method for constructing curved surface based on Taylor series

Abstract

The invention discloses a three-dimensional point cloud semantic segmentation method for constructing a curved surface based on Taylor series. Previous point-based point cloud semantic segmentation networks indirectly learn from geometry by adding more information, such as Euclidean distance, attention mechanism, or applying various transformations, such as graphic construction, voxelization. These operations may result in complex pre-processing and large amounts of calculations. Point Net + + is used as a basic network, 3D point cloud is used as input data, point cloud down-sampling is carried out through farthest point sampling, and after a down-sampling center point is obtained, neighborhood points of the point cloud are obtained through ball query. In the local geometrical shape feature extraction module, the Taylor series is utilized to convert the point cloud into a triangular curved surface, so that the problem of information loss caused by a conventional local geometrical shape information extraction method or deformation of a local detail geometrical shape in a transformation process is relieved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and realizes semantic segmentation of three-dimensional point clouds based on a deep learning method. Background Art

[0002] In recent years, with the development of various 3D sensors (such as LiDAR, RGB-D cameras, etc.), 3D data has become easier to obtain, and various 3D point cloud datasets have sprung up. As data acquisition is easier, a large number of works based on deep learning for point cloud semantic segmentation have been proposed. These methods can be roughly divided into multi-view based methods, voxel based methods, and point based methods. Multi-view methods describe 3D objects with multiple views, that is, converting 3D shapes into 2D images and grid spaces. Voxel-based methods voxelize point clouds, such as Oc-Tree-based networks O-CNN and OctNet, which use efficient sub-manifold sparse convolutions to process point clouds. However, these transformation methods may result in a lot of computation and loss of geometric information due to occlusion or lower resolution. Point-based methods directly process point clouds, such as PointNet, which learns from global information through multi-layer perceptrons (MLPs) and maximum pooling operations. PointNet++ introduces set abstraction to capture features from local point sets, and introduces farthest point sampling to uniformly downsample between two set abstractions. However, these methods only focus on the information of a single point, without considering the local geometric shape connection. Some works have realized the importance of local geometry for point cloud learning. In order to learn from local structural information, some studies learn from grids, distance relationships, or graphics. However, these methods learn from geometry indirectly by adding more information (such as Euclidean distance, attention mechanism) or applying various transformations (such as graphics construction, voxelization). These operations may lead to complex preprocessing and a lot of calculations. Moreover, these complex artificial geometric features are usually learned from implicit local shape representations. Such local geometry learning methods will inevitably lead to information loss, or deformation or loss of local detail geometry during the transformation process. Summary of the invention

[0003] In order to overcome the shortcomings of existing local geometric shape contact information extraction methods, which lead to information loss or deformation of local detail geometric shapes, this paper proposes a 3D point cloud semantic segmentation method based on Taylor series surface construction. This method uses PointNet++ as the basic network. The network framework of PointNet++ is as follows Figure 1 As shown, the present invention uses 3D point cloud as input data, uses farthest point sampling to downsample the point cloud, obtains the center point of the downsampled point, and then uses ball query to obtain the neighborhood points of the point cloud. In the local geometric shape feature extraction module, the point cloud is converted into a triangular surface using Taylor series. Figure 2 The construction process of the triangular surface is shown in Figure 1. Finally, the geometric shape features of the triangular surface are embedded into the geometric feature module of PointNet++ as the output of local feature extraction, so as to perform point cloud features in the next layer. The local feature extractor after embedding the geometric shape features of the triangular surface into PointNet++ is shown in Figure 1. Figure 3 shown.

[0004] The technical solution adopted by the present invention is to construct a surface based on Taylor series to perform semantic segmentation of point cloud. The method includes:

[0005] Step 1: Use the farthest point sampling to obtain the center point of the downsampled point cloud, use the ball query to obtain the neighborhood points of the downsampled center point, and then input the neighborhood point cloud into the local feature extractor of PointNet++ to separate the geometric information of the point cloud from other additional information;

[0006] Step 2: Use the neighborhood point cloud geometry information and normal vector information obtained in step 1 to construct the triangular surface of the point cloud using Taylor series, and then calculate the centroid coordinates of the original point cloud. The final output point cloud geometry feature is the sum of the normal vector information, the distance information from the original point cloud to the surface, and the centroid coordinates of the point cloud.

[0007] Step 3: Use the original local feature extractor of PointNet++ to extract the additional information of the neighborhood point cloud, then add it to the geometric information features obtained in step 2, and finally use maximum pooling to obtain the aggregated neighborhood features, which are used as the original features for the next layer of the network.

[0008] Compared with the prior art, the present invention has the following beneficial effects:

[0009] (1) Previous point cloud semantic segmentation algorithms used complex manual designs to implicitly learn local geometric shape feature information of point clouds from local parts, which resulted in information loss or deformation of detailed local geometric shapes. The present invention uses Taylor series to construct triangular surfaces, completes the characterization of local detailed geometric shape information of point clouds, and enables the network to extract more representative geometric shape features;

[0010] (2) The triangular surface feature constructed based on Taylor series proposed in the present invention has a small amount of calculation, can be easily inserted into different networks, and has good applicability to different methods. Description of the drawings:

[0011] Attached Figure 1 : The basic framework of PointNet++ network.

[0012] Attached Figure 2 : The process of constructing a triangular surface.

[0013] Attached Figure 3 : Framework diagram after the triangular surface features are embedded in PointNet++. Specific implementation method:

[0014] The present invention will be further described below in conjunction with the accompanying drawings.

[0015] Step 1: As attached Figure 1 As shown, the present invention uses PointNet++ as a classifier and the farthest point sampling as a downsampling algorithm. For the selected point set, the point farthest from the point set among the remaining points is calculated and used as the candidate point obtained by downsampling. After the downsampling is completed, the ball query algorithm is used to calculate N neighborhood points for each center point obtained by downsampling.

[0016] Step 2: For the local neighborhood centered at the point (t,f(t)) of the 2D curve f(·), the Taylor series can be expressed as

[0017]

[0018] For the convenience of calculation, it can be simplified as

[0019]

[0020] Where (t, f(t)) is the global position on the curve f(·), and the first-order derivative f'(t) can intuitively represent the local orientation near the point (t, f(t)). To further express the local curve, the corresponding tangent direction is used to represent the local direction

[0021]

[0022] where x i =t,y i =f(t),a i / b i =-f'(a). (a i ,b i ) is the normal vector of the tangent line, where a i 2 +b i 2 = 1. That is, the rough description of the local curve can be defined as: c i =(x i ,y i ,a i ,b i ,a i x i +b i y i );

[0023] Step 3: Represent the point set as X = {x1 ,…,x n}, try to vector v i =(a i ,b i ,c i ) and point X i =(x i ,y i ,z i ), the surface is defined as a i (xx i )+b i (yy i )+c i (zz i )=0, that is, a i x+b i y+c i z-(a i x i +b i y i +c i z i )=0, so the surface position can be expressed as p i =a i x i +b i y i +c i z i ;

[0024] Step 4: Calculate the normal vector v using the cross product i , the calculated v i is directionless, that is, v i It can point to the inside or outside of the surface. For the sake of uniformity and efficiency, it will keep a i is positive, so the final calculated triangular surface is represented by t i =(a i ,b i ,c i ,p i );

[0025] Step 5: Denote a set of triangular surfaces as T = {t 1 ,…,t n}, and replace X with the recalculated triangle centroid X'. The final output geometric feature is the concatenation of X' and T;

[0026] Step 6: Use the local feature extractor of PointNet++ to extract the additional information of the original point cloud, concatenate it with the output of step 5, and then perform maximum pooling to use it as the final output feature.

Claims

1. A 3D point cloud semantic segmentation method based on Taylor series surface construction, which is characterized by the following steps: Step 1: Use PointNet++ as the classifier and the farthest point sampling as the downsampling algorithm. For the selected point set, calculate the point farthest from the remaining points in the point set and use it as the candidate point obtained by downsampling. After the downsampling is completed, use the ball query algorithm to calculate N neighborhood points for each center point obtained by downsampling. Step 2: For the local neighborhood centered at the point (t,f(t)) of the 2D curve f(·), the Taylor series can be expressed as For the convenience of calculation, it can be simplified as Where (t, f(t)) is the global position on the curve f(·), and the first-order derivative f'(t) can intuitively represent the local orientation near the point (t, f(t)). To further express the local curve, the corresponding tangent direction is used to represent the local direction where x i =t,y i =f(t),a i / b i =-f'(a). (a i ,b i ) is the normal vector of the tangent line, where a i 2 +b i 2 = 1. That is, the rough description of the local curve can be defined as: c i =(x i ,y i ,a i ,b i ,a i x i +b i y i ); Step 3: Represent the point set as X = {x1,…,x n }, try to vector v i =(a i ,b i ,c i ) and point X i =(x i ,y i ,z i ), the surface is defined as a i (xx i )+b i (yy i )+c i (zz i )=0, that is, a i x+b i y+c i z-(a i x i +b i y i +c i z i )=0, so the surface position can be expressed as p i =a i x i +b i y i +c i z i ; Step 4: Calculate the normal vector v using the cross product i , the calculated v i is directionless, that is, v i It can point to the inside or outside of the surface. For the sake of uniformity and efficiency, it will keep a i is positive, so the final calculated triangular surface is represented by t i =(a i ,b i ,c i ,p i ); Step 5: Denote a set of triangular surfaces as T = {t1,…,t n }, and replace X with the recalculated triangle centroid X'. The final output geometric feature is the concatenation of X' and T; Step 6: Use the local feature extractor of PointNet++ to extract the additional information of the original point cloud, concatenate it with the output of step 5, and then perform maximum pooling to use it as the final output feature.

2. The method according to claim 1, characterized in that: When using the Taylor series to calculate the point cloud surface in step 2, only the first-order derivative is used.

3. The method according to claim 1, characterized in that: In step 4, a i Always keep it positive to ensure that the calculated normal vectors have a uniform direction.

4. The method according to claim 1, characterized in that: In step 5, the input samples of the original point cloud are replaced with the centroid coordinates of the triangular surface, and the centroid coordinates are used to replace the original point cloud.

Images