Point cloud semantic segmentation method based on two-dimensional and three-dimensional convolution

Abstract

The invention discloses a point cloud semantic segmentation method based on two-dimensional and three-dimensional convolution. Aiming at the problem that the dependency relationship between pixels or points is explored and the relationship between objects is neglected in the existing method, and considering that the variance of ALS point cloud coordinates on an XY plane is greater than the variance along a Z-axis direction, firstly, the representativeness of features obtained by 3D convolution is enhanced by adding 2D convolution; in this way, the distribution of points on the XY plane can be more concerned. In the feature extraction module, the original single 3D KPConv operator in the KPConv is replaced by 2D KPConv and 3D KPConv, and features of more representative points are extracted by two-dimensional and three-dimensional convolution information; at the object level, a spatial channel attention mechanism is applied to account for global information. According to the method, information among all point pairs is obtained by applying a space attention matrix, and the output of two-dimensional and three-dimensional convolution is embedded among the point pairs, so that information gain is carried out, the relation between points and objects is improved, and semantic consistency is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and realizes semantic segmentation of three-dimensional point clouds based on a deep learning method. Background Art

[0002] With the development of LiDAR technology, it has become easier to obtain point cloud data in various scenarios. Airborne laser scanning (ALS) point clouds have become an indispensable data type in the generation of digital terrain models (DTMs), landscape models, 3D city models, and land use maps. These point cloud-based products are required in many disciplines, such as urban planning, land management, forest surveys, tourism, and disaster management. The interpretation of ALS point clouds is a prerequisite for their use in these applications. One of the interpretation methods is semantic segmentation, which is to label each point in the dataset with a semantic label. Inspired by the success of deep learning in image-related tasks, many works have explored deep learning-based methods for 3D interpretation tasks, such as image-based methods, voxel-based methods, and point-based methods. These methods have achieved good results in exploring local features, but they are still insufficient in extracting global information. Some existing methods use fully connected CRF and attention modules to enforce global consistency and optimize semantic predictions of images and point clouds. However, these methods explore the dependencies between pixels or points, but ignore the relationships between objects, which are useful for large-scale and complex outdoor scenes. Summary of the invention

[0003] In order to overcome the problem of not being able to fully utilize the global context information at the point and object levels in the point cloud, this paper proposes a point cloud semantic segmentation method based on two-dimensional and three-dimensional convolution. This method takes into account that the variance of the ALS point cloud coordinates in the XY plane is greater than the variance along the Z axis, so firstly, 2D convolution is added to enhance the representativeness of the features obtained by 3D convolution, so as to pay more attention to the point distribution on the XY plane. In terms of three-dimensional information representation, this method uses KPConv as the basic network. The network framework of KPConv is shown in the figure below. Figure 1 The present invention replaces the original single 3D KPConv operator in KPConv with 2D KPConv and 3D KPConv, and uses the two-dimensional and three-dimensional convolution information to extract more representative point features. The structure diagram of the two-dimensional and three-dimensional joint convolution is shown in Figure 2 As shown. In addition, at the object level, a spatial channel attention mechanism is applied to consider global information. By applying the spatial attention matrix, the information between all point pairs is obtained, and the output of the two-dimensional and three-dimensional convolution is embedded between the point pairs to perform information gain, improve the connection between points and objects, and enhance semantic consistency. Figure 3 Shows the structural diagram of the spatial attention mechanism.

[0004] The technical solution adopted by the present invention is to perform point cloud semantic segmentation based on two-dimensional and three-dimensional convolution. The method includes:

[0005] Step 1: Construct a 2D KPConv convolution kernel based on the 3D KPconv convolution kernel, use 2DKPConv to enhance the point cloud information, and then use the 3D KPConv convolution kernel to extract features from the enhanced point cloud;

[0006] Step 2: In order to model the relationship between any two members in the point cloud, given the input feature matrix F of the point set, F is projected into three different feature subspaces to form query, keyword and value, and the spatial attention matrix is ​​calculated based on this;

[0007] Step 3: Embed the features output from step 1 into the point pair information after spatial attention modeling in step 2 to enhance the feature information, improve the connection between points and objects, and enhance semantic consistency.

[0008] Compared with the prior art, the present invention has the following beneficial effects:

[0009] (1) Previous point cloud semantic segmentation algorithms did not consider the distribution characteristics of airborne laser scanning point clouds in the XY space, and directly input the original point cloud, causing the network to directly perform the same learning on the three different directions of XYZ, ignoring the geometric orientation of the point cloud information itself. The present invention uses 2D convolution to enhance the point cloud in the XY plane, thereby improving the representativeness of the point cloud information;

[0010] (2) Previous point cloud semantic segmentation algorithms only explored the dependency between pixels or points, but ignored the relationship between objects. The present invention adopts a method of fusing two-dimensional and three-dimensional convolution with spatial attention to embed the output features of two-dimensional and three-dimensional convolution into the point pair information extracted by spatial attention, thereby improving the relationship between points and objects and enhancing semantic consistency. Description of the drawings:

[0011] Attached Figure 1 : The basic framework of KPConv network.

[0012] Attached Figure 2 : Framework diagram of 2D and 3D joint convolution.

[0013] Attached Figure 3 : Framework diagram of the spatial attention module. Specific implementation method:

[0014] The present invention will be further described below in conjunction with the accompanying drawings.

[0015] Step 1: Given a point cloud P and the corresponding feature F, at point p, the point convolution of F with the 3D KPConv convolution kernel function g is written as

[0016]

[0017] Where N p is a set of neighbors around p within a fixed radius r∈R, N p ={p i ∈P|||p i -p||≤r}. i is one of the neighbors of p in the point set P, and its corresponding feature is f i For simplicity, let the input of function g be x i =p i -p, and x i is the relative position of the neighboring points relative to the center point p, and N′ is the number of neighboring points. Represents the g domain of 3D KPConv, which is a 3D spherical space with p as the center and radius r. Similar to the image convolution kernel, the kernel function g has different weights for different parts in the kernel domain. The different regions in A set of nuclear point locations. k yes is a three-dimensional position in , K is the number of kernel points of the kernel function g. The corresponding weight matrix of the kernel point is expressed as {W pk |k <K}。

[0018] Any input The kernel function g is defined as

[0019]

[0020] where h is p k and x i There is a linear correlation between i Close to the kth core point p k When h is larger, the 2D kernel function l is very similar to the 3D kernel function g, except that N q and the nuclear point position q k The point convolution of the 2D kernel function l at q (the projection of point p on the XY plane) is defined as

[0021]

[0022] N q YesN p 2D projection of . Therefore, N q and N p have the same number of points N′. Define the input of function l as yi = q i -q and y i is q i The relative position with respect to the center point q is the domain of l, where l is a two-dimensional circular surface centered at q with a radius of r The two-dimensional kernel points in are written as q o is The 2D coordinates in, O is the number of kernel points of the 2D kernel function l. The corresponding weight matrix of the 2D kernel points is represented as {W qo | o < O}, and the 2D kernel function l is defined as

[0023]

[0024] Step 2: Combine the 2D KPConv convolution kernel and the 3D KPConv convolution kernel obtained in Step 1 to replace the single 3D KPConv convolution kernel in the original KPConv operator, so as to extract more representative features F′

[0025] Step 3: Project the F′ extracted in Step 2 into three different feature subspaces to form queries, keys, and values, namely U, V, and T, which are calculated as follows: U = α s (F′), V = β s (F′), T = γ s (F′). Where α s , β s and γ s are transformation functions implemented through different fully connected (FC) layers

[0026] Step 4: Use the dot product between V j and the transpose of U i to represent the correlation between points i and j in the point cloud, that is

[0027]

[0028] where sa ij is the normalized spatial attention map, which estimates the influence of point j on point i and forms the spatial attention matrix SA

[0029] Step 5: Perform matrix multiplication between the spatial attention SA and the value T projected into the sub-feature space, multiply by a learnable scale parameter α, and then sum element-wise with the output feature F′

[0030] F sa = αSAA · T + F′

[0031] The obtained feature Fsa That is, the output features of two-dimensional and three-dimensional convolution are embedded into the point pair information extracted by spatial attention.

Claims

1. A point cloud semantic segmentation method based on two-dimensional and three-dimensional convolution, which is characterized by the following steps: Step 1: Given a point cloud P and the corresponding feature F, at point p, the point convolution of F with the 3D KPConv convolution kernel function g is written as where N p is a set of neighbors around p within a fixed radius r ∈ R, and N p ={p i ∈ P ||| p i - p|| ≤ r}. p i is one of the neighborhoods of p in the point set P, and its corresponding feature is f i . For simplicity, the input of the function g is set to x i = p i - p, and x i is the relative position of the adjacent point with respect to the central point p, and N′ is the number of adjacent points. denotes the g domain of 3D KPConv, which is a 3D spherical space centered at p with a radius of r. Similar to the image convolution kernel, the kernel function g has different weights for different parts within the kernel domain. The different regions in are located by a set of kernel points of k . p is a three - dimensional position in pk , and K is the number of kernel points of the kernel function g. The corresponding weight matrix of the kernel points is denoted as {W | k < K}. The kernel function g for any input where h is p k and x i There is a linear correlation between i Close to the kth core point p k When h is larger, the 2D kernel function l is very similar to the 3D kernel function g, except that N q and the nuclear point position q k The point convolution of the 2D kernel function l at q (the projection of point p on the XY plane) is defined as N q is the 2D projection of N p . Therefore, N q and N p have the same number of points N′. Define the input of the function l as y i =q i -q and y i is the relative position of q i relative to the center point q, which is the domain of l, and l is a two-dimensional circular surface centered at q with a radius of r The two-dimensional kernel points in q o are the 2D coordinates in, O is the number of kernel points of the 2D kernel function l. The corresponding weight matrix of the 2D kernel points is denoted as {W qo |o < O}, and the 2D kernel function l is defined as Step 2: Combine the 2D KPConv convolution kernel and the 3D KPConv convolution kernel obtained in step 1 to replace the single 3D KPConv convolution kernel in the original KPConv operator to extract a more representative feature F′; Step 3: Project F′ extracted in step 2 into three different feature subspaces to form query, keyword and value, which are U, V, and T respectively. U, V and T are calculated as follows: U = α s (F′), V = β s (F′), T = γ s (F′). s , β s and γ s is the transformation function implemented by different fully connected (FC) layers; Step 4: Using V j and U i The dot product between the transposes is used to represent the correlation between points i and j in the point cloud, that is Among them ij is the normalized spatial attention map, which estimates the influence of point j on point i, forming the spatial attention matrix SA; Step 5: Perform matrix multiplication between the spatial attention SA and the value T projected into the sub-feature space, multiply by a learnable scale parameter α, and then sum element-wise with the output feature F′ F sa =αSAA·T+F′ The obtained feature F sa That is, the output features of two-dimensional and three-dimensional convolution are embedded into the point pair information extracted by spatial attention.

2. The method according to claim 1, characterized in that: The convolution kernel used in step 2 is the result of combining the 3D KPConv convolution kernel with the 2DKPConv convolution kernel.

3. The method according to claim 1, characterized in that: In step 3, the output feature F′ is projected into three sub-feature spaces respectively, instead of using the original feature F.

4. The method according to claim 1, characterized in that: The final output in step 5 is the result of element-wise summing of the projected value and the output feature.

Images