A point cloud semantic segmentation network based on dynamic graph convolution and spatial attention

Abstract

The application discloses a point cloud semantic segmentation network based on dynamic graph convolution and spatial attention, and relates to the technical field of point cloud semantic segmentation feature extraction.The application comprises the following steps: inputting F-dimensional point cloud X of N points; and constructing a local dynamic graph G=(V, epsilon).The application uses the DenseNet idea to splice multiple levels of features to improve the fitting capacity of the network; uses a pooling method combining maximum pooling and average pooling to obtain the most representative semantic information on all local graphs; uses a spatial attention mechanism to mine spatial semantic information and extract deep semantic correlation between points; removes a spatial conversion network while proposing a new edge feature to reduce the parameter quantity of the network; and the designed GEConv module can enrich the diversity of features, and the introduced spatial attention module can enable the network to learn to pay more attention to important information.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of point cloud semantic segmentation feature extraction, and particularly relates to a point cloud semantic segmentation network based on dynamic graph convolution and spatial attention. BACKGROUND

[0002] Semantic segmentation is a basic task in computer vision and has attracted much attention of researchers, and the task is also a key step in three-dimensional scenes. When studying objects in three-dimensional space, in addition to shape and position information, we also pay attention to the appearance features of the objects. In a popular way, we only need to pay attention to point cloud information. Point cloud is one of the representatives of three-dimensional data, and higher-level semantic understanding of the spatial information describing point cloud data is attracting more and more attention of researchers.

[0003] Point-wise MLP method: The point-wise MLP method uses a shared MLP as the basic unit of the network, and introduces some special networks to obtain more extensive and rich local structures.

[0004] Pointnet++[3] groups points by different layers and constantly learns from larger local areas. In view of the problems of unevenness and density changes of point cloud, the network proposes a multi-scale and multi-resolution grouping method. Engelmann et al.[5] defined two neighborhoods in the world space using K-means clustering and in the feature space using KNN, and introduced the distance loss and centroid loss between points to realize the normative feature learning. RandLA-Net[6] proposes a new feature aggregation module for capturing and preserving local geometric features by using random points for sampling, and the network achieves very high efficiency in memory and calculation. PointWeb[7] uses a web to find the relationship between each pair of points in a local area, and proposes an adaptive feature adjustment (AFA) module for information exchange and feature refinement. There is also a part of the network (Edgeconv[4] and NetVLAD[8]) that uses local-global feature connection to complete the capture of local information and scene-level global features.

[0005] Graph-based methods: Graph-based methods utilize graph networks to capture the shape and geometry of point clouds. Super Point Graph[9] groups point clouds according to simple shapes and whether super points are associated, and uses Super Point Graph to obtain structure and context information, which can be used in large-scale point clouds, decomposing them into problems of geometric uniform segmentation, context segmentation, and super point embedding. Landrieu and Boussaha

[10] over-segment point clouds into pure super points through a new supervised framework, and use contrastive loss to identify the boundaries between each object. PyramNet

[11] is called a pyramid network, which uses GEM modules to represent point clouds as directed acyclic graphs, uses covariance matrices to construct adjacent similarity matrices, and uses PAN modules to extract different semantic features using four different size convolution kernels. GAC

[12] can segment distinguishable features while also assigning attention weight through the spatial position and feature difference of different neighboring points and feature channels, learning relevant features in the local neighborhood set. PointGCR

[13] represents point clouds as undirected graphs and obtains global context information through channel dimensions.

[0006] In addition to the above methods, there are many other methods. Point convolution methods propose effective convolution operators for point clouds. KPConv

[14] determines convolution weights through the Euclidean distance to the kernel point, and the position of the kernel point is expressed as an optimization problem of the best coverage range in the spherical space. In each layer, grid subsampling is used to achieve high robustness under different density point clouds. 3DCNN-DQN-RNN

[15] proposes a dilated point convolution (DPC) operation to complete the aggregation of dilated neighboring features, and does not use K nearest neighbors for aggregation, and many experiments are used to prove that the receptive field has a great influence on the performance of the aggregation method. Using recurrent neural networks (RNNs) to capture inherent context features is also a common method in this field. The GB-RCU network

[16] first obtains input-level context by converting point blocks into multi-scale blocks and grid blocks, and uses the features extracted by PointNet[2] to input into the comprehensive unit (CU) or recurrent comprehensive unit (RCU) in turn to obtain output-level context. The network shows that spatial context is very important for improving segmentation. 3P-RNN

[17] uses a point-by-point pyramid pool (3P) module to obtain local structures from coarse to fine, and long-distance spatial dependencies are obtained using bidirectional hierarchical RNNs, and then RNNs are applied for end-to-end learning.

[0007] The existing technology mainly has the following problems:

[0008] PointNet proposed the pioneering work that learns per-point features using shared MLPs and learns global features using a symmetric pooling function. However, PointNet does not consider local features to be processed. PointNet++ was born, which divides points into layers and groups, and gradually learns from larger local areas. PointNet++ ignores the relationship between points, and Edgeconv is repeatedly stacked to capture local information and scene-level global features.

[0009] Edgeconv ignores the vector direction between adjacent points, so it still loses part of the local geometric information. It also has a disadvantage that deep features and their fields may be too similar to provide useful edge vectors. These methods mainly focus on the extraction of global or local area features of point cloud data, and do not fully mine the semantic feature information of the point cloud; and the correlation between points cannot be captured, and these correlations are also very important for large scene semantic segmentation.

[0010] Therefore, the present application is provided to solve the defects of the prior art, and provides a point cloud semantic segmentation network based on dynamic graph convolution and spatial attention. SUMMARY

[0011] The purpose of the present application is to provide a point cloud semantic segmentation network based on dynamic graph convolution and spatial attention. To solve the technical problems proposed in the background art.

[0012] To solve the above technical problems, the present application is realized by the following technical scheme:

[0013] The present application is a point cloud semantic segmentation network based on dynamic graph convolution and spatial attention, comprising the following steps:

[0014] Input an F-dimensional point cloud X of N points;

[0015] Construct a local dynamic graph G =(V, ), wherein G represents the center point Connect the neighborhood points A directed graph representing the local structure of the point cloud is constructed, V represents the vertex, represents the edge;

[0016] Define each point and its K nearest neighbors to construct a local directed graph structure, each node of the directed graph points to itself, so the edge feature is defined as:

[0017]

[0018] Use the operation of combining maximum pooling and average pooling to obtain K feature information integrated as a feature, as a new feature of the point ; ;

[0019] In the construction of local graphs, the entire point cloud becomes an independent group, but we think that such operations lose part of the global information, and also ignore the correlation between point clouds;

[0020] To make up for the lack of global feature information extraction, we use a multi-layer perception to directly process the original input point cloud, and use this operation to extract high-dimensional global features ;

[0021] Therefore, the output of the GEConv module is ;

[0022] In order to optimize the network structure, this paper uses the idea of feature splicing, and the input of each layer of feature extraction module is the splicing result of all previous output features ;

[0023] In order to capture more point-to-point correlations, we try to introduce a spatial attention module based on feature extraction; The spatial attention module will input the features (F) along the channel direction respectively Maximum pooling and average pooling operation, splice the two outputs in the channel dimension, and then perform convolution operation, and finally get the spatial attention map (A) with the same size as the input through the sigmoid function ;

[0024] Multiply all the features in the channel dimension with the input features to selectively aggregate context information and get new features .

[0025] Further, the F-dimensional point cloud X is represented as:

[0026]

[0027] Where, is point cloud 1, is point cloud n, is the point cloud itself and its three-dimensional coordinates.

[0028] Further, the construction of the local graph G =(V, ) is based on the K points around each point.

[0029] Further, the K nearest neighbors are represented as:

[0030]

[0031] wherein denotes the Euclidean norm.

[0032] Further, the K sets of node features are denoted as:

[0033] ;

[0034] The point new features are denoted as:

[0035]

[0036] wherein, denotes a fusion Relu operation, denotes a max pooling operation, denotes an average pooling operation, denotes the extracted features.

[0037] Further, the high-dimensional global feature is denoted as:

[0038]

[0039] wherein, is an output feature, is an input point cloud, is a multi-layer perception operation.

[0040] Further, the is denoted as:

[0041]

[0042] wherein, is a detail feature obtained by a local graph, is global information learned by a multi-layer perception, is a concatenation operation.

[0043] Further, the concatenation result is:

[0044]

[0045] wherein, denotes all output features before the l+1 layer network.

[0046] Further, the (concatenation result) is:

[0047]

[0048] wherein, is the spatial attention weight of the feature layer F; and is the result of the average pooling and the maximum pooling operation; is the concatenation operation; is the convolution operation; is the sigmoid function.

[0049] Further, the new feature is:

[0050]

[0051] wherein, is the optimized feature map, the dimensions of the output are the same as the input feature F; after the adjustment of the spatial attention module, more important feature information and position information are enhanced.

[0052] The present application has the following beneficial effects:

[0053] The present application reduces the parameter quantity of the network by removing the spatial conversion network while proposing a new edge feature; the designed GEConv module can enrich the diversity of the feature; the introduced spatial attention module can enable the network to learn to focus on more important information. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0055] Figure 1 is a partial schematic view of the present application;

[0056] Figure 2 is a specific structure schematic view of the GEConv of the present application;

[0057] Figure 3 is a spatial attention module schematic view of the present application;

[0058] Figure 4 is an effect schematic view of the present application;

[0059] Figure 5 is a flowchart of the present application. DETAILED DESCRIPTION

[0060] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0061] Please refer to Figures 1-5 The application is a point cloud semantic segmentation network based on dynamic graph convolution and spatial attention, comprising the following steps:

[0062] inputing an F-dimensional point cloud X of N points;

[0063] constructing a local dynamic graph G =(V, ), wherein G represents a center point connecting the neighborhood points to construct a directed graph representing the local structure of the point cloud, V represents the vertex, and E represents the edge;

[0064] defining each point and its K nearest neighbors to construct a local directed graph structure, each node of the directed graph points to itself, and therefore the edge feature is defined as:

[0065]

[0066] wherein, is a nonlinear function representing a set of learnable parameters.

[0067] using the operation of combining maximum pooling and average pooling to integrate K feature information into one feature as a new feature of the point ;

[0068] When constructing a local graph, the entire point cloud becomes an independent group, but we believe that such an operation loses part of the global information and ignores the correlation between point clouds;

[0069] In order to make up for the defects of insufficient extraction of global feature information, a multilayer perceptron is used to directly process the original input point cloud, and a high-dimensional global feature is extracted by using the operation;

[0070] Therefore, the output of the GEConv module is ;

[0071] ​To optimize the network structure, the idea of feature concatenation is used in this paper, and the input of each layer of feature extraction module is the concatenation of all previous output features ;

[0072] To capture more point-to-point correlations, we try to introduce a spatial attention module based on the extracted features. The spatial attention module will perform max-pooling and average-pooling operations on the input features (F) along the channel direction, concatenate the two outputs in the channel dimension, and then perform convolution operations. Finally, the spatial attention map with the same size as the input is obtained through the sigmoid function ;

[0073] Element-wise multiplication of all position features in the channel dimension with the input features selectively aggregates contextual information to obtain new features .

[0074] The F-dimensional point cloud X is represented as:

[0075]

[0076] Where X1, Xn are the point clouds, and X is the point cloud itself and its three-dimensional coordinates, which can also include additional coordinates such as color and surface normal.

[0077] The construction of the local graph G = (V, ) is based on the K nearest neighbors around each point.

[0078] The K nearest neighbors are represented as:

[0079]

[0080] Where ||·|| represents the Euclidean norm.

[0081] The K node feature set is represented as:

[0082] ;

[0083] The new feature of point is represented as:

[0084]

[0085] Where, represents the fusion Relu operation, represents the max-pooling operation, represents the average-pooling operation, and represents the extracted features.​

[0086] high-dimensional global feature is expressed as:

[0087]

[0088] wherein, is the output feature, is the input point cloud, is a multi-layer perception operation.

[0089] is expressed as:

[0090]

[0091] wherein, is a detailed feature obtained through a local graph, is global information learned through a multi-layer perception, is a concatenation operation.

[0092] concatenation result is:

[0093]

[0094] wherein, represents all output features before the l+1 layer network.

[0095] ( ) is:

[0096]

[0097] wherein, is a spatial attention weight of the feature layer F; and is the result of the average pooling and the maximum pooling operation; is a concatenation operation; is a convolution operation; is a sigmoid function.

[0098] new feature is:

[0099]

[0100] wherein, is an optimized feature map, the output dimensions of which are the same as the input feature F; after adjustment by the spatial attention module, more important feature information and position information are enhanced.

[0101] The implementation process of the above technical solution is:

[0102] 1. Change of edge convolution

[0103] We use to represent an F-dimensional point cloud of N points, where F can represent not only 3D coordinates , but also color, classification value, etc. We use KNN algorithm to connect neighborhood points to construct a directed graph representing the local structure of the point cloud , where the vertices , edges .

[0104] Each node of the directed graph points to itself, so the edge feature is defined as:

[0105]

[0106] where is a nonlinear function composed of learnable parameters .

[0107] (1) In PointNet , this makes it only focus on global shape information, ignoring local feature information.

[0108] (2) In PCNN , this only focuses on the features around the point.

[0109] (3) In PointNet++ , the difference between the point and the surrounding points is taken as input, only focusing on local features.

[0110] (4) In DGCNN

[18] , use to capture the center information, capture local information, so as to focus on both global and local information of the point cloud.

[0111] But we think the above methods still have their shortcomings, we propose a new edge feature, namely the average symmetric edge feature .

[0112]

[0113] where: The parameters are learnable; ​This is implemented using a multilayer perceptron. We map the captured center and local information to a high-dimensional feature space, and learn the geometric correlations between points from this space to obtain the edge features of the point cloud. These edge features not only contain information about the center point and its local neighborhood, but also encode the center point's location. After obtaining the local neighborhood information, we use max pooling as the aggregation function. This selects the most important features while removing redundant information, thus reducing the number of parameters in the network.

[0114] 2. GEConv module

[0115] First, we will consider each point and its K nearest neighbors To construct a local directed graph structure;

[0116] in This represents the Euclidean norm.

[0117] When extracting local information, we first use k-nearest neighbors to determine new local regions, obtain the features of K edges, and then obtain a set of K node features. :

[0118]

[0119] Based on the features obtained above, we will use a shared network. Update to obtain K feature information The above K features are then combined into one feature using max pooling, which is used as a point. New features :

[0120]

[0121] When constructing a local graph, the entire point cloud becomes a series of independent groups. However, we believe that this operation loses some global information and also ignores the correlation between point clouds.

[0122] To compensate for the deficiency in extracting global feature information, we use a multilayer perceptron to directly process the original input point cloud, and use this operation to extract high-dimensional global features. :

[0123]

[0124] in, For output features, To input point clouds, This is for multilayer perceptron operation.

[0125] Thus, the output of the GEConv module is:

[0126]

[0127] where, is the detailed feature obtained by the local map, is the global information learned by the multi-layer perception, is the concatenation operation.

[0128] To optimize the network structure, the application uses the idea of feature concatenation, and the input of each layer of feature extraction module is the concatenation of all previous output features :

[0129]

[0130] 3. Spatial attention

[0131] Attention mechanism can help the model assign different weights to each input part and extract more key information without significantly increasing the computational overload of the model. We try to introduce a spatial attention module on the basis of feature extraction to capture more point-to-point correlations.

[0132] The spatial attention module performs maximum and average pooling operations on the input features (F) along the channel direction, concatenates the two outputs in the channel dimension, then performs convolution operation, and finally obtains the spatial attention map (A) with the same size as the input through the sigmoid function. ).

[0133]

[0134] where: is the spatial attention weight of the feature layer F; and are the results of the average pooling and maximum pooling operations; is the concatenation operation; is the convolution operation; is the sigmoid function.

[0135] Element-wise multiplication of all position features in the channel dimension with the input features selectively aggregates contextual information to obtain new features .

[0136]

[0137] is the optimized feature map, and the dimensions of the output are the same as the input feature F. After the adjustment of the spatial attention module, more important feature information and position information are enhanced.

[0138] In summary:

[0139] The most important thing of the application is to design a new graph convolution network architecture GLA-Net, by establishing a dynamic graph structure of the query point, designing an edge convolution module with residual learning idea, the convolution block fuses the graph-based method and the point-based method, respectively extracts the local features and global features on the point cloud, and adopts the residual learning method to fuse the two kinds of features, which has great advantages in feature representation, and effectively alleviates the network degradation problem caused by the deepening of network layers. The application splices multiple levels of features by using the DenseNet idea to improve the fitting ability of the network. In addition, we use the newly designed edge feature function to help the network learn better domain features. We use the combination of maximum pooling and average pooling to make up for the loss of global information caused by using maximum pooling alone. We introduce a spatial attention module to extract the semantic information between points and fully exploit the correlation between points.

[0140] Edge convolution change part: we map the captured center information and local information to a high-dimensional feature space, learn the geometric correlation between points from the high-dimensional feature space, and obtain the edge features of the point cloud. The edge features obtained not only have the center point and local neighborhood information, but also encode the position information of the center point. After obtaining the local neighborhood information between points, we use the combination of maximum pooling and average pooling as the aggregation function to select the most important features while removing some redundant information interference, which has a certain reduction effect on the parameter amount in the network.

[0141] GEConv part: first, we construct a local directed graph structure for each point and its K nearest neighbors to extract local information, first use K nearest neighbors to determine the new local area, obtain K edge features, and obtain K node feature sets For the features obtained above, we will update them using a shared network to obtain K feature information and use the combination of maximum pooling and average pooling to integrate the above K features into one feature as the new feature of point . When constructing the local graph, the whole point cloud will become independent groups, but we think that such an operation loses part of the global information, and also ignores the correlation between point clouds. In order to make up for the defects of insufficient extraction of global feature information, we use the multi-layer perception to directly process the original input point cloud, and use the operation to extract high-dimensional global features . Thus the output of GEConv can be obtained. In order to optimize the network structure, the application uses the idea of feature splicing, and the input of each layer feature extraction module is the splicing of all previous output features.

[0142] Spatial attention part: the spatial attention module respectively performs maximum pooling and average pooling operations on the input features (F) along the channel direction, splices the two outputs in the channel dimension, then performs convolution operation, and finally obtains the spatial attention map (A) with the same size as the input through the sigmoid function . The features of all positions are multiplied with the input features in the channel dimension, so as to selectively aggregate the context information and obtain new features . The output dimension of the optimized feature map is the same as that of the input feature F. After the adjustment of the spatial attention module, the more important feature information and position information are enhanced.

[0143] In the description of the specification, the description of the terms "one embodiment", "example", "specific example" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are contained in at least one embodiment or example of the application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0144] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details, nor limit the application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the specification. The specification selects and describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and utilize the application. The application is limited only by the claims and their full scope and equivalents.

Claims

1. A point cloud semantic segmentation network based on dynamic graph convolution and spatial attention, characterized in that, comprising the steps of: inputting an F-dimensional point cloud X of N points; Construct a local dynamic graph G = (V, ), wherein G represents a center point Connect neighborhood points A representative point cloud local structure is constructed A directed graph V represents a vertex, represent an edge; Define each point and its K nearest neighbors to construct a local directed graph structure, where each node points to itself, thus defining the edge feature as: ; The operation of combining max-pooling and average-pooling will obtain K characteristic information Integrated as a feature, as a point New feature ; In constructing the local graph, the entire point cloud becomes a number of independent groups, and a multi-layer perception machine is used to directly process the original input point cloud to extract high-dimensional global features ; Thus, the output of the GEConv module is ; and the input of each layer of feature extraction module is the splicing result of all previous output features ; On the basis of extracting features, a spatial attention module is introduced; the spatial attention module respectively performs maximum pooling and average pooling operations on the input features (F) along the channel direction, splices the two outputs in the channel dimension, then performs convolution operation, and finally obtains a spatial attention map (S) with the same size as the input through a sigmoid function ). Element-wise multiplication of features at all positions with input features in channel dimension to selectively aggregate context information to get new features .

2. The point cloud semantic segmentation network based on dynamic graph convolution and spatial attention according to claim 1, characterized in that, the F-dimensional point cloud X is represented as: ; wherein, is a point cloud 1, is a point cloud n, is the point cloud itself and its three-dimensional coordinates.

3. The point cloud semantic segmentation network based on dynamic graph convolution and spatial attention according to claim 1, characterized in that, The local graph G = (V, ) is constructed based on K neighboring points around each point.

4. The point cloud semantic segmentation network based on dynamic graph convolution and spatial attention according to claim 1, characterized in that, The K nearest neighbors is represented as: ; wherein denotes the Euclidean norm.

5. The point cloud semantic segmentation network based on dynamic graph convolution and spatial attention according to claim 4, characterized in that, K sets of node features is represented as: ； The point New features Is expressed as: ; wherein, represents a fusion Relu operation, represents a max pooling operation, represents an average pooling operation, represents the extracted features.

6. The point cloud semantic segmentation network based on dynamic graph convolution and spatial attention according to claim 1, characterized in that, The high-dimensional global feature is represented as: ; wherein, is an output feature, is an input point cloud, is a multi-layer perceptron operation.

7. The point cloud semantic segmentation network based on dynamic graph convolution and spatial attention according to claim 1, characterized in that, The is represented as: ; wherein, is a detail feature obtained by a local map, is global information learned by a multi-layer perception, is a concatenation operation.

8. The point cloud semantic segmentation network based on dynamic graph convolution and spatial attention according to claim 1, characterized in that, The splicing result Is: ; where, represents all output features before the l+1 layer network.

9. The point cloud semantic segmentation network based on dynamic graph convolution and spatial attention according to claim 1, characterized in that, The (a) is: ) is: ; wherein, is a spatial attention weight for feature layer F; and is a result of average pooling and max pooling operations; is a concatenation operation; is a convolution operation; is a sigmoid function.

10. The point cloud semantic segmentation network based on dynamic graph convolution and spatial attention according to claim 1, characterized in that, The new feature Is: ; wherein, is the optimized feature map, whose output dimensions are the same as the input feature F.

Images