A dynamic graph semantic feature extraction method for point cloud classification and segmentation

Abstract

The present invention discloses a method for extracting semantic features of dynamic graphs for point cloud classification and segmentation, and relates to the technical field of point cloud feature extraction. The present invention comprises the following steps: inputting C-dimensional data P with n points; constructing a local dynamic graph, defining x i k neighbor points #imgabs0#edge feature e ij =h θ (x i ,x j ), through the edge function h θ Aggregation point x i Neighbor features. The present invention obtains the most representative semantic information on all local graphs through maximum pooling, and obtains the position with the most obvious semantic features; divides all feature graphs into groups according to the channel dimension, and obtains the strongest features of semantic features in each local graph, so that the similarity between each group of global statistical features and local position features is used as an attention guide to generate a mask for semantic feature extraction, and uses graph attention to assign different weights to the features on the local graph, and adopts summation to aggregate the information between points in the local graph; finally, the features are adaptively fused.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud feature extraction, and in particular relates to a dynamic graph semantic feature extraction method for point cloud classification and segmentation. Background Art

[0002] Computer vision has always been a very active research topic, especially three-dimensional vision; and three-dimensional data are usually presented in the form of projected images, grids, voxels, and point clouds.

[0003] Point clouds retain the original geometric information of objects in three-dimensional space. Their expression is simple and does not require any discretization. With the rapid development of 3D acquisition technology, point clouds have become easy to obtain, including various 3D scanners and LiDARs. Three-dimensional point cloud data is widely used in various fields of modern engineering, including robotics and remote sensing, due to its wide range of acquisition methods and high precision. In recent years, deep learning has become a hot topic in the study of computer vision and natural language processing. Due to the irregularity and discretization of point clouds, the three-dimensional point cloud feature extraction method based on deep learning is challenging. Point cloud feature extraction is very important for the subsequent classification and segmentation tasks of point clouds. This invention mainly studies the classification and partial segmentation of point clouds;

[0004] Graph-based methods: Point clouds are discretely distributed in space, and there is no significant connection between points. However, graph structures can well construct the relationship between points. Simonovsky et al., pioneers of graph theory in point clouds, regard each point as a vertex of the graph and connect each vertex to all its neighboring points through directed edges. Then, edge conditional convolution (ECC) was proposed using filters, multi-layer perceptrons (MLP), etc. PointNet is the first network to directly process point clouds. However, it cannot handle the neighborhood information around the point well, and there is a lack of good information interaction between points. PointNet++ solves the above problems by building a graph. It uses FPS sampling to obtain a point set, and then uses ball search to find the neighborhood of each point and fuse the neighborhood information. However, the point feature learning in PointNet++ is learned in an isolated way, so LocalSpecGCN was proposed. It converts the constructed local graph into a spectral domain to learn features, and then uses recursive clustering and pooling to solve the problem of isolated learning point features. Similarly, in order to solve the problem of information interaction between PointNet points, DGCNN uses the nearest neighbor algorithm KNN to construct the features of local graph fusion points, and then dynamically updates the graph in the feature space. DeepGCN adopts the concepts that have been successful in training deep convolutional neural networks, mainly residual connections, dense connections, and dilated convolutions. It proposes to allow very deep graph convolutional neural networks GCNs to be trained. So-Net uses point-to-node knn instead of node-to-point knn to accurately control the receptive field to extract point cloud features. GACNet uses graph attention to aggregate the features of neighboring points, specifically assigning appropriate attention weights to different neighboring points, and learning features based on the dynamics of neighboring points.

[0005] Attention-based methods: The attention mechanism was first proposed in the field of visual images and later widely used in the field of NLP, such as text summarization and reading comprehension. Since then, the attention mechanism has been widely used in computer vision. SENet brings an effective and lightweight gating mechanism that can recalibrate the feature map by the importance of the channel. It considers that the importance of pixels in different channels may be different. CBAM considers that the importance of pixels at different positions in the same channel may also be different, that is, at the spatial level. So spatial attention is added on the basis of SENet. Influenced by the Transformer proposed in the field of NLP, PAT uses self-attention mechanism and group shuffle attention to obtain the important features of point clouds. L2G proposes a local-to-global autoencoder to learn the local and global structures of point clouds simultaneously through local-to-global reconstruction. Point Transformer extracts point cloud features based on the vector self-attention mechanism, and the input of attention is the position and features of the downsampled point cloud. PCT avoids defining the order of point cloud data and learning features through offset attention mechanism by using the order invariance inherent in transformer.

[0006] Based on other methods: MVCNN uses a multi-view approach to 3D shape recognition. It uses images of the same 3D shape from different perspectives and combines them to extract 3D shape description operators. PVNet is the first method that combines multi-view and point cloud for 3D shape recognition. It uses high-level features in multi-view data to model the intrinsic correlation and recognizability of different structural features in point cloud data. VoteNet uses volume representation to represent point clouds. It selects a set of seed points through the backbone network, learns their features, and then performs Hough voting on these seed points. Kd-net constructs a kD-tree for the input point cloud, and then performs hierarchical feature extraction from leaf to root. CP-Net considers the importance of each point and extracts point cloud features through adaptive key point sampling for classification tasks.

[0007] The existing technology mainly has the following problems:

[0008] Point cloud is a kind of non-Euclidean data that is unstructured, irregular, and discretely distributed in space. Graph convolution networks (GCNs) have been widely used because of their advantage of being able to represent non-Euclidean data with graphs. Building a graph in a point cloud can well establish the connection between points, and the points on the graph can be regarded as a regular and ordered point set to be input into the neural network. There are two ways to build a graph: in the spectral domain and in the spatial domain. When building a graph in the spectral domain, feature decomposition wastes computing resources and memory, so the graph is generally built in the spatial domain. Pointnet++ extends PointNet, which not only considers global information but also local details. The point set is obtained by sampling the farthest point, and then the local graph is constructed by ball search, and then MLP and maximum pooling are used to aggregate the local features of the point cloud. DGCNN constructs a local graph by using the k-nearest neighbor algorithm, aggregates the local and global features of the point cloud using an asymmetric function, and then uses MLP and maximum pooling to aggregate the information of the points. PointWeb obtains a point set by sampling the farthest point, and then constructs a local graph by knn. An AFA module is proposed that focuses on studying the relationship between points in a local point cloud image, using MLP and maximum pooling for information aggregation. When most graph-based methods process local features of point clouds, they usually use 1x1 convolution to increase the number of channels after composition, and then use maximum pooling to aggregate features. However, there is no direct semantic information interaction between channels after aggregating features. Moreover, only maximum pooling is used, which loses the features of each node and the relationship information between nodes.

[0009] To this end, in order to solve the defects of the prior art, the present invention provides a dynamic graph semantic feature extraction method for point cloud classification and segmentation. Summary of the invention

[0010] The purpose of the present invention is to provide a dynamic graph semantic feature extraction method for point cloud classification and segmentation, so as to solve the prior art problems in the above-mentioned background technology.

[0011] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0012] The present invention is a method for extracting semantic features of dynamic graphs for point cloud classification and segmentation, comprising the following steps:

[0013] Input C-dimensional data P with n points;

[0014] Construct a local dynamic graph G = (V, E) and define x i The k neighbor points of Edge Features ij =h θ (x i ,x j ), then the output x of a layer of graph isi ′ is expressed as:

[0015] x i ′=h θ (x i ,x j )

[0016] Through the edge function h θ Aggregation point x i Neighborhood characteristics;

[0017] The most representative semantic information X′ on the local graph is obtained through maximum pooling, and the position with the most obvious semantic features is obtained;

[0018] The single-channel semantic features are adjusted by using the feature space distribution of the group as a whole; X′={x 1 ,...,x C′}, the feature map is divided into G groups according to the channel dimension, that is, each group has m (m = C' / G) dimensional features; first, the mean of each channel in the group is obtained to represent the single channel feature c i , and then update the features within the group, and add the different features of the points within the group to obtain the semantic features p of the entire group g ;

[0019] Then the semantic features p of the whole group g Perform normalization processing;

[0020] Finally, the semantic features of the whole group p g The weight W of the intra-group feature is obtained by processing with the sigmoid function σ(), and then multiplied by the original group data to obtain the interactive feature.

[0021] Sum to aggregate the features of the local graph and get a single graph attention

[0022] The graph attention and semantic features are aggregated through learnable scalars α and β to obtain the aggregated feature f.

[0023] Further, the C-dimensional data P is expressed as:

[0024] P = {p i ∈R C ,i=0,...,N-1}.

[0025] Furthermore, the local graph G=(V, E) is constructed based on k neighboring points around each point.

[0026] Furthermore, the edge function h θ It is expressed as:

[0027] h θ (xi ,x j )=h θ (x i ,x j -x i )

[0028] Among them, x i is the global feature, and xj-xi is the local feature;

[0029] And through the multi-layer perceptron MLP processing, the semantic features X of all local graphs are obtained, which can be expressed as:

[0030] X=MLP(h θ (x i ,x j )).

[0031] Furthermore, the most representative semantic information X′ is expressed as:

[0032] X′=max(X).

[0033] Furthermore, the single channel feature c i It is expressed as:

[0034]

[0035] Among them, i represents the feature on the channel, N represents N points, and the features within the group are updated to obtain:

[0036]

[0037] Among them, j represents a point, and its value range is 1 to N;

[0038] Add the different features of the points in the group, and the semantic feature p of the whole group g , expressed as:

[0039]

[0040] Among them, p g Represents the semantic features of a group, i represents the single channel of the g-th group, and g represents the G-th group feature.

[0041] Furthermore, the semantic feature p of the whole group g After normalization, it is expressed as:

[0042]

[0043] The learning scale parameter γ and offset β are introduced to adjust the group normalization data to obtain the semantic feature p of the whole group. g The normalized adjustment is expressed as:

[0044] ag =γ·p g +β.

[0045] Furthermore, the semantic feature p of the whole group g and the original group feature x i～ m={x i ,...,x m} multiplied to get the interactive features It is expressed as:

[0046]

[0047] Then concatenate each set of data to obtain the feature

[0048] Furthermore, we get the weight

[0049] W = t(X);

[0050] W is the weight, t() is the multi-layer perceptron MLP after softmax processing;

[0051] Single local graph feature Weight The representation of the single graph attention is:

[0052]

[0053] The feature composition of all local graphs Then it is processed by MLP.

[0054] Furthermore, the identifier of the aggregate feature f is:

[0055] f=α·f g +β·f p .

[0056] The present invention has the following beneficial effects:

[0057] The present invention obtains the strongest feature of semantic features in the local graph, and then uses the similarity between each group of global statistical features and local position features as an attention guide to generate a mask for semantic feature extraction, and adopts an asymmetric function to obtain the local and global features of the point cloud, and then uses maximum pooling to aggregate the global and local features of the point and its neighboring points. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0059] Figure 1 It is a local dynamic schematic diagram of the present invention;

[0060] Figure 2 It is a schematic diagram of the effect of the present invention;

[0061] Figure 3 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] See also Figure 1-3 As shown, the present invention is a dynamic graph semantic feature extraction method for point cloud classification and segmentation.

[0064] 1. Dynamic graph construction

[0065] Traditional GCN will have the problem of over-smoothing as the network deepens, and the receptive field size remains unchanged. To solve this problem, we construct a local dynamic graph and dynamically update the graph in the feature space.

[0066] Input C-dimensional data P with n points = {p i ∈R C ,i=0,...,N-1}, the original data C=3, which only represents the X, Y, and Z coordinates;

[0067] Use the KNN algorithm to find the k points around each point and construct a local graph G = (V, E), such as Figure 1 As shown. Define x i The k neighbor points of Edge Features ij =h θ (x i ,x j ), then the output of a layer of graph is expressed as:

[0068] x i ′=h θ (x i ,x j )

[0069] To aggregate point x i The neighbor features are selected as follows: θ . This not only has the global feature x iThere are also local features xj-xi.

[0070] h θ (x i ,x j )=h θ (x i ,x j -x i )

[0071] Then the multi-layer perceptron MLP is used to output the semantic features X of all local graphs.

[0072] X=MLP(h θ (x i ,x j ))

[0073] 1.1 Semantic Feature Interaction

[0074] One channel corresponds to one feature map, which is a detection of a certain feature. The strength of a certain value in the channel is a reflection of the strength of the current feature, indicating that the feature distribution in the same channel is different.

[0075] Different channels may focus on different features, that is, the feature distribution of different channels is also different. When the network is at a shallow level, shallow features are noticed, and when the network layer is deeper, some high-level semantic features are paid attention to.

[0076] In order to extract semantic interaction features and better perform classification and segmentation tasks;

[0077] In order to obtain the most obvious semantic features, we first use the maximum pooling to obtain the most representative semantic information X′ on the local graph:

[0078] X′=max(X)

[0079] The spatial distribution of the group’s features is used to adjust the single-channel semantic features. 1 ,...,x C′}, the feature map is divided into G groups according to the channel dimension, that is, each group has m (m = C' / G) dimensional features. First, the mean of each channel in the group is obtained to represent the single channel feature c i :

[0080]

[0081] Where i represents the feature on the channel, and N represents N points. Update the features within the group:

[0082]

[0083] Among them, j represents a point, and its value range is 1 to N.

[0084] Add the different features of the points in the group to get the semantic feature p of the whole group g , g indicates the Gth group of features.

[0085]

[0086] Then the semantic features p of the whole group are g Perform normalization.

[0087]

[0088] The ε is to avoid the denominator being 0, μ g is the semantic feature p of the whole group g The mean value, σ g Yes g The standard deviation of .

[0089] As with the commonly used normalization method, we introduce a learnable scale parameter γ and offset β to adjust the group normalization data and obtain the semantic feature p of the whole group g The normalized adjustment is expressed as:

[0090] a g =γ·p g +β

[0091] Finally, the semantic features of the whole group p g The weight of the features within the group is obtained by processing with the sigmoid function σ() and then compared with the original group data x i～m ={x i ,...,x m} multiplied to get the interactive features

[0092]

[0093] Finally, each set of data is spliced ​​to obtain the feature

[0094] 1.2 Graph Attention

[0095] Only using the maximum pooling to aggregate local features loses the features of each node and the relationship information between nodes. However, the relationship between nodes has different importance. We let the local point cloud features guide themselves to design different weights W; the method t() is a multi-layer perceptron MLP after softmax processing:

[0096] W=t(X)

[0097] Single local graph feature Weight In order not to lose the features of each node and the relationship information between nodes, we use summation to aggregate the features of the local graph, and the output of a single graph attention is:

[0098]

[0099] The feature composition of all local graphs After that, it is processed by a multi-layer perceptron (MLP).

[0100] 2. Feature Aggregation

[0101] Finally, for the aggregation of the two branch features, we introduce two learnable scalars to aggregate features, allowing them to adaptively aggregate semantic features and fine-grained features between points, thereby enhancing the contextual information of the local graph.

[0102] f=α·f g +β·f p .

[0103] 3. Summary

[0104] The main feature of the present invention is the interaction of semantic features and graph attention as well as the fusion of the features of these two branches.

[0105] Semantic feature interaction part: One channel corresponds to one feature map, which is the detection of a certain feature. A group of channels corresponds to a group of feature maps. The focus of a single feature map (i.e., semantics) is different from that of a group of feature maps. By obtaining the strongest feature of the semantic feature in the local map, the similarity between each group of global statistical features and local position features is used as an attention guide to generate a mask for semantic feature extraction. After data normalization, the attention score is multiplied by the original feature.

[0106] Graph attention part: Use the k-nearest neighbor algorithm to find the nearest k neighbor points of each point, use an asymmetric function to obtain the local and global features of the point cloud, then use graph attention to assign different weights to different features on the graph, and then use summation to aggregate the features on the graph.

[0107] Feature fusion part: Use two learnable scalars to adaptively fuse the semantic feature interaction part features and the graph attention part features.

[0108] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0109] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the present invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A dynamic graph semantic feature extraction method for point cloud classification and segmentation, characterized by: The following steps are involved: Input C-dimensional data P with n points; Construct a local dynamic graph G = (V, E) and define x i The k neighbor points of Edge Features ij =h θ (x i ,x j ), then the output x of a layer of graph is i ′ is expressed as: x i ′=h θ (x i ,x j ) Through the edge function h θ Aggregation point x i Neighborhood characteristics; The most representative semantic information X′ on the local graph is obtained through maximum pooling, and the position with the most obvious semantic features is obtained; The single-channel semantic features are adjusted by using the feature space distribution of the group as a whole; X′={x 1 ,...,x C′ }, the feature map is divided into G groups according to the channel dimension, that is, each group has m (m = C' / G) dimensional features; first, the mean of each channel in the group is obtained to represent the single channel feature c i , and then update the features within the group, and add the different features of the points within the group to obtain the semantic features p of the entire group g ; Then the semantic features p of the whole group g Perform normalization processing; Finally, the semantic features of the whole group p g The weight W of the intra-group feature is obtained by processing with the sigmoid function σ(), and then multiplied by the original group data to obtain the interactive feature. Sum to aggregate the features of the local graph and get a single graph attention Aggregate the graph attention and semantic features through learnable scalars α and β to obtain the aggregated feature f; The semantic feature p of the whole group g and the original group feature x i～m ={x i ,...,x m } multiplied to get the interactive features It is expressed as: Then concatenate each set of data to obtain the feature Get weight W = t(X); W is the weight, t() is the multi-layer perceptron MLP after softmax processing; Single local graph feature Weight The representation of the single graph attention is: The feature composition of all local graphs After that, it is processed by MLP; The identifier of the aggregate feature f is: f=α·f g +β·f p 。 2. A method for extracting dynamic graph semantic features for point cloud classification and segmentation according to claim 1, characterized in that: The C-dimensional data P is expressed as: P={p i ∈R C ,i=0,...,N-1}。 3. The method for extracting semantic features of dynamic graphs for point cloud classification and segmentation according to claim 1, characterized in that: The local graph G=(V, E) is constructed based on k neighboring points around each point.

4. The method for extracting semantic features of dynamic graphs for point cloud classification and segmentation according to claim 1, characterized in that: The edge function h θ It is expressed as: h θ (x i ,x j )=h θ (x i ,x j -x i ) Among them, x i is the global feature, x j -x i It is a local feature; And through the multi-layer perceptron MLP processing, the semantic features X of all local graphs are obtained, which can be expressed as: X=MLP(h θ (x i ,x j ))。 5. The method for extracting dynamic graph semantic features for point cloud classification and segmentation according to claim 1, characterized in that: The most representative semantic information X′ is expressed as: X′=max(X).

6. The method for extracting dynamic graph semantic features for point cloud classification and segmentation according to claim 1, characterized in that: The single channel feature c i It is expressed as: Among them, i represents the feature on the channel, N represents N points, and the features within the group are updated to obtain: Among them, j represents a point, and its value range is 1 to N; Add the different features of the points in the group, and the semantic feature p of the whole group g , expressed as: Among them, p g Represents the semantic features of a group, i represents the single channel of the g-th group, and g represents the G-th group feature.

7. The method for extracting semantic features of dynamic graphs for point cloud classification and segmentation according to claim 1, characterized in that: The semantic feature p of the whole group g After normalization, it is expressed as: The learning scale parameter γ and offset β are introduced to adjust the group normalization data to obtain the semantic feature p of the whole group. g The normalized adjustment is expressed as: a g =γ·p g +b.

Images