A point cloud classification and segmentation method based on long-short-range dependency graph learning

Abstract

The application belongs to the technical field of point cloud analysis, and particularly relates to a point cloud classification and segmentation method based on long-range-short-range dependency graph learning. The application constructs a long-range dependency graph and a short-range dependency graph on a point cloud by calculating the point pair connection weights of the global range and the local range of the point cloud. Then, prior knowledge is introduced to design a graph structure loss for the long-range dependency graph and the short-range dependency graph respectively, so as to guide the learning of the graph structure. Then, the convolution operation on the graph is designed according to the threshold and the topK strategy to select the neighbor nodes, and the node features on the graph are fused. Finally, the size of the point cloud is gradually reduced through the graph pooling technology, and the multiple graph structure learning and graph convolution operations are stacked, and a corresponding deep neural network is constructed for different tasks. The application can end-to-end construct the point cloud into a graph to better serve the subsequent feature extraction and point cloud analysis task, and has been successfully applied to point cloud classification, part segmentation and semantic segmentation tasks.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of point cloud analysis, and particularly relates to a point cloud classification and segmentation method based on long-range-short-range dependency graph learning. BACKGROUND

[0002] Recently, point cloud analysis has become a hot topic in the field of 3D visual understanding and has attracted extensive attention from the academic and industrial circles. In particular, in practical applications such as autonomous driving and robots, point cloud data is widely concerned due to its convenience of acquisition and its characteristics of not being easily affected by environmental factors such as weather. On the other hand, deep neural networks have been successfully applied to many computer vision and pattern recognition tasks. Therefore, many methods attempt to promote the research of deep learning on geometric data in order to effectively process point cloud data. So far, thanks to the application of deep neural networks, great progress has been made in many point cloud analysis tasks, including 3D shape classification, partial segmentation, semantic segmentation, etc. Among them, some successful models can be roughly divided into point-wise multi-layer perceptron-based methods (MLP-based), convolution-based methods (Convolution-based), graph-based methods (Graph-based) and Transformer-based methods. In addition, since the rise of graph neural networks, graph convolution methods have become the first method for processing geometric data, such as social networks, traffic networks, etc. Therefore, using graph neural networks to process irregular point clouds has been widely used in point cloud analysis. Recent efforts explore local geometric features and contextual information by using local and global graphs, showing promising performance on many tasks. For example, Wang Yue et al. proposed a dynamic graph convolutional neural network (DGCNN) to construct a graph from a point cloud using a K-nearest neighbor algorithm in the feature field, so as to extract local features of the point cloud in the form of graph convolution. Lei Huan et al. mapped the point cloud to a unit sphere, and then constructed a graph structure on the sphere to ensure the translational invariance and anisotropy of the convolution kernel. Some other scholars introduced attention or self-attention mechanisms to define convolution and pooling operations on the graph.

[0003] Although the above methods achieve satisfactory performance in point cloud classification and segmentation tasks, they mainly rely on complex local feature extraction, and their main focus is still concentrated on the construction of convolutional operators. Most methods have little awareness that the quality of the graph structure has a great influence on future feature extraction and analysis tasks. Obviously, a good graph structure can make the extracted features more accurate. However, previous methods usually use fixed K-Nearest Neighbor or ball query algorithm to construct the graph structure. Although these techniques are simple, they cannot guarantee that the constructed graph structure is optimal because the KNN and ball query algorithm are independent of the network learning process. In view of this, how to construct a suitable graph structure on the point cloud to optimally serve the subsequent tasks has become a technical problem to be solved at present. SUMMARY

[0004] To solve the problems of the prior art, one purpose of the present application is to design a general graph structure learning framework to construct long-range dependency graph and short-range dependency graph on point cloud, and to mine local features and global context information of point cloud. At the same time, make full use of known prior knowledge in the graph structure learning stage, and design dependent graph structure loss to guide the learning of graph structure.

[0005] Another purpose of the present application is to integrate the graph structure learning framework and the graph convolution operation in a unified learning framework, make full use of the given label and the prior information related to the graph structure, and learn an optimal graph structure to better serve different downstream tasks.

[0006] In order to achieve the above purpose, the present application provides a point cloud classification and segmentation method based on long-range-short-range dependency graph learning, and the specific technical solutions are as follows:

[0007] Step S1: constructing the given point cloud P = {p1, p2,..., pn} into long-range dependency graph G1 and short-range dependency graph G2 through the graph structure learning architecture. N} respectively. And

[0008] Step S2: introducing some prior knowledge such as sparsity, distance correlation, etc. in the process of learning graph structure, and further designing corresponding graph structure loss for long-range dependency graph and short-range dependency graph respectively. And

[0009] Step S3: considering the "dimension disaster" phenomenon in the process of calculating the graph structure loss, reducing the dimension of the feature through a simple linear transformation, and learning the graph structure in a low-dimensional space.

[0010] Step S4: designing node selection and feature aggregation strategy through setting threshold and topK strategy, and designing convolution operation on the graph on this basis.

[0011] Step S5: Adaptively fuse the remote dependency graph using the learnable parameter γ. and short-range dependency graph Features above;

[0012] Step S6: Reduce the resolution of the point cloud using graph pooling and stack multiple graph structure learning and graph convolutional layers to achieve feature extraction at different scales.

[0013] Step S7: For point cloud classification and segmentation tasks, design corresponding deep neural networks to extract the overall description and point-by-point features of the point cloud, and then use a fully connected network to achieve shape classification of the point cloud or segmentation of different target regions.

[0014] Specifically, step S1 includes: given a point cloud P = {p1, p2, ..., p...} with N points. N The corresponding feature set is X = {x1, x2, ..., x}. N For any pair of points p in the point cloud i and p j Through a simple nonlinear function S ij =g(p i x i p j x j Calculate the weights of the edges ε(i,j) between these pairs. The nonlinear function is designed as follows:

[0015] S ij =g(p i x i p j x j ) = sigmoid((|p i -p j ||||x i -x j |)W),

[0016] in The parameters of a simple neural network are represented by sigmoid(·), which maps the function's output to the range (0, 1), and its value represents the strength of the connection. Then, let S = (S... ij ) N×N For remote dependency graph On the one hand, it uses the adjacency matrix; on the other hand, it combines the K-nearest neighbor graph with the long-range dependency graph to restrict the receptive field of view to the local neighborhood, thereby constructing a short-range dependency graph. Specifically as follows:

[0017]

[0018] in

[0019]

[0020] A=(A ij ) N×N is a local geometry matrix, is a short-range dependency graph adjacency matrix.

[0021] The step S2 includes definition of the long-range dependency graph structure loss and the short-range dependency graph structure loss:

[0022]

[0023]

[0024] where ||·|| denotes L2,1 norm. 2，1

[0025] The step S3 includes mapping the input features to a low-dimensional space by a linear transformation , and then performing graph structure learning and loss function calculation in the low-dimensional space, so as to avoid the "curse of dimensionality" phenomenon.

[0026] The step S4 includes setting a threshold θ to remove part of nodes with weak connection strength, selecting K nodes with the strongest connection according to the topK principle (all nodes are selected if the number of nodes is less than K), and finally aggregating node features by average aggregation to realize convolution operation.

[0027] The step S5 includes setting a learnable parameter γ to adaptively fuse features on the long-range dependency graph and the short-range dependency graph :

[0028]

[0029] where and are node features extracted on the two graphs respectively, is the fused feature.

[0030] The step S6 includes designing a graph pooling operation by using point cloud downsampling technology, so as to reduce the point cloud resolution, and stacking multiple layers of graph structure learning and graph convolution to realize multi-scale feature extraction of the point cloud.

[0031] ​For the point cloud shape classification task, the step S7 comprises: aggregating the output features of the last layer of graph convolution into a high-dimensional vector through a global maximum pooling operation and taking the high-dimensional vector as the overall shape descriptor of the point cloud, and then inputting the high-dimensional vector into a fully connected neural network to predict the category of the target; for the point cloud segmentation task, the point cloud is gradually restored to the original size through an up-sampling operation corresponding to the graph pooling operation, which firstly interpolates the nodes on the coarse graph through nearest neighbor interpolation and inputs the nodes into a multi-layer perceptron to refine the features, and then gradually decodes and restores the original size of the point cloud through 2 times of up-sampling operation of the high-dimensional features output by the graph convolution layer to obtain the feature representation of each point on the point cloud, and then after the last up-sampling, the representation of each node on the point cloud is refined by using a graph structure learning and a graph convolution operation, and finally each point is classified through a fully connected neural network to realize the component segmentation of the target point cloud.

[0032] The advantage of the present application is that a general graph structure learning framework is designed to construct long-range dependency graphs and short-range dependency graphs on point clouds to mine local features and global context information of the point clouds. At the same time, the known prior knowledge is fully utilized at the graph structure learning stage, and a dependent graph structure loss is designed to guide the learning of the graph structure. In addition, another advantage of the present application is that the graph structure learning framework and the graph convolution operation are integrated in a unified learning framework, the given label and the prior information related to the graph structure are fully utilized, and thus an optimal graph structure is learned to better serve different downstream tasks. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The graph structure learning architecture diagram.

[0034] Figure 2 The node selection and graph convolution diagram.

[0035] Figure 3 The graph structure learning and graph convolution fusion diagram.

[0036] Figure 4 The residual multi-layer perceptron structure diagram.

[0037] Figure 5 The point cloud classification network structure diagram.

[0038] Figure 6 The point cloud segmentation network structure diagram. DETAILED DESCRIPTION

[0039] The following describes a point cloud classification and segmentation method based on graph structure learning according to an embodiment of the present invention, with reference to the accompanying drawings. First, the graph structure learning architecture, a schematic diagram of graph convolution operation, a unified process of graph structure learning and graph convolution, and a residual multilayer perceptron are described with reference to the accompanying drawings. Finally, these components are used as basic units, and two networks are designed under a unified framework to achieve point cloud shape classification and component segmentation tasks.

[0040] like Figures 1-6 As shown, the method includes the following steps:

[0041] S1 Graph Structure Learning Architecture: Input point cloud P = {p1, p2, ..., p...} N The corresponding feature set is X = {x1, x2, ..., x}. N}, arbitrarily select a pair of points p i and p j Form a pair of points and denote ε(i, j) as the edge between those points. For example... Figure 1 As shown, there are a total of N×N point pairs on the point cloud, which are then processed by a simple nonlinear function S. ij =g(p i x i p j x j The weights of edge ε(i,j) are calculated using a simple feedforward neural network, which fits the nonlinear function in this architecture.

[0042] S ij =g(p i x i p j x j ) = sigmoid((|p i -p j ||||x i -x j |)W),

[0043] The "||" symbol represents a concatenation operation. For the parameters of the neural network. The sigmoid(·) function maps the output of the function to the range (0, 1), and its value represents the strength of the connection. Let S = (S ij ) N×N For remote dependency graph The adjacency matrix is ​​used. Furthermore, since the local structure of the point cloud is crucial for feature extraction, another branch of the graph structure learning architecture restricts edge connections on the point cloud to local neighborhoods using a K-nearest neighbor graph, thereby constructing a short-range dependency graph. Specifically as follows:

[0044]

[0045] where,

[0046]

[0047] A = (A ij ) N×N is the adjacency matrix of K-NN graph, and let is the adjacency matrix of short-range dependency graph.

[0048] S2-S3 graph structure loss: In the process of graph structure learning, some prior knowledge is introduced to design graph structure loss on long-range dependency graph and short-range dependency graph respectively, so as to guide the learning of graph structure in the network training process. The specific graph structure loss is as follows:

[0049]

[0050]

[0051] where, is the graph structure loss of long-range dependency graph, and obviously when minimizing the loss function in the training stage, the difference between the feature x i and the feature x j is greater, that is, the Euclidean distance is greater, then the connection weight S ij of the assigned edge ε(i, j) tends to be smaller. This conforms to the assumption that the greater the feature difference, the smaller the dependency. is the graph structure loss of short-range dependency graph, and when minimizing the loss function in the training stage, the distance between p i and p j is farther, then the weight assigned is smaller. ||·|| 2，1 represents L2,1 norm

[0052]

[0053] is used to control the sparsity of the adjacency matrix S and .||·|| F is the Frobenius norm of the matrix, and α, β are hyperparameters for controlling the sparsity regularization.

[0054] Further considering the deficiency of Euclidean distance in high-dimensional space, the application adopts a simple linear mapping to map high-dimensional features to a low-dimensional space first, and learns the graph structure and calculates the graph structure loss in the low-dimensional space:

[0055]

[0056]

[0057]

[0058]

[0059]

[0060] wherein, is a linear mapping.

[0061] S4-S5 node selection and graph convolution operation: as shown in Figure 2 , for a vertex x i on the long-range dependency graph or the short-range dependency graph, the weights of the edges with the vertex as the end point are arranged in descending order and denoted as [S i1 , S i2 ,..., S iN ], then a threshold θ is set to screen the weights, and the weights not exceeding the threshold are reset to 0, which is described as follows:

[0062]

[0063] Then, according to the topK strategy, the K nodes with the largest weights (or all nodes if there are less than K nodes) are selected, and the node features are aggregated in an average aggregation manner:

[0064]

[0065] Here, Φ(·) is a feature mapping function, x i * is the output of the graph convolution. The graph structure learning and the graph convolution operation are integrated together, that is, the graph convolution operation is performed on the constructed long-range dependency graph and short-range dependency graph respectively, and finally the node features on the two graphs are adaptively fused through a learnable parameter γ, and the structure is as shown in Figure 3 .

[0066]

[0067] and are the node features on the long-range dependency graph and the short-range dependency graph respectively. In the present technology, the feature mapping function Φ(·) is constructed as a multi-layer perceptron with a residual connection, and the structure is as shown in Figure 4 .

[0068] S6 graph pooling operation: the vertices of the coarse graph are determined by the farthest point sampling algorithm, and the neighborhood nodes are aggregated onto the vertices of the coarse graph by using a pooling operator, so as to realize the graph pooling operation on the point cloud:

[0069]

[0070] Wherein, the pooling function is executed as a maximum pooling function. The size of the point cloud is reduced by the graph pooling operation, and then the graph structure learning and the graph convolution operation are performed on different size point clouds, so as to construct a deep neural network to extract features of the point cloud at different scales in a hierarchical manner.

[0071] S7 Point cloud classification and segmentation network: taking the graph structure learning architecture, the graph convolution operation and the graph pooling operation proposed in steps S1-S6 as basic units, a corresponding deep neural network is designed for the point cloud classification and segmentation task respectively. In the classification task, the point cloud features are extracted by stacking the graph structure learning and the graph convolution operation for 3 times, and the point cloud is represented as a high-dimensional vector by the global maximum pooling operation, and the classification of the point cloud is realized by the fully connected network, and the overall network structure is as shown in Figure 5 The feature extraction stage of the point cloud segmentation network adopts the same architecture as the classification network, except that the segmentation task needs to classify each point on the point cloud, so in order to obtain the feature of each point, the present application adopts the operation on the point cloud to gradually restore the point cloud to the original size and introduce a layer of graph structure learning and graph convolution operation after the last upsampling to refine the point cloud features, and finally the features of each point on the point cloud are input into the fully connected network to realize the segmentation of the point cloud, as shown in Figure 6 .

[0072] In the training process of the network, the task target is taken as the main loss and the designed graph structure loss is taken as the auxiliary, so that the network builds a more suitable graph structure to best serve different tasks, and the specific loss function is as follows:

[0073]

[0074] Wherein, the cross loss of the point cloud classification or segmentation task:

[0075]

[0076] L is the number of layers of the graph structure learning, λ is a hyperparameter for controlling the balance between the task loss and the graph structure loss, y ij is the true label, when the i-th sample belongs to the j-th class, y ij =1, otherwise y ij =0, p ij is the probability of predicting that the i-th sample belongs to the j-th class.

Claims

1. A point cloud classification and segmentation method based on long-short range dependency graph learning, characterized in that, The method is used for 3D vision processing, comprising the following steps: Step S1: constructing a given point cloud by a graph structure learning architecture respectively constructed as long-range dependency graphs and short-range dependency graphs ; Step S2: Introduce some prior knowledge such as sparsity, distance correlation, etc. in the process of learning graph structure, and further design corresponding graph structure loss for long-range dependency graph and short-range dependency graph respectively and ; Step S3: In the process of calculating the graph structure loss, the dimensionality of the features is reduced by a simple linear transformation to learn the graph structure in a low-dimensional space, considering the "dimensionality disaster" phenomenon; Step S4: The node selection and feature aggregation strategy is designed by setting a threshold and topK strategy, and the convolution operation on the graph is designed on this basis; Step S5: Fusing features on the remote dependency graph and the short-range dependency graph adaptively through learnable parameters ​​​ Step S6: The resolution of the point cloud is reduced by using the graph pooling operation, and multiple layers of graph structure learning and graph convolution layers are stacked to realize feature extraction at different scales; Step S7: For point cloud classification and segmentation tasks, a corresponding deep neural network is designed to extract the overall description and point-by-point features of the point cloud, and then a fully connected network is used to realize shape classification or segmentation of different target regions of the point cloud; Step S1 includes: giving a Point cloud of individual points Its corresponding feature set is For any pair of points in the point cloud and Through a simple nonlinear function Calculate the edge between this pair The weights and nonlinear function are designed as follows: ， wherein denote parameters of a simple neural network, maps the output of the function to , whose magnitude represents the strength of the connection, and then letting be the adjacency matrix of the long-range dependency graph , on the other hand, combining the long-range dependency graph with a K- nearest neighbor graph, the receptive field is limited to a local neighborhood to construct a short-range dependency graph , which is as follows: ， Wherein ， is a local geometry matrix, is a short-range dependency graph is an adjacency matrix; The step S4 comprises setting a threshold value The nodes with weak connection strength are pruned, the K nodes with the strongest connection are selected according to the topK principle, and finally the convolution operation is realized by aggregating the node features in an average aggregation manner.

2. The method of claim 1, wherein, The step S2 comprises the definition of long-range and short-range dependent graph structure loss: ， wherein denotes norm.

3. The method of claim 1, wherein, The graph structure learning architecture considers the "dimensionality disaster" phenomenon, and maps the node features to a low-dimensional space by linear transformation, so as to learn the graph structure in a low-dimensional space and define the graph structure loss: ， wherein is a linear mapping.

4. The method of claim 1, wherein, According to the step S5 comprises: setting learnable parameters Adaptively fusing remote dependency graphs With short-range dependency graphs On features: ， wherein with are the node features extracted from the two graphs respectively, is the fused feature.

5. The method of claim 1, wherein, According to the step S6, the point cloud downsampling technology is used to design the graph pooling operation, so as to reduce the resolution of the point cloud, and multiple layers of graph structure learning and graph convolution are stacked to realize multi-scale feature extraction of the point cloud.

6. The method of claim 1, wherein, For point cloud classification tasks, the features extracted are fused by a max pooling operator to obtain the overall shape descriptor of the point cloud, and then the shape descriptor is input into a fully connected network to predict the category of the target.

7. The method of claim 1, wherein, For point cloud segmentation tasks, the low-resolution point cloud is gradually restored to the original size by an upsampling operator, and finally the graph structure learning and graph convolution layer are used to refine the features, and the features of each point are input into a fully connected network to realize the segmentation of the target point cloud.

8. The method of claim 1, wherein, In the training process of the network, the task target is taken as the main loss and the designed graph structure loss is taken as the auxiliary, so that the network constructs a more suitable graph structure to best serve different tasks, and the specific loss function is as follows: ， wherein, a cross loss for a point cloud classification or segmentation task, a number of layers for graph structure learning, a hyperparameter for balancing a control task loss and a graph structure loss.

Images