Three-dimensional point cloud target detection method based on graph neural network

Abstract

The invention provides a three-dimensional point cloud target detection method based on a graph neural network. The three-dimensional point cloud target detection method comprises a graph structure construction step of obtaining a first layer feature and a second layer feature according to a state vector of a neighborhood point and relative positions of a central point and the neighborhood point; a graph attention aggregation and point cloud alignment step: adopting a graph attention mechanism to realize iterative update of graph node features until the Tth layer of features are updated; and a target detection step: inputting the Tth layer of features of each graph node to a classification multi-layer perceptron and a positioning multi-layer perceptron to complete target classification and target positioning tasks. According to the method, adaptive weighted aggregation of neighbor features is realized by constructing a multi-layer voxel graph structure and introducing a graph attention mechanism, meanwhile, the alignment module is added to improve the robustness under target position change, efficient feature aggregation with semantic selectivity is realized, and meanwhile, the alignment module and a bounding box fusion strategy are integrated, so that the robustness is improved. And the accuracy and the model generalization ability in the three-dimensional detection task are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a three-dimensional point cloud target detection technology, in particular to a three-dimensional point cloud target detection technology based on a graph neural network. BACKGROUND

[0002] With the rapid development of automatic driving, intelligent robots and other technologies, environmental perception has become one of the core links. The point cloud data collected by the laser radar (LiDAR) plays a key role in the three-dimensional target detection task due to its high precision and excellent spatial expression capability. Traditional point cloud processing methods mainly adopt three structure conversion methods: one is to voxelize the point cloud to adapt to the convolutional neural network; the second is to project the point cloud into a bird's eye view (BEV) or other two-dimensional image view to simplify the model modeling process; the third is to directly extract features from the original point set, such as the PointNet++ series method. However, these methods generally have certain limitations: the voxelization and projection operations will cause the loss of spatial information, affecting the detection accuracy; the point set processing method, although retains the original structure, has large computational overhead and unstable neighborhood construction; although the graph neural network provides a new modeling method, the existing methods often fail to accurately express the semantic correlation of adjacent points, and the feature aggregation strategy used is relatively rough, which is difficult to meet the demand for fine expression of the detection task. In addition, most graph neural network models focus on semantic segmentation or classification tasks, and have not been specially optimized in structure design and aggregation mechanism for the target detection task.

[0003] Several typical problems currently exist in three-dimensional point cloud target detection, including: information loss caused in the voxelization or projection process; the feature aggregation method in traditional graph neural networks is simple and difficult to highlight key neighbor features, leading to inaccurate detection; the model is unstable when the target position shifts slightly; and multiple points predict repeated bounding boxes without effective merging mechanism. That is, existing three-dimensional target detection methods generally have problems such as large information loss, rough feature aggregation, and high computational complexity when processing point cloud data, making it difficult to balance detection accuracy and efficiency. In particular, in the graph neural network method, the traditional aggregation method is difficult to effectively model the semantic relationship between points, resulting in blurred or misdetected target boundaries. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a three-dimensional point cloud target detection method that improves the accuracy and stability of the detection result by constructing a multi-layer voxel graph structure, introducing a graph attention mechanism to realize adaptive weighted aggregation of neighbor features, and adding an alignment module to improve the robustness under target position changes.

[0005] The technical solution adopted by the present application to solve the above technical problems is a three-dimensional point cloud target detection method based on a graph neural network, comprising the steps of:

[0006] Graph structure construction step: divide the point cloud into voxels with a set edge length, and randomly select a point p in each voxel j As a center point, determine the points p in the neighborhood i As a neighborhood point, establish a graph edge connection relationship set E; splice the state vector of the neighborhood point p i with the relative position of the center point and the neighborhood point, and input the first multi-layer perception machine to obtain the first layer feature feature 1 ; and then input the first layer feature feature j related to the center point p 1 through maximum pooling aggregation to the second multi-layer perception machine to obtain the node feature of the center node, that is, the second layer feature feature 2 ;

[0007] Graph attention aggregation and point cloud alignment step: the graph attention mechanism is used to realize the iterative update of the graph node feature, from initialization t = 2, until the update to the Tth layer feature, and the (t+1) th layer feature is iteratively updated as follows:

[0008]

[0009] Wherein, x represents the three-dimensional coordinates of the graph node, i is the neighborhood point serial number, j is the center point serial number, p is the graph attention mechanism responsible for aggregating neighbor edge features, f t is the edge feature extraction processing of the tth iteration, g t is the processing for updating the node's own feature of the tth iteration; h t is a feature-driven alignment mechanism introduced in the tth iteration to compensate for the position error in the point cloud due to target motion or sensor noise, that is, the offset term of the adjacent point is added to each point position: h t , f t and g t are realized by a multi-layer perception machine based on graph attention GAT;

[0010] Target detection step: input the Tth layer feature feature T of each graph node to the classification multi-layer perception machine and the multi-layer perception machine to complete the classification of the target and the positioning task of the target.

[0011] Preferably, the total loss function in the training process is composed of three parts: classification loss, positioning loss and regularization loss; the classification loss is used to represent the prediction error of the category to which each graph node belongs; the positioning loss is used to predict the geometric difference between the boundary box and the real boundary box; the regularization loss prevents the model from overfitting, and the L1 regularization term of each multi-layer perception machine.

[0012] Further, the positioning task of the target outputs a prediction bounding box of the detected target, and in the case that multiple overlapping bounding boxes are predicted for the same target by multiple points, the candidate prediction bounding boxes are merged, and a unique confidence score is output for each target, and the specific process is as follows:

[0013] Cluster box generation: the highest score is selected from the candidate prediction bounding boxes to form a current main box, and overlapping boxes with an intersection over union (IoU) exceeding a threshold value are collected to form a cluster;

[0014] Box merging: the center position and size of all bounding boxes in the cluster are taken as the median to obtain a merged box;

[0015] Occlusion factor calculation: the occlusion factor is calculated according to the spatial proportion of the points inside the main box in the merged box;

[0016] Confidence calculation: the final confidence is obtained by fusing the classification score, IoU and occlusion factor.

[0017] The present application effectively improves the flexibility and discriminability of feature aggregation by introducing a graph attention mechanism, so that the model can pay more attention to the information of key neighbor points; the added alignment module enhances the adaptability of the model to target position changes and improves the stability of the detection; further, the improved bounding box merging strategy combines the median box, IoU and occlusion factor to make the final output bounding box more accurate and less repetitive; at the same time, through the reasonable design of the loss function, the model is more stable in the training process, and the classification and positioning performance are improved. Overall, the present application achieves good three-dimensional target detection effect in the point cloud scene.

[0018] The present application has the beneficial effect that the present application improves the expression ability of the model to the spatial structure of the point cloud and the stability of the detection result by constructing a voxel graph structure, introducing a graph attention mechanism and an alignment module, and designing a bounding box fusion strategy. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The target detection process is shown in Figure 1.

[0020] Figure 2 The detection results of the embodiment model are shown in Figure 2. DETAILED DESCRIPTION

[0021] As shown in Figure 3, the three-dimensional target detection includes the following steps: Figure 1

[0022] S1: graph structure construction step:

[0023] The original point cloud is used to construct a multi-layer graph structure. First, the point cloud is divided into voxels with an edge length of L voxel_size , and a point p j is randomly selected as the center point p vertex ​. Search for points p in the space with a fixed radius r within the neighborhood of the center point p i As a neighborhood point p key , establish a graph edge connection relationship set E:

[0024] E={(p j ,p i )|||x j -x i ||2<r}

[0025] Where ||·||2 is the L2 norm, x j represents the three-dimensional coordinates of p j , i is the neighborhood point sequence number, and j is the center point sequence number.

[0026] For the first layer feature feature j formed by the edge of the center point p i and its neighborhood point p 1 The state vector s i of p i and the relative position x i -x j are concatenated and input into the first multi-layer perception MLP to obtain:

[0027] feature 1 =MLP(concat(s i ,x i -x j ))

[0028] The first layer feature feature j corresponding to all edges of the center point p 1 is aggregated by maximum pooling Maxpool and input into the second multi-layer perception MLP to obtain the node feature of the center node, that is, the second layer feature feature 2 :

[0029] feature 2 =MLP(Maxpool(feature 1 ))

[0030] This module supports efficient construction of graph structure and extraction of local geometric features, providing a basis for subsequent graph attention aggregation.

[0031] S2: Graph attention aggregation and point cloud alignment step

[0032] The graph attention mechanism is used to realize the iterative update of the graph node feature, and the alignment module is introduced to improve the robustness of target detection.

[0033] Initialize t=2, in the t+1 iteration, the center point feature pj Edge feature of adjacent edge Performing aggregated update:

[0034]

[0035] where v t is the node feature of the t-th iteration, e t is the edge feature of the t-th iteration, f t is the edge feature extraction process of the t-th iteration, p is the graph attention mechanism responsible for aggregating neighbor edge features, g t is the node feature update process of the t-th iteration.

[0036] To compensate for the position error in the point cloud caused by target motion or sensor noise, a feature-driven alignment mechanism h t is introduced, that is, an offset term of the adjacent point is added to each point position:

[0037]

[0038] is the t-th layer feature of the adjacent point p i , which is calculated in the same way as p j .

[0039] The t-th layer feature of the adjusted center node is:

[0040]

[0041] where functions h t , f t and g t are implemented by a multilayer perceptron (MLP) based on graph attention (GAT), and residual connections are used to improve training stability. The normalized weights of the graph attention mechanism are calculated by a softmax function.

[0042] The embodiment sets the number of iteration rounds to T = 3, and finally outputs the node feature feature T to a classification multilayer perceptron (MLP) cls and a localization multilayer perceptron (MLP) loc for subsequent classification and localization tasks. The MLP cls outputs a classification probability, and the MLP loc outputs a bounding box parameter.

[0043] S3: loss function calculation step

[0044] The total loss function designed by the embodiment is composed of three parts: classification loss, localization loss and regularization loss, and the overall loss function expression is as follows:

[0045] l = a l cls + b l loc + g l reg

[0046] where a, b, g are the weights for adjusting the three loss terms.

[0047] 1. Classification loss:

[0048] is used to measure the prediction error of the class that each point belongs to in a system containing N points. Let the predicted class distribution of each point be The total number of classes is M, and the cross-entropy loss is defined as follows:

[0049]

[0050] where is the one-hot code class label of the i'th vertex, is the predicted probability that the i'th vertex belongs to the j'th class.

[0051] 2. Positioning loss

[0052] is used to fit the geometric difference between the predicted bounding box and the true bounding box. The predicted bounding box is represented by 7 degrees of freedom parameters:

[0053] b = (x', y', z', l, h, w, q)

[0054] x', y', z' are the three-dimensional base points of the bounding box, l, h, w are the three-dimensional dimensions of the bounding box, and q is the orientation of the bounding box.

[0055] The 7 degrees of freedom parameters are normalized and encoded as follows: x′ y′ z′ l h w θ

[0056]

[0057] where the subscript v represents the true value, l m , h m , and w m are the reference dimensions of the length, height, and width of the bounding box, respectively, and q m is the angle range of the bounding box.

[0058] The Huber loss L Huber is used to measure the deviation between the prediction and the true value, and the overall positioning loss is defined as:

[0059] ​​​​​​​

[0060] wherein, represents whether the vertex belongs to the target category that needs to be positioned, 1 if yes, otherwise 0; δ represents the normalized encoding of the predicted bounding box, δ v represents the normalized encoding of the real bounding box, and {δ} is the set of predicted bounding boxes.

[0061] 3. Regularization loss

[0062] To prevent overfitting of the model, the embodiment performs L1 regularization on each MLP layer, l reg L1 regularization term containing all MLP layers.

[0063] The loss function of the whole application comprehensively considers the classification accuracy, positioning accuracy and model complexity, which can effectively guide the model to learn more robust spatial and semantic features.

[0064] S4: bounding box merging and confidence score module

[0065] In the three-dimensional target detection task, the same target may be predicted by multiple points to form multiple overlapping bounding boxes, and these candidate boxes need to be merged, and a unique confidence score is output for each target. Therefore, the application introduces an improved non-maximum suppression strategy, referring to the method of Point-GNN, introducing the median box mechanism, the occlusion factor and the IoU weight on the basis of the traditional NMS, to realize more accurate and stable bounding box fusion and scoring.

[0066] The specific process includes the following four steps:

[0067] 1. Cluster generation: select the highest scoring candidate from the remaining candidate boxes to form the current main box, and collect overlapping boxes with an intersection over union (IoU) exceeding a threshold to form a cluster;

[0068] 2. Box merging: take the median of the center positions and sizes of all bounding boxes in the cluster to obtain a merged box;

[0069] 3. Occlusion factor calculation: calculate the occlusion factor according to the spatial proportion of the points inside the main box in the merged box;

[0070] 4. Confidence calculation: fuse the classification score, IoU and occlusion factor to obtain the final confidence.

[0071] Embodiment

[0072] In the Ubuntu 20.04 system environment, based on the TensorFlow framework and the RTX4090 GPU platform, the KITTI 3D detection task is used as the evaluation data set, and only the laser radar point cloud data (without image information) is used as the input. The model parameter configuration is as follows: in the vehicle detection task, the classification category is set to 4 categories, the graph construction radius is 4 meters, and the voxel size is 0.2 meters; in the pedestrian and cyclist detection task, the classification category is 6 categories, the graph construction radius is set to 1.6 meters, and the voxel is also 0.2 meters. The iteration number of the graph attention network (GAT) module is 3 times, and the feature extraction in each layer of GAT is realized by MLP. The initial feature encoding adopts a four-layer MLP structure, and the channel number is (32, 64, 128, 300). The optimizer uses SGD, and different learning rate strategies are set for different tasks. The non-maximum suppression (NMS) threshold set in the bounding box fusion part is: Car class 0.01, Pedestrian / Cyclist class 0.2. As shown in Figure 2 , the application achieves good three-dimensional target detection effect in the point cloud scene.

Claims

1. A method for detecting 3D point cloud targets based on graph neural networks, characterized in that, Including the following steps: Graph structure construction steps: Divide the point cloud into voxels with a set side length , and randomly select a point p within each voxel. j As the center point, determine the points p in the neighborhood. i As a neighboring point, establish a graph edge connection set E; and consider the neighboring point p... i The state vector, concatenated with the relative positions of the center point and neighboring points, is input into the first multilayer perceptron to obtain the first layer feature. 1 Then, with respect to the center point p... j The first layer of features 1 The node features of the central node are obtained by max pooling aggregation and then input into a second-layer perceptron, i.e., the second-layer features. 2 ; Graph attention aggregation and point cloud alignment steps: A graph attention mechanism is used to iteratively update the graph node features, starting from initialization t=2 until updating to the T-th layer features. The iterative update of the (t+1)-th layer features is as follows: Where x represents the 3D coordinates of a graph node, i is the neighbor node index, j is the center node index, ρ is the graph attention mechanism responsible for aggregating neighbor edge features, and f t For the edge feature extraction processing in the t-th iteration, g t This is the processing used to update the node's own features in the t-th iteration; To compensate for positional errors in the point cloud caused by target motion or sensor noise in the t-th iteration, a feature-driven alignment mechanism h is introduced. t That is, add the offset term of the adjacent point to the position of each point: h t f t and g t All are implemented using a multilayer perceptron based on graph attention GAT; Object detection steps: Collect the T-th layer features of each graph node. T The input is fed into the classification multilayer perceptron and the localization multilayer perceptron to complete the target classification and target localization tasks.

2. The method as described in claim 1, characterized in that, The method for determining the neighborhood point is as follows: with a fixed radius r, search in space for points p within the neighborhood that are less than r away from the center point. i As neighboring points, establish a set E of graph edge connectivity relationships: E={(p j ,p i )|||x j -x i ||2<r} Where ||·||2 is the L2 norm, x j p j The three-dimensional coordinates are given, where i is the neighboring point number and j is the center point number.

3. The method as described in claim 1, characterized in that, Let the neighboring point p i state vector s i Relative position x of the center point and neighboring points i -x j After concatenation and input into the first multilayer perceptron (MLP), the first layer features are obtained. 1 : feature 1 =MLP(concat(s i ,x i -x j ))。 4. The method as described in claim 1, characterized in that, Will be at the center point p j The first layer of features 1 The node features of the center node are obtained by max-pooling and then input into a second-layer perceptron (MLP), i.e., the second-layer features. 2 : feature 2 =MLP(Maxpool(feature 1 ))。 5. The method as described in claim 1, characterized in that, The total loss function during training consists of three parts: classification loss, localization loss, and regularization loss; The classification loss is used to represent the prediction error of the category to which each graph node belongs; The localization loss is used to measure the geometric difference between the predicted bounding box and the true bounding box. Regularization loss prevents the model from overfitting and is the L1 regularization term for each multilayer perceptron.

6. The method as described in claim 1, characterized in that, The target localization task outputs predicted bounding boxes for the detected targets. For cases where multiple overlapping bounding boxes are predicted from multiple points for the same target, these candidate predicted bounding boxes are merged, and a unique confidence score is output for each target. The specific process is as follows: Box cluster generation: Select the highest-scoring candidate predicted bounding boxes to form the current main box, and collect overlapping boxes with which the Intersection over Union (IoU) exceeds a threshold to form a cluster; Boundary merging: The merged bounding box is obtained by taking the median of the center position and size of all bounding boxes in the cluster; Occlusion factor calculation: The occlusion factor is calculated based on the spatial proportion of points inside the main frame within the merged frame; Confidence calculation: The final confidence score is obtained by fusing the classification score, IoU and occlusion factor.

Images