3D point cloud object detection method based on second-order multi-attention mechanism of point column

Abstract

The present invention provides a 3D point cloud target detection method based on a second-order multi-attention mechanism of point columns, which includes the following steps: proposing a method for respectively realizing target detection by three mechanisms, namely a second-order point attention mechanism, a second-order channel attention mechanism, and a pseudo-image space attention mechanism based on point columns; providing a network, which is mainly composed of a second-order point attention mechanism, a point column feature network, a second-order channel attention mechanism, a backbone network, a pseudo-image space attention mechanism, and an SSD detection head; then voxelizing the point cloud, performing a second-order point attention mechanism operation on the point cloud, converting it into features of a pseudo image, performing a second-order channel attention mechanism operation on the features of the pseudo image, outputting features of the pseudo space, performing a pseudo-image space attention mechanism operation on the features of the pseudo space, and outputting a detection result; through the present invention, a relatively high detection speed is guaranteed and the accuracy of extraction is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the field of 3D pure lidar point cloud targets, and specifically relates to a method for realizing target detection based on three mechanisms: a second-order point attention mechanism based on point columns, a second-order channel attention mechanism, and a pseudo-image space attention mechanism. Background Art

[0002] Currently, 3D point cloud object detection methods are gaining increasing popularity in fields such as computer vision, autonomous driving, robotics, and virtual reality. Compared to object detection based on two-dimensional images, lidar (LiDAR) can provide more reliable depth information, more accurately locate objects, and provide shape information. However, due to the lack of texture, occlusion, truncation, and uneven reflections in 3D point clouds, lidar point clouds are sparse and have large density variations. This often affects the accuracy of traditional 3D object detection methods based on handcrafted features. In recent years, as deep neural networks have demonstrated excellent feature extraction capabilities and can process high-dimensional data, the accuracy of 3D point cloud object detection methods based on deep neural networks has been improved to a certain extent. Nevertheless, due to the high sparsity and inherent irregularity of point clouds, the accuracy of detection results for some categories still has significant room for improvement.

[0003] For example, in 2016, Li et al. proposed VeloFCN, which converts point clouds into a front-view feature map representation and then uses off-the-shelf detectors. (See B.Li, T.Zhang, and T.Xia, “VeloFCN: Vehicle detection from 3D lidar using a fully convolutional network,” in Robotics, 2016.) In 2017, Qi et al. proposed PointNet, the first model to train a deep neural network using raw point cloud data. (See CRQi, H.Su, K.Mo, and LJGuibas, “Pointnet: Deep learning on point sets for 3D classification and segmentation,” in CVPR, 2017.) In 2018, Martin Simon et al. introduced Complex-Yolo, a model that projects point clouds onto a two-dimensional plane and uses image-based methods for object detection, thereby accelerating network inference. However, this projection method is limited by the sparsity of point clouds, making convolution ineffective in extracting features. (See M. Simon, S. Milz, K. Amende, and H.-M. Gross. “Complex-YOLO: Real-time 3D object detection on point clouds,” arXiv:1803.06199, 2018.) To alleviate the occlusion problem caused by overlapping front views, Yang et al. proposed PIXOR, which rasterizes point clouds into a more compact BEV representation. However, a significant drawback is the need for manual feature extraction. However, manual design not only fails to fully utilize the three-dimensional information of objects, but also is not conducive to generalization to other radar applications. (See BY ang, W. Luo, and R. Urtasun, “Pixor: Real-time 3D object detection from point clouds,” in CVPR, 2018.) In 2018, Zhou et al. first proposed an end-to-end trainable network, VoxelNet, a general 3D detection framework. Unlike most previous work, VoxelNet starts to learn information-rich feature representations and can simultaneously learn different feature representations from point clouds. However, the disadvantage of 3D convolution is that it is too time-consuming and faces a large amount of computation, resulting in slow network inference speed.(See Y.Zhou and O.Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” In CVPR, 2018.). Yan et al. then proposed SECOND, which reduces memory consumption and speeds up computation through sparse convolution operations. (See Y.Yan, Y.Mao, and B.Li, “SECOND: Sparsely embedded convolutional detection,” In Sensors, 18(10), 2018.). In order to improve inference speed using the standard 2D convolutional detection pipeline, H.Lang et al. proposed PointPillars in 2019 to encode point clouds into vertical columns, which is essentially a special division of voxels. (See AHLang, SV ora, H.Caesar, L.Zhou, JY ang, and O.Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in CVPR, 2019.).

[0004] In addition, the prior art paper A.H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in CVPR 2019, proposes a method that first partitions the input point cloud into regions, voxelizes the point cloud, and converts it into a sparse pseudo-image. A fixed number of points are randomly retained in each pillar. In this step, the feature dimensions of the points in the pillars are augmented from the original 4D information to 9D, resulting in 9-dimensional features for each LiDAR point. A 2D network is used in the backbone network for feature learning. The backbone network consists of two subnetworks: a top-down network that generates features at increasingly smaller spatial resolutions, and a second network that performs upsampling and concatenation of the top-down subnetworks. The final output features are the concatenation of all features of the same dimension but with different step sizes. In the detection head module, an SSD detection head is used for bounding box regression. A 2D joint cross-section (Iou) is used to match the prior box with the ground truth. The height and elevation of the bounding box are not used for matching; instead, 2D matching is employed here, with the height and elevation serving as additional regression targets. Although the PointPillars network proposes using pillars to voxelize point clouds to improve speed, the downsampling process in the backbone network often loses feature information from the input image. Furthermore, points within voxels are correlated with each other, so processing each point in the point cloud in isolation inevitably loses some useful geometric information, which in turn affects detection accuracy. In the backbone network, processing each channel in isolation ignores the correlation between channels, resulting in a loss of useful information and reduced detection accuracy. After generating the pseudo-image, the features in the pseudo-space are treated identically. Since not all features in the pseudo-space contribute equally to the detection task, regions with greater task relevance are more important. Simply treating them identically will also reduce the final detection accuracy. Therefore, a real-time and accurate 3D point cloud object detection method is urgently needed that achieves a dynamic balance between speed and accuracy. Summary of the Invention

[0005] In view of the above-mentioned defects of the prior art, the purpose of the present invention is to provide a real-time and accurate 3D point cloud target detection method, which can achieve a dynamic balance between speed and accuracy, and solves the problem that the existing methods cannot perform higher-precision target detection in real time through three mechanisms: a second-order point attention mechanism based on point columns, a second-order channel attention mechanism, and a pseudo-image space attention mechanism.

[0006] Technical problems solved by the present invention:

[0007] First, in the feature extraction network step, the backbone network often loses feature information of the input image during downsampling. Furthermore, points within voxels are correlated with each other. Processing points in the point cloud in isolation will inevitably lose some useful geometric information, thus affecting detection accuracy. This paper proposes a point column-based second-order point attention mechanism that retains more useful information by linking points within the same voxel. This allows for the extraction of more refined feature information at a relatively low inference speed.

[0008] Second, in the backbone network, processing each channel in isolation ignores the correlation between channels, which results in the loss of some useful information and reduced detection accuracy. This paper proposes a point-column-based second-order channel attention mechanism that connects channels, retains more useful feature information, and improves overall detection accuracy.

[0009] Third, after generating the pseudo image, the features in the pseudo space are treated identically. Since not all features in the pseudo space contribute equally to the detection task, regions with greater relevance to the task are more important, and simply treating them identically will also affect their detection accuracy. Taking this into account, the present invention proposes a pseudo image spatial attention mechanism based on point columns. This mechanism assigns different weights to each pixel in the pseudo space based on the importance of the region to the task, thereby obtaining more accurate detection results.

[0010] The technical solution adopted by the present invention to solve the technical problem is: a 3D point cloud target detection method based on a second-order multi-attention mechanism of point columns, which includes the following steps:

[0011] S1: Propose three mechanisms based on point column second-order point attention mechanism, second-order channel attention mechanism, and pseudo image space attention mechanism to achieve target detection methods respectively;

[0012] S2: Based on S1, a network is provided. The network mainly consists of a second-order point attention mechanism, a point column feature network, a second-order channel attention mechanism, a backbone network, a pseudo-image space attention mechanism, and an SSD detection head. The network is also divided into a second-order attention module, a second-order point attention module, and a second-order channel attention module.

[0013] S3: voxelize the point cloud, then perform a second-order point attention mechanism on the point cloud to convert it into pseudo-image features;

[0014] S4: Perform a second-order channel attention mechanism on the features of the pseudo image and output the features of the pseudo space;

[0015] S5: Perform pseudo-image space attention mechanism operation on the features of pseudo space and output the detection results;

[0016] Among them, the SSD detection head uses the features of the backbone to predict the three-dimensional bounding box of the object; the second-order attention module includes global maximum pooling, covariance pooling and row convolution; when the point feature is used as the input of the second-order attention module in S3, the second-order point attention mechanism weight will be obtained as the output, and this process is the second-order point attention module; when the channel feature is input into the second-order attention module, the second-order channel attention mechanism weight will be obtained, and this process is the second-order channel attention module.

[0017] In a given K-th voxel, for all points X in the voxel K ∈R N×C , where N represents the maximum number of points, C represents the number of channels, and after global maximum pooling, the vector E consisting of the maximum values ​​on each dimension is obtained K ∈R N ×1 , E K ∈R N×1 Input to a fully connected layer, where N×1 represents a vector of N rows and 1 column, and the vector Q is obtained K ∈R t×1 , where t is the number of points after the reduction by the W1 fully connected layer. The ReLU activation function is used after the W1 fully connected layer to calculate the covariance matrix I between two points in the same voxel K ∈R t×t , where t is the number of points in the second-order point attention mechanism, t is the number of channels in the second-order channel attention mechanism, and t×t is the dimension. The covariance matrix is ​​convolved row by row to obtain the vector J∈R t×1 Then the vector J∈R t×1 Input to the W2 fully connected layer and use the activation function Sigmoid function to obtain the N-dimensional attention vector s∈R N×1 , in S3, the second-order point attention mechanism is expressed as:

[0018] s=σ(W2RC(Cov(σ(W1(GMP(X))))))

[0019] Where Cov(·) is the covariance matrix of the calculation point, RC(·) is the row convolution, GMP(·) is the global maximum pooling, σ is the ReLU activation function, W1∈R t×N With W2∈R N×t are two different fully connected layers, X is a point X∈R in a given K-th voxel N×C ;

[0020] The second-order channel attention mechanism is similar to the second-order point attention mechanism. After the channel features pass through the second-order attention module, the output produces similar weights. In S4, the second-order channel attention mechanism is expressed as:

[0021] M=σ(W2RC(Cov(σ(W1(GMP(Y))))))

[0022] Where Y∈R C×H×W is the feature of the pseudo image, and the superscripts H and W are the height and width of the pseudo image.

[0023] Different weights are assigned to each pixel in the pseudo space according to the importance of the region in the pseudo space to the task, so as to obtain more accurate detection results. The feature P and signal G of the pseudo space are used as input, and the spatial attention weight generated by the final output is S. The pseudo image spatial attention mechanism is expressed as:

[0024]

[0025] The relationship between P and G can be expressed as:

[0026]

[0027] Where, and To perform linear transformation operation using 1×1 convolution, To perform a linear transformation operation on P using a 1×1 convolution, To perform a linear transformation operation on G using a 1×1 convolution.

[0028] The 3D ground truth box is parameterized as (x, y, z, w, l, h, θ), where (x, y, z) represents the center position, (w, l, h) and θ represent the size and orientation angle of the box, and the localization regression residual between the ground truth and the anchor is defined as follows:

[0029]

[0030] Δθ=sin(θ gt -θ a ),

[0031] Where, gt is the ground truth, a is the parameter of the anchor box, (x gt ,y gt ,z gt ) is the center coordinate of the 3D truth box, (l gt ,w gt ,h gt ) is the length, width and height of the 3D truth box, θ gt is the yaw angle of the 3D truth frame around the Z axis, (x a,y a ,z a ) is the center coordinate of the anchor box, (l a ,w a ,h a ) are the length, width and height of the anchor box.

[0032] θ a is the yaw angle of the anchor box around the Z axis;

[0033] The regression loss is expressed as:

[0034]

[0035] Where SmoothL1 is the SmoothL1 loss function;

[0036] Because the angle localization loss cannot distinguish the flipped box, a softmax classification loss L is used in the discretized direction dir , which enables the network to learn the orientation. For object classification loss, a focal loss is used:

[0037] L cls =-a(1-p) r log p

[0038] Where p is the probability of correctly detecting the box, r and a are parameter settings, and the final total loss is expressed as:

[0039]

[0040] Where N pos represents the number of correct detection boxes, β loc , β cls , β dir is the default value.

[0041] The present invention has the advantages and beneficial effects:

[0042] The second-order point attention mechanism in the present invention takes into account the correlation between points in voxels. Compared with the existing method of processing points in voxels in isolation, it can retain more useful geometric feature information and improve the accuracy of detection. Similarly, the second-order channel attention mechanism in the present invention takes into account the correlation between channels, further improving the detection accuracy. The pseudo-image space attention mechanism proposed in the present invention takes into account that not all pseudo-space features have the same contribution to the detection task. The greater the relevance to the task, the greater the importance of the region. Therefore, different weights are assigned to each pixel point in the pseudo-space feature, further improving its feature extraction effect. Therefore, the three mechanisms based on point columns ensure both a relatively high detection speed and the accuracy of extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a diagram of an embodiment of the second-order attention module architecture of the present invention;

[0044] Figure 2 This is the overall framework diagram of the 3D point cloud target detection method based on the second-order multi-attention mechanism of point columns in the present invention. DETAILED DESCRIPTION

[0045] The embodiments of the present invention are described in detail below. The following embodiments are implemented based on the technical solutions of the present invention, and provide detailed implementation methods and specific operating procedures. However, the protection scope of the present invention is not limited to the following embodiments.

[0046] The present invention provides a method for implementing target detection based on three mechanisms: a second order point attention mechanism (SOPA), a second order channel attention mechanism (SOCA), and a spatial attention mechanism (SAPI). The three mechanisms include:

[0047] S1: Propose three mechanisms based on point column second-order point attention mechanism, second-order channel attention mechanism, and pseudo image space attention mechanism to realize target detection respectively;

[0048] S2: Based on S1, a network is provided. The network mainly consists of a second-order point attention mechanism, a point column feature network, a second-order channel attention mechanism, a backbone network, a pseudo-image space attention mechanism, and an SSD detection head. The network is also divided into a second-order attention module, a second-order point attention module, and a second-order channel attention module.

[0049] S3: voxelize the point cloud, then perform a second-order point attention mechanism on the point cloud to convert it into pseudo-image features;

[0050] S4: Perform a second-order channel attention mechanism on the features of the pseudo image and output the features of the pseudo space;

[0051] S5: Perform pseudo-image space attention mechanism operation on the features of pseudo space and output the detection results;

[0052] Among them, the SSD detection head uses the features of the backbone to predict the three-dimensional bounding box of the object; the second-order attention module includes global maximum pooling, covariance pooling and row convolution; when the point feature is used as the input of the second-order attention module in S3, the second-order point attention mechanism weight will be obtained as the output, and this process is the second-order point attention module; when the channel feature is input into the second-order attention module, the second-order channel attention mechanism weight will be obtained, and this process is the second-order channel attention module.

[0053] In a given K-th voxel, for all points X in the voxel K ∈R N×C , where N represents the maximum number of points, C represents the number of channels, and after global maximum pooling, the vector E consisting of the maximum values ​​on each dimension is obtained K ∈R N ×1 , E K ∈R N×1 Input to a fully connected layer, where N×1 represents a vector of N rows and 1 column, and the vector Q is obtained K ∈R t×1 , where t is the number of points after the reduction by the W1 fully connected layer. The ReLU activation function is used after the W1 fully connected layer to calculate the covariance matrix I between two points in the same voxel K ∈R t×t , where t is the number of points in the second-order point attention mechanism, t is the number of channels in the second-order channel attention mechanism, and t×t is the dimension. The covariance matrix is ​​convolved row by row to obtain the vector J∈R t×1 Then the vector J∈R t×1 Input to the W2 fully connected layer and use the activation function Sigmoid function to obtain the N-dimensional attention vector In S3, the second-order point attention mechanism is expressed as:

[0054] s=σ(W2RC(Cov(σ(W1(GMP(X))))))

[0055] Where Cov(·) is the covariance matrix of the calculation point, RC(·) is the row convolution, GMP(·) is the global maximum pooling, σ is the ReLU activation function, W1∈R t×N With W2∈R N×t are two different fully connected layers, X is a point X∈R in a given K-th voxel N×C ;

[0056] The second-order channel attention mechanism is similar to the second-order point attention mechanism. After the channel features pass through the second-order attention module, the output produces similar weights. In S4, the second-order channel attention mechanism is expressed as:

[0057] M=σ(W2RC(Cov(σ(W1(GMP(Y))))))

[0058] Where Y∈R C×H×W is the feature of the pseudo image, and the superscripts H and W are the height and width of the pseudo image.

[0059] Different weights are assigned to each pixel in the pseudo space according to the importance of the region in the pseudo space to the task, so as to obtain more accurate detection results. The feature P and signal G of the pseudo space are used as input, and the spatial attention weight generated by the final output is S. The pseudo image spatial attention mechanism is expressed as:

[0060]

[0061] The relationship between P and G can be expressed as:

[0062]

[0063] Where, and To perform linear transformation operation using 1×1 convolution, To perform a linear transformation operation on P using a 1×1 convolution, To perform a linear transformation operation on G using a 1×1 convolution.

[0064] The 3D ground truth box is parameterized as (x, y, z, w, l, h, θ), where (x, y, z) represents the center position, (w, l, h) and θ represent the size and orientation angle of the box, and the localization regression residual between the ground truth and the anchor is defined as follows:

[0065]

[0066] Δθ=sin(θ gt -θ a ),

[0067] Where, gt is the ground truth, a is the parameter of the anchor box, (x gt ,y gt ,z gt ) is the center coordinate of the 3D truth box, (l gt ,w gt ,h gt ) is the length, width and height of the 3D truth box, θ gt is the yaw angle of the 3D truth frame around the Z axis, (x a ,y a ,z a ) is the center coordinate of the anchor box, (l a ,w a ,h a) are the length, width and height of the anchor box.

[0068] θ a is the yaw angle of the anchor box around the Z axis;

[0069] The regression loss is expressed as:

[0070]

[0071] Where SmoothL1 is the SmoothL1 loss function;

[0072] Because the angle localization loss cannot distinguish the flipped box, a softmax classification loss L is used in the discretized direction dir , which enables the network to learn the orientation. For object classification loss, a focal loss is used:

[0073] L cls =-a(1-p) r log p

[0074] Where p is the probability of correctly detecting the box, r and a are parameter settings, and the final total loss is expressed as:

[0075]

[0076] Where N pos Indicates the number of correct detection boxes. Here β loc =2,β cls =1,β dir =0.2.

[0077] The above describes in detail the preferred embodiments of the present invention. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible by those skilled in the art without inventive effort. Therefore, any technical solution that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A 3D point cloud object detection method based on a second-order multi-attention mechanism based on point pillars, characterized by: include: S1: Propose three mechanisms based on point column second-order point attention mechanism, second-order channel attention mechanism, and pseudo image space attention mechanism to realize target detection respectively; S2: Based on S1, a network is provided, which consists of a second-order point attention mechanism, a point column feature network, a second-order channel attention mechanism, a backbone network, a pseudo-image space attention mechanism and an SSD detection head. The network is also divided into a second-order attention module, a second-order point attention module and a second-order channel attention module; S3: voxelize the point cloud, then perform a second-order point attention mechanism on the point cloud to convert it into pseudo-image features; S4: Perform a second-order channel attention mechanism on the features of the pseudo image and output the features of the pseudo space; S5: Perform pseudo-image space attention mechanism operation on the features of pseudo space and output the detection results; Among them, the SSD detection head uses the features of the backbone to predict the 3D bounding box of the object; the second-order attention module includes global maximum pooling, covariance pooling and row convolution; when the point feature is used as the input of the second-order attention module in S3, the second-order point attention mechanism weight will be obtained as the output, and this process is the second-order point attention module; when the channel feature is input to the second-order attention module, the second-order channel attention mechanism weight will be obtained, and this process is the second-order channel attention module; In a given K-th voxel, for all points X in the voxel K ∈R N×C , where N represents the maximum number of points, C represents the number of channels, and after global maximum pooling, the vector E consisting of the maximum values ​​on each dimension is obtained K ∈R N×1 R , E K ∈R N×1 Input to a fully connected layer, where N×1 represents a vector of N rows and 1 column, and the vector Q is obtained K ∈R t×1 , where t is the number of points after the reduction by the W1 fully connected layer. The ReLU activation function is used after the W1 fully connected layer to calculate the covariance matrix I between two points in the same voxel K ∈R t×t , where t is the number of points in the second-order point attention mechanism, t is the number of channels in the second-order channel attention mechanism, and t×t is the dimension. The covariance matrix is ​​convolved row by row to obtain the vector J∈R t ×1 Then the vector J∈R t×1 Input to the W2 fully connected layer and use the activation function Sigmoid function to obtain the N-dimensional attention vector s∈R N×1 , in S3, the second-order point attention mechanism is expressed as: s=σ(W2RC(Cov(σ(W1(GMP(X)))))) Where Cov(·) is the covariance matrix of the calculation point, RC(·) is the row convolution, GMP(·) is the global maximum pooling, σ is the ReLU activation function, W1∈R t×N With W2∈R N×t are two different fully connected layers, X is a point X∈R in a given K-th voxel N×C ; The second-order channel attention mechanism is similar to the second-order point attention mechanism. After the channel features pass through the second-order attention module, the output produces similar weights. In S4, the second-order channel attention mechanism is expressed as: M=σ(W2RC(Cov(σ(W1(GMP(Y)))))) Where Y∈R C×H×W is the feature of the pseudo image, and the superscripts H and W are the height and width of the pseudo image.

2. The method for 3D point cloud object detection based on a second-order multi-attention mechanism using point pillars according to claim 1, characterized in that: Different weights are assigned to each pixel in the pseudo space according to the importance of the region in the pseudo space to the task, so as to obtain more accurate detection results. The feature P and signal G of the pseudo space are used as input, and the spatial attention weight generated by the final output is S. The pseudo image spatial attention mechanism is expressed as: The relationship between P and G can be expressed as: Where, To perform linear transformation operations using 1×1 convolution, To perform a linear transformation operation on P using a 1×1 convolution, To perform a linear transformation operation on G using a 1×1 convolution.

3. The method for 3D point cloud object detection based on a second-order multi-attention mechanism using point pillars according to claim 2, characterized in that: The 3D ground truth box is parameterized as (x, y, z, w, l, h, θ), where (x, y, z) represents the center position, (w, l, h) and θ represent the size and orientation angle of the box, and the localization regression residual between the ground truth and the anchor is defined as follows: Δθ=sin(θ gt -θ a ), Where, gt is the ground truth, a is the parameter of the anchor box, (x gt ,y gt ,z gt ) is the center coordinate of the 3D truth box, (l gt ,w gt ,h gt ) is the length, width and height of the 3D truth box, θ gt is the yaw angle of the 3D truth frame around the Z axis, (x a ,y a ,z a ) is the center coordinate of the anchor box, (l a ,w a ,h a ) is the length, width and height of the anchor box, θ a is the yaw angle of the anchor box around the Z axis; The regression loss is expressed as: Where SmoothL1 is the SmoothL1 loss function; Because the angle localization loss cannot distinguish the flipped box, a softmax classification loss L is used in the discretized direction dir , which enables the network to learn the orientation. For object classification loss, a focal loss is used: L cls =-a(1-p) r In the log p formula, p is the probability of correctly detecting the box, r and a are parameter settings, and the final total loss is expressed as: Where N pos represents the number of correct detection boxes, β loc , β cls , β dir is the default value.

Images