A lidar target detection method based on spatiotemporal attention mechanism

Abstract

This invention discloses a laser radar target detection method based on a spatiotemporal attention mechanism, comprising a model training phase and a target detection phase. During the model training phase, model weights are updated through multiple cycles to obtain final model weights, which include an attention weight matrix, thereby improving target detection accuracy. During the target detection phase, the attention weight matrix is ​​used to extract features from key target areas, and the laser radar point cloud of the current frame is sampled based on the feature extraction results, thereby achieving target detection. Based on the attention mechanism, the present invention takes into account the spatial variations of the detection target, thereby improving sampling speed and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of laser radar target detection, and in particular to a laser radar target detection method based on a spatiotemporal attention mechanism. Background Art

[0002] Currently, advanced intelligent driving technology is developing rapidly. With the increasing maturity of lidar (LiDAR) sensing sensors, target perception based on lidar point clouds has become a crucial component of advanced intelligent driving. There are two main approaches to 3D target perception based on lidar point clouds: 1. Filtering, ground segmentation, and point cloud clustering are performed on the raw lidar point cloud to achieve 3D target perception. However, this traditional lidar target perception method suffers from the inability to identify specific target categories, and the clustering results are easily affected by point cloud noise and ground segmentation. 2. Convolutional neural network technologies, such as PointPillars and 3DSSD, build neural network models for 3D target perception based on lidar point clouds. These technologies collect tens of thousands of lidar point cloud data, annotate targets, and detect target-related features through model training. Target perception based on convolutional neural network models can significantly improve target perception accuracy and achieve good results in various scenarios. Currently, this is the most widely used technology for intelligent driving target perception.

[0003] Prior art 1 discloses a 4D millimeter-wave three-dimensional target detection method based on a self-attention mechanism, including: real-time acquisition and preprocessing of 4D millimeter-wave radar point cloud data; inputting the preprocessed 4D millimeter-wave radar point cloud data into a pre-trained three-dimensional target detection model, and outputting target detection results; the three-dimensional target detection model includes: a bird's-eye view voxelization module, a column self-attention feature extraction module, a CNN backbone network and a PRN detection head; the bird's-eye view voxelization module is used to voxelize the 4D millimeter-wave radar point cloud data from a bird's-eye view perspective to extract feature information F of the entire space; the column self-attention feature extraction module is used to use the feature information F to extract global features of the point cloud based on the self-attention mechanism to generate a BEV pseudo image; the CNN backbone network is used to extract features from the BEV pseudo image and output a feature map; the PRN detection head is used to perform target detection on the feature map and output a 3D target detection result.

[0004] The shortcomings of the first existing technique include: 1) The downsampling process of the pillars generated by the bird's-eye view voxelization module loses some millimeter-wave radar point cloud information, resulting in lower-quality target detection results. 2) Attention feature extraction based on the segmented pillars fails to leverage the prior information from the current frame's target detection results on the next frame's point cloud, resulting in limited improvements in detection accuracy and speed achieved through attention feature extraction.

[0005] Prior art 2 discloses a three-dimensional target detection method that fuses multi-perspective features of 4D millimeter wave and laser point cloud, the method comprising: simultaneously collecting millimeter wave radar point cloud data and laser radar point cloud data; inputting the millimeter wave radar point cloud data and the laser radar point cloud data into a pre-established and trained millimeter wave and laser radar fusion network, and outputting three-dimensional target detection results; the millimeter wave and laser radar fusion network is used to learn the interaction information of the laser radar and the millimeter wave radar from the BEV perspective, and to learn the interaction information of the laser radar and the millimeter wave radar from the perspective, and to splice this interaction information to achieve the fusion of the millimeter wave radar point cloud data and the laser radar point cloud data.

[0006] The shortcomings of the second existing technique include: 1) The voxelization module outputs Pillar features from millimeter-wave radar point cloud data and LiDAR point cloud data, which are then converted to the BEV view for target detection. However, detection accuracy and speed are limited by the size of the divided pillar features. 2) Detecting multi-scale targets using a single-scale feature map is poor. Summary of the Invention

[0007] The purpose of the present invention is to provide a lidar target detection method based on a spatiotemporal attention mechanism to improve the target detection accuracy and target detection rate of the lidar.

[0008] To solve the above technical problems, the present invention provides a technical solution: a lidar target detection method based on a spatiotemporal attention mechanism, which builds a target detection model, including a model training phase and a target detection phase;

[0009] In the model training phase, the target labeled dataset is used for training. The following steps are performed on the dataset frame by frame:

[0010] S101: If the current frame is the first frame of the LiDAR point cloud, sample the first frame of the LiDAR point cloud according to a certain sampling rule to obtain the sampling initial seed set P seed If the current frame is not the first frame of the lidar point cloud, the attention map F containing multiple key target areas obtained in S107 is used RIO , according to the attention map F of the key target area RIO The result of the convolution is used to sample the current frame lidar point cloud to obtain the sampling initial seed set P seed ;

[0011] S102, sampling the initial seed set P seed Perform fusion sampling to obtain the feature point cloud P FS ;

[0012] S103, the feature point cloud P FS Mapping to BEV view;

[0013] S104, build a multi-scale feature pyramid backbone network based on the FPN network, and transform the BEV view M bev Input into the backbone network and extract feature maps of different scales through convolution operations;

[0014] S105. Using a fully connected layer under the feature maps of each scale to construct a multi-scale detection head; using the multi-scale detection head to perform target detection on the lidar point cloud of the current frame to obtain a target detection result;

[0015] S106, selecting the next frame of lidar point cloud;

[0016] S107, based on the motion information v of the laser radar carrier and the time interval t between the current frame and the previous adjacent frame, use motion transformation to calculate the rotation transformation matrix T, and then extract the key target area in the laser radar point cloud of the current frame based on the target posture information p and dimension information s in the target detection result of S105, and combine the obtained rotation transformation matrix T, and form an attention map F with the key target areas corresponding to multiple targets. RIO ;

[0017] S108, return to S101;

[0018] In the model training phase, S101 to S108 are iterated in a loop, and the loss is calculated based on the target detection results obtained by the target detection model and the target annotation true value in the data set used for training. The model weight is updated in multiple cycles to obtain the final model weight. The final model weight includes the attention weight matrix F w ;

[0019] In the target detection phase, the target detection model is configured according to the final model weights obtained in the model training phase. The target detection model performs the following steps in the target detection phase:

[0020] S201: If the current frame is the first frame of the laser radar point cloud, the first frame of the laser radar point cloud is sampled according to a certain sampling rule to obtain the sampling initial seed set P seed If the current laser radar point cloud is not the first frame, then the attention map F containing multiple key target areas obtained in S207 is obtained. RIO , using the attention weight matrix F obtained in the model training phase w Perform feature extraction on the key target area F, sample the current frame lidar point cloud according to the feature extraction results, and obtain the sampling initial seed set P seed ;

[0021] Then, S202 to S207 are executed in sequence. The specific operation steps of S202 to S207 are the same as S102 to S107;

[0022] Finally, execute S208 and return to S201.

[0023] According to the above scheme, the data set used in the model training phase has labeled targets including motor vehicles, non-motor vehicles and pedestrians.

[0024] According to the above scheme, in S101, when the current frame is not the first frame of the lidar point cloud, the initial seed set P is sampled. seed The way to obtain it is to focus on the attention map F of the key target area RIO Convolution obtains feature F out ,

[0025] F out =Conv(F RIO )

[0026] Then F out And the current frame point cloud set P obtains the sampling initial seed set index S index ,

[0027] S index =F out *P

[0028] Then, according to the sampling initial seed set index S index Extract the sampling initial seed set P from the current frame point cloud set P seed ,

[0029] P seed =P[S index ].

[0030] According to the above scheme, the fusion sampling in S102 is specifically to use the farthest point sampling based on Euclidean distance and the farthest point sampling based on feature distance in the 3DSSD backbone network to sample the initial seed set P respectively. seed Sampling is performed to obtain the sampling results Ld(P) and Lf(P), and then the sampling initial seed set P is obtained based on the fusion strategy C(P) seed The fusion sampling result P FS , the fusion strategy C(P) is,

[0031] C(P)=Ld(P)+Lf(P).

[0032] According to the above scheme, the loss function in PointPillar is used to calculate the loss during the model training phase.

[0033] According to the above scheme, the Adam optimizer is used to update the model weights during the model training phase.

[0034] The beneficial effects of the present invention are:

[0035] 1. Target region extraction and sampling seed set extraction are performed based on the spatiotemporal attention mechanism, which improves sampling accuracy and speed. It contains a rich feature point cloud and can improve the resolution of the BEV view without increasing additional computational costs. The high-resolution BEV view can improve target detection accuracy.

[0036] 2. The multi-scale feature pyramid backbone network and multi-scale detection head can improve the detection rate of target detection and cover targets of different scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of a lidar target detection method based on a spatiotemporal attention mechanism according to an embodiment of the present invention;

[0038] Figure 2 2 is a schematic diagram of the backbone network structure of a multi-scale feature pyramid according to an embodiment of the present invention.

[0039] Figure 2 Conv1, Conv2, and Conv3 are multi-layer convolutional neural networks, M L 、M M 、M S They are feature maps used to detect large-scale targets, medium-scale targets, and small-scale targets, respectively. DETAILED DESCRIPTION

[0040] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0041] A lidar target detection method based on the spatiotemporal attention mechanism. This method builds a target detection model, including a model training phase and a target detection phase.

[0042] The training dataset in the model training phase is a LiDAR point cloud collected in real scenes and annotated frame by frame. The annotated targets include motor vehicles, non-motor vehicles, and pedestrians. The target detection model performs the following steps in the model training phase:

[0043] S101: If the current frame is the first frame of the laser radar point cloud, the center of gravity of the first frame of the laser radar point cloud is calculated, and the point farthest from the center of gravity in the first frame of the laser radar point cloud is selected as the sampling initial seed set P. seed If the current laser radar point cloud is not the first frame, then the attention map F containing multiple key target areas obtained in S107 is obtained.RIO , the attention map F of the key target area RIO Convolution obtains feature F out , that is, F out =Conv(F RIO ), and then according to F out And the current frame point cloud set P obtains the sampling initial seed set index S index , specifically S index =F out *P; then according to the sampling initial seed set index S index Extract the sampling initial seed set P from the current frame point cloud set P seed , that is, P seed =P[S index ];

[0044] S102, using the farthest point sampling based on Euclidean distance and the farthest point sampling based on feature distance in the 3DSSD backbone network to sample the initial seed set P respectively seed Sampling is performed to obtain the sampling results Ld(P) and Lf(P) respectively; based on the fusion strategy C(P), the sampling initial seed set P is obtained seed The fusion sampling result is the feature point cloud P FS , the size of the feature point cloud is (N m , C m ), where N m is the number of point clouds after fusion sampling, C m is the number of point cloud feature channels after fusion sampling;

[0045] S103, based on the feature point cloud P FS The (X, Y) coordinate information of the feature point cloud P FS Mapped to the BEV view with a certain mapping resolution (the mapping resolution f in this embodiment is 0.10m. Considering that the original lidar point cloud has been fused and sampled, this mapping resolution can improve the accuracy of the BEV view); the generated BEV view M bev Size (H, W, C m ), where H and W represent the length and width of the BEV view respectively, and M bev The pixel value M bev (i, j) = max(P cell ), where i and j are the coordinate values ​​of the pixels in the BEV view, P cell is the point cloud set within the pixel range of a certain pixel;

[0046] S104, build a multi-scale feature pyramid backbone network based on the FPN network, and transform the BEV view M bev Input into the backbone network and extract feature maps of different scales through convolution operations;

[0047] S105. Use a fully connected layer to construct a multi-scale detection head under the feature maps of each scale; use the multi-scale detection head to perform target detection on the lidar point cloud of the current frame, and obtain the target detection result after NMS processing;

[0048] S106, selecting the next frame of lidar point cloud;

[0049] S107, based on the motion information v of the laser radar carrier and the time interval t between the current frame and the previous adjacent frame, use motion transformation to calculate the rotation transformation matrix T, and then extract the key target area in the laser radar point cloud of the current frame based on the target posture information p and dimension information s in the target detection result of S105, and combine the obtained rotation transformation matrix T, and form an attention map F with the key target areas corresponding to multiple targets. RIO ;

[0050] S108, return to S101;

[0051] In the model training phase, S101 to S108 are iterated in a loop. The target detection results obtained by the target detection model and the target annotation true value in the training dataset are calculated through the loss function in PointPillar, and the model weight is updated using the Adam optimizer. After multiple iterations, the final model weight is obtained. The final model weight includes the attention weight matrix F w ;

[0052] In the target detection phase, the target detection model is configured according to the final model weights obtained in the model training phase. The target detection model performs the following steps in the target detection phase:

[0053] S201: If the current frame is the first frame of the laser radar point cloud, the center of gravity of the first frame of the laser radar point cloud is calculated, and the point farthest from the center of gravity in the first frame of the laser radar point cloud is selected as the sampling initial seed set P. seed If the current laser radar point cloud is not the first frame, then the attention map F containing multiple key target areas obtained in S207 is obtained. RIO , using the attention weight matrix F obtained in the model training phase w Attention map F for the key target area RIO Perform feature extraction and obtain F out =F w (F RIO ), and then according to F out And the current frame point cloud set P obtains the sampling initial seed set index S index , specifically S index =F out *P; then according to the sampling initial seed set index S indexExtract the sampling initial seed set P from the current frame point cloud set P seed , that is, P seed =P[S index ];

[0054] Then, S202 to S207 are executed in sequence. The specific operation steps of S202 to S207 are the same as S102 to S107;

[0055] Finally, execute S208 and return to S201.

[0056] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A lidar target detection method based on a spatiotemporal attention mechanism, characterized by: This method builds a target detection model, including the model training phase and the target detection phase; In the model training phase, the target labeled dataset is used for training. The following steps are performed on the dataset frame by frame: S101: If the current frame is the first frame of the LiDAR point cloud, sample the first frame of the LiDAR point cloud according to a certain sampling rule to obtain the sampling initial seed set P seed If the current frame is not the first frame of the lidar point cloud, the attention map F containing multiple key target areas obtained in S107 is used RIO , according to the attention map F of the key target area RIO The result of the convolution is used to sample the current frame lidar point cloud to obtain the sampling initial seed set P seed ; S102, sampling the initial seed set P seed Perform fusion sampling to obtain the feature point cloud P FS ; S103, the feature point cloud P FS Mapping to BEV view; S104, build a multi-scale feature pyramid backbone network based on the FPN network, and transform the BEV view M bev Input into the backbone network and extract feature maps of different scales through convolution operations; S105. Using a fully connected layer under the feature maps of each scale to construct a multi-scale detection head; using the multi-scale detection head to perform target detection on the lidar point cloud of the current frame to obtain a target detection result; S106, selecting the next frame of lidar point cloud; S107, based on the motion information v of the laser radar carrier and the time interval t between the current frame and the previous adjacent frame, use motion transformation to calculate the rotation transformation matrix T, and then extract the key target area in the laser radar point cloud of the current frame based on the target posture information p and dimension information s in the target detection result of S105, and combine the obtained rotation transformation matrix T, and form an attention map F with the key target areas corresponding to multiple targets. RIO ; S108, return to S101; In the model training phase, S101 to S108 are iterated in a loop, and the loss is calculated based on the target detection results obtained by the target detection model and the target annotation true value in the data set used for training. The model weight is updated in multiple cycles to obtain the final model weight. The final model weight includes the attention weight matrix F w ; In the target detection phase, the target detection model is configured according to the final model weights obtained in the model training phase. The target detection model performs the following steps in the target detection phase: S201: If the current frame is the first frame of the laser radar point cloud, the first frame of the laser radar point cloud is sampled according to a certain sampling rule to obtain the sampling initial seed set P seed If the current laser radar point cloud is not the first frame, then the attention map F containing multiple key target areas obtained in S207 is obtained. RIO , using the attention weight matrix F obtained in the model training phase w Perform feature extraction on the key target area F, sample the current frame lidar point cloud according to the feature extraction results, and obtain the sampling initial seed set P seed ; Then, S202 to S207 are executed in sequence. The specific operation steps of S202 to S207 are the same as S102 to S107; Finally, execute S208 and return to S201.

2. The lidar target detection method based on the spatiotemporal attention mechanism according to claim 1, characterized in that: The dataset used in the model training phase has labeled targets including motor vehicles, non-motor vehicles, and pedestrians.

3. The lidar target detection method based on the spatiotemporal attention mechanism according to claim 1, characterized in that: In S101, when the current frame is not the first frame of the lidar point cloud, the initial seed set P is sampled. seed The way to obtain it is to focus on the attention map F of the key target area RIO Convolution obtains feature F out , F out =Conv(F RIO ) Then F out And the current frame point cloud set P obtains the sampling initial seed set index S index , S index =F out *P Then, according to the sampling initial seed set index S index Extract the sampling initial seed set P from the current frame point cloud set P seed , P seed =P[S index ]。 4. The lidar target detection method based on the spatiotemporal attention mechanism according to claim 1, characterized in that: The fusion sampling in S102 is specifically to use the farthest point sampling based on Euclidean distance and the farthest point sampling based on feature distance in the 3DSSD backbone network to sample the initial seed set P respectively. seed Sampling is performed to obtain the sampling results LD(P) and Lf(P), and then the sampling initial seed set P is obtained based on the fusion strategy C(P) seed The fusion sampling result P FS , the fusion strategy C(P) is, C(P)=Ld(P)+Lf(P).

5. The lidar target detection method based on the spatiotemporal attention mechanism according to claim 1, characterized in that: The loss function in PointPillar is used to calculate the loss during the model training phase.

6. The lidar target detection method based on the spatiotemporal attention mechanism according to claim 1, characterized in that: During the model training phase, the Adam optimizer is used to update the model weights.

Images