A point cloud scene flow prediction method based on attention mechanism
Through the point cloud scene flow prediction method based on attention mechanism, local areas are directly modeled and aggregated, which solves the problem of high speed and computing requirements in point cloud scene flow prediction, achieves efficient and accurate prediction effects, and improves applicability in real scenarios.
Patent Information
- Application Number
- CN202210063938.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-20
- Publication Date
- 2025-09-02
- Estimated Expiration
- 2042-01-20
AI Technical Summary
The existing methods run slowly in point cloud scene flow prediction, have high computing requirements, and are difficult to deploy in real autonomous driving scenarios, and have high labeling costs and insufficient generalization capabilities.
The point cloud scene flow prediction method based on attention mechanism is adopted, and the real scene flow data set is generated through pseudo-labels, and local areas are directly modeled and aggregated. The point cloud neural network is used for feature extraction and upsampling, combined with the output layer of the residual connection is corrected, and L1 Loss is used for optimization training.
The operation speed and accuracy of point cloud scene flow prediction are improved, the calculation requirements are reduced, and the generalization ability in real scenes is enhanced.
Smart Images
Figure CN114387422B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer vision and autonomous driving, and in particular to a point cloud scene flow prediction method based on an attention mechanism. Background Art
[0002] With the development of artificial intelligence and deep learning, intelligence and automation have become the new era trends. Autonomous driving technology has emerged in this context, with enormous social and commercial value. In recent years, as autonomous driving research has matured and deepened, autonomous vehicles have gradually moved from demonstration-level to real-world deployment. Autonomous driving is a complex and extensive system, encompassing perception of the surrounding environment, positioning using GPS and maps, and motion planning following this perception and positioning information. Each of these stages involves numerous subcomponents, collectively forming the complete autonomous driving technology process. Unlike previous AI efforts focused on images, audio, and video, autonomous vehicles operate directly in the real three-dimensional world, presenting greater challenges. For example, they must be able to quickly and accurately predict and understand the three-dimensional motion information of their surroundings. For autonomous vehicles, accurately estimating the three-dimensional motion of their surroundings not only helps them better perceive the environment but also enables more accurate motion planning and decision-making based on the motion of objects. Conversely, the lack of 3D motion information or misprediction can lead to collisions with the surrounding environment, resulting in irreparable consequences. Therefore, accurate and real-time prediction of the surrounding three-dimensional motion information is crucial to the accuracy and safety of autonomous driving. Currently, 3D point cloud data acquired by LiDAR (Lidar) is becoming a core and reliable data source for autonomous driving, as it provides accurate 3D position and geometric structure information. The goal of scene flow is to estimate spatial motion by predicting the 3D motion vector of each point between two frames based on this point cloud data. This task requires predicting the 3D motion vector of each point in space from two adjacent frames of point cloud data. The output of this prediction serves as input for downstream tasks such as point cloud tracking and segmentation, helping autonomous vehicles better understand their environment and plan their movements.
[0003] In the task of scene flow prediction, traditional methods such as ICP utilize an iterative approach to match two point cloud frames for prediction. With the advancement of deep learning, deep learning-based algorithms such as FlowNet3D have gradually become the mainstream approach for scene flow prediction due to their higher accuracy. At the same time, the attention mechanism has demonstrated its superiority in multiple fields, including images, video, and text. Since point cloud data is a collection of discrete and unordered three-dimensional points, and the attention mechanism itself is a set operation, it is naturally well-suited for processing point cloud data. Furthermore, since scene flow requires processing two adjacent frames, the attention mechanism can more simply and efficiently fuse the information between the two point clouds, facilitating the extraction of motion features from temporal information. Therefore, we combine the attention mechanism with point cloud scene flow prediction to automatically predict the motion vectors of three-dimensional point clouds, achieving accurate and reliable results while maintaining a high operating speed to ensure that autonomous vehicles can respond more efficiently and quickly to their surroundings. At the same time, we proposed a pseudo-label data annotation method that can quickly and accurately generate a large amount of scene flow training data in real scenarios. This large amount of data can help us further enhance the robustness and generalization ability of the method so that it can run stably in different real scenarios.
[0004] A series of deep learning-based point cloud neural networks, such as PointNet and PointNet++, provide an end-to-end architecture and method for efficient point cloud feature extraction. These networks have been widely used in autonomous driving and 3D vision applications, including 3D detection, 3D segmentation, and 3D tracking. Point cloud feature extraction networks can effectively address the difficulties inherent in the disordered and discrete nature of point clouds.
[0005] Most current methods employ an encoder-decoder architecture, using a point cloud feature extraction network to encode two adjacent input point clouds, mapping them into a high-dimensional feature space. A fusion network then fuses the high-dimensional features of the two point clouds. Finally, a complex decoder architecture decodes the fused high-dimensional features and outputs the scene flow for each point. However, this complex encoder-decoder architecture results in slower algorithm execution and places higher demands on the computing power of autonomous vehicles, limiting the deployment of scene flow prediction in real-world autonomous driving scenarios. Summary of the Invention
[0006] In response to the above-mentioned deficiencies in the prior art, the present invention provides a point cloud scene flow prediction method based on an attention mechanism, which solves the problem that point clouds are difficult to handle due to their disorder and discreteness.
[0007] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a point cloud scene flow prediction method based on the attention mechanism, comprising the following steps:
[0008] S1. Build a real scene flow dataset;
[0009] S2. Preprocess the input data based on a real scene flow dataset, establish a point cloud feature extraction model, generate candidate scene flows for local areas, aggregate the candidate scene flows through an attention mechanism, upsample the candidate scene flows, and modify the output using an output layer with a residual connection;
[0010] S3, using the L1 Loss of the predicted scene flow and the real scene flow as the loss to optimize and train the network;
[0011] S4. Test the trained network on the constructed real scene flow dataset to obtain test results.
[0012] Furthermore, step S1 is specifically as follows: based on the public autonomous driving datasets KITTI and Waymo, a pseudo-label annotation method is used to generate point cloud scene flow datasets KITTI_SF and Waymo_SF in real lidar scenes, and the foreground point cloud is separated from the background point cloud by the object annotation box in the autonomous driving dataset. Assuming that the background point cloud remains stationary in two adjacent frames of the point cloud, the scene flow pseudo-label of the background point cloud can be obtained by using the vehicle posture transformation between the two frames obtained by the IMU sensor, and the same object in adjacent frames of the foreground point cloud is linked by the Track ID of the object annotation box, and the scene flow pseudo-label of each foreground object is obtained by the posture of the object box in the two frames of the point cloud.
[0013] Furthermore: the specific steps of step S2 are:
[0014] S21, performing uniform preprocessing on the input point cloud at the input level so that each frame of input point cloud is collected into a uniform number;
[0015] S22, use point-based point cloud neural network to extract features from the input point cloud, set up two layers of point cloud feature extraction layers, and encode the point cloud into a shallow feature space;
[0016] S23, after the feature extraction of the two-frame point cloud is completed, the candidate scene flow of the local area is generated;
[0017] S24, aggregate the candidate scene flows through the attention mechanism to obtain the scene flow of low-resolution points;
[0018] S25, upsampling the scene flow of the low-resolution point using the method in step S24;
[0019] S26. Use an output layer with a residual connection, and use the point cloud features of the penultimate layer as the residual to correct the final output after passing through two linear layers.
[0020] Furthermore, step S21 specifically includes randomly selecting a specified number of point cloud frames that are greater than a preset value, and using a random repeated sampling method to supplement the input point cloud to a given number of point cloud frames that are less than a preset value.
[0021] Furthermore, step S22 is specifically as follows: obtaining a specified number of key points from the input point cloud by the farthest point sampling method, taking the key point as the center, finding the area points within a certain radius to construct a local point set, applying a multi-layer perceptron and maximum pooling to the point set to extract the geometric structure information of the local area and aggregate it to the center point to complete the extraction of one layer of features.
[0022] Furthermore, step S23 is specifically as follows: taking each point of the current frame as the center point, searching for its domain points in the next frame according to a pre-set radius, and after obtaining the domain point set, calculating the position vector of each domain point of the second frame to the center point of the first frame, thereby forming a candidate scene flow.
[0023] Furthermore, step S24 is specifically as follows: using the feature of the center point as the query, the feature of each domain point as the key, and the constructed candidate scene flow as the value, the attention mechanism is used to obtain the weight of each candidate scene flow, and the candidate scene flow is weighted summed to obtain the point scene flow. The calculation formula is:
[0024]
[0025] In the above formula, flow is the scene flow of the domain point, i is the coordinate index of the domain point, and n is the number of domain points.
[0026] Furthermore, step S25 is specifically as follows: for each point in the high-resolution point cloud, its boundary point is searched in the low-resolution point cloud whose scene flow has been calculated with a preset radius to form a candidate point set, and then the feature of the high-resolution point is used as the query, the feature of the low-resolution point is used as the key, and the low-resolution scene flow is used as the value. The high-resolution point scene flow is calculated by the formula in step S24.
[0027] Furthermore, step S3 is specifically as follows: using the L1 Loss of the predicted scene flow and the real scene flow as the loss to optimize the network. Except for the prediction and real scene flow of the last layer, L1 Loss of different resolutions is added to each layer of attention for supervision, which is conducive to faster convergence of the network. During training, the network is trained end-to-end using the Aadam optimizer.
[0028] The beneficial effects of the present invention are as follows: the present invention proposes a point cloud scene flow prediction network based on an attention mechanism, which directly models the geometric structure of the local area of the first frame point cloud and the second frame point cloud, obtains the candidate scene flow, and then aggregates it through the attention mechanism. It no longer needs to go through the complex process of encoding the input point cloud into a deep high-dimensional space and then decoding it, which greatly improves the reasoning speed of the point cloud scene flow. At the same time, because the attention mechanism can more accurately model the local area of the point cloud, this method can maintain a high prediction accuracy while greatly improving the running speed. In addition, since the labeling cost of the point cloud scene flow is very expensive, most of the existing methods are based on synthetic data sets of CAD models for training. The present invention improves the algorithm principle and proposes a real lidar point cloud scene flow pseudo-label generation method, further improving the algorithm performance and generalization ability in real scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is the overall flow chart of the present invention;
[0030] Figure 2 This is the network structure diagram of the attention mechanism adopted in this invention. DETAILED DESCRIPTION
[0031] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.
[0032] like Figure 1 As shown, a point cloud scene flow prediction method based on attention mechanism includes the following steps:
[0033] S1. Constructing a real scene flow dataset. Because manual annotation of real point cloud scene flow labels is extremely difficult, we first generate a point cloud scene flow dataset for real lidar scenarios using pseudo-label annotation methods based on the publicly available autonomous driving datasets KITTI and Waymo. The KITTI dataset, jointly developed by the Karlsruhe Institute of Technology in Germany and Toyota Research Institute of America, is currently the most commonly used dataset for evaluating computer vision algorithms in autonomous driving scenarios. The Waymo dataset, a data open source project (WaymoOpen Dataset) announced on its blog by Waymo, the autonomous driving company under Google's parent company Alphabet, has become another important dataset for autonomous driving tasks due to its larger data volume and more complex scenes. First, we use the object annotation boxes in the autonomous driving dataset to separate the foreground point cloud from the background point cloud. For the background point cloud, we assume that it remains stationary between two adjacent point cloud frames. Therefore, we can directly obtain the scene flow pseudo-labels for the background point cloud using the ego-vehicle pose transformation between the two frames obtained by the IMU sensor. For the foreground point cloud, we use the TrackID of the object annotation box to link the same objects in adjacent frames. The scene flow pseudo-labels for each foreground object are obtained based on the pose of the object box in the two point clouds. Specifically, two scene flow datasets, KITTI_SF and Waymo_SF, were constructed. The training set of KITTI_SF contains 20 point cloud sequences with a total of 8,008 point cloud pairs, and the test set is 150 point cloud pairs of KITTI SceneFlow; Waymo_SF contains 100 point cloud sequences with a total of 15,000 point cloud pairs. The training set and test set are divided into 85% and 15% respectively.
[0034] S2. Preprocess the input data based on the real scene flow dataset, establish a point cloud feature extraction model, generate candidate scene flows in the local area, and aggregate the candidate scene flows through the attention mechanism. The local area attention mechanism adopted by the present invention is as follows: Figure 2 As shown in Figure 2, the candidate scene stream is upsampled and the output is corrected using an output layer with a residual connection.
[0035] S21. At the input level, perform uniform preprocessing on the input point cloud, so that each frame of the input point cloud is first sampled to a uniform number. For KITTI_SF, the input points are sampled to 16,384 points, and for Waymo_SF, the input points are sampled to 40,000 points. For frames with more point clouds than the preset value, a specified number of points are randomly selected. For frames with less than the preset value, the input point cloud is supplemented to the given number by random repeated sampling.
[0036] S22. The present invention adopts a point-based point cloud neural network to extract features from the input point cloud. First, a specified number of key points are obtained from the input point cloud through the farthest point sampling method. Then, with these key points as the center points, the area points within a certain radius are found to construct a local point set. Then, the multi-layer perceptron (MLP) and maximum pooling (Max-Pooling) are applied to the point set to extract the geometric structure information of the local area and aggregate it to the center point, thereby completing the extraction of one layer of features. Two such point cloud feature extraction layers are set in this method, which can encode the point cloud into a relatively shallow feature space, without the need for four-layer feature extraction like other methods, greatly improving the computational efficiency.
[0037] S23. After feature extraction is completed for both frames of point cloud, candidate scene streams are generated for the local area. Specifically, each point in the current frame is used as the center point, and its domain point is searched in the next frame according to a pre-set radius. This cross-frame search structure enables the direct establishment of local connections between two adjacent frames of point clouds. After obtaining the neighborhood point set, the position vector from each domain point of the second frame to the center point of the first frame is calculated to form a candidate scene stream. This utilizes the precise local geometric position information of the two frames of point clouds, and no longer requires feature-level fusion and decoding to obtain the scene stream.
[0038] S24. After obtaining the candidate scene streams, we aggregate them using the attention mechanism. Specifically, we use the features of the center point as the query, the features of each domain point as the key, and the constructed candidate scene streams as the value. We use the attention mechanism to obtain the weight of each candidate scene stream, and finally perform a weighted sum of the candidate scene streams to obtain the scene stream for that point.
[0039]
[0040] S25. Scene flow upsampling module. After obtaining the low-resolution point scene flow, it is upsampled using the same method as in step 4. Specifically, for each point in the high-resolution point cloud, its neighborhood point is searched within the low-resolution point cloud for which the scene flow has been calculated, using a pre-set radius to form a candidate point set. The high-resolution point features are then used as the query, the low-resolution point features as the key, and the low-resolution scene flow as the value. The same attention operation as in step 4 is then performed to obtain the high-resolution point scene flow.
[0041] S26. Output layer. The present invention uses an output layer with a residual connection. The point cloud features of the penultimate layer are passed through two linear layers as the residual to correct the final output. This method can correct scene flow prediction values with large errors caused by outliers.
[0042] S3. The network is optimized and trained using the L1 Loss of the predicted scene flow and the real scene flow. The point cloud scene flow is predicted by the optimized and trained network. The present invention uses the L1 Loss of the predicted scene flow and the true scene flow as the loss to optimize the network. In addition to the prediction and true scene flow of the last layer, the present invention adds L1 loss of different resolutions to each layer of the attention mechanism for supervision, which is conducive to faster convergence of the network. During training, the Adam optimizer is used to train the network end-to-end for a total of 150 rounds. The initial learning rate is 0.001, which is reduced to one-tenth of the original value at 40 and 75 rounds respectively.
[0043] S4. Test the trained network on the constructed real scene flow dataset to obtain test results.
Claims
1. A point cloud scene flow prediction method based on attention mechanism, characterized in that: The following steps are involved: S1. Build a real scene flow dataset; S2. Preprocess the input data based on a real scene flow dataset, establish a point cloud feature extraction model, generate candidate scene flows for local areas, aggregate the candidate scene flows through an attention mechanism, upsample the candidate scene flows, and modify the output using an output layer with a residual connection; S3, using the L1 Loss of the predicted scene flow and the real scene flow as the loss to optimize and train the network; S4. Test the trained network on the constructed real scene flow dataset to obtain test results; The specific steps of step S2 are: S21, performing uniform preprocessing on the input point cloud at the input level so that each frame of input point cloud is collected into a uniform number; S22, use point-based point cloud neural network to extract features from the input point cloud, set up two layers of point cloud feature extraction layers, and encode the point cloud into a shallow feature space; S23, after the feature extraction of the two-frame point cloud is completed, the candidate scene flow of the local area is generated; S24, aggregate the candidate scene flows through the attention mechanism to obtain the scene flow of low-resolution points; S25, upsampling the scene flow of the low-resolution point using the method in step S24; S26, using an output layer with a residual connection, the point cloud features of the penultimate layer are passed through two linear layers as the residual to correct the final output; The step S25 is specifically as follows: for each point in the high-resolution point cloud, its boundary point is searched in the low-resolution point cloud for which the scene flow has been calculated with a preset radius to form a candidate point set, and then the feature of the high-resolution point is used as the query, the feature of the low-resolution point is used as the key, and the low-resolution scene flow is used as the value. The high-resolution point scene flow is calculated by the formula in step S24.
2. The point cloud scene flow prediction method based on the attention mechanism according to claim 1 is characterized in that Specifically, step S1 comprises: based on the public autonomous driving datasets KITTI and Waymo, a pseudo-label annotation method is used to generate point cloud scene flow datasets KITTI_SF and Waymo_SF in real lidar scenes, and the foreground point cloud is separated from the background point cloud by the object annotation box in the autonomous driving dataset. Assuming that the background point cloud remains stationary in two adjacent frames of the point cloud, the scene flow pseudo-label of the background point cloud can be obtained by using the vehicle posture transformation between the two frames obtained by the IMU sensor, and the same object in adjacent frames of the foreground point cloud is linked by the Track ID of the object annotation box, and the scene flow pseudo-label of each foreground object is obtained by the posture of the object box in the two frames of the point cloud.
3. The point cloud scene flow prediction method based on the attention mechanism according to claim 2 is characterized in that The step S21 specifically includes: randomly selecting a specified number of point cloud frames that are greater than a preset value, and using a random repeated sampling method to supplement the input point cloud to a given number of point cloud frames that are less than a preset value.
4. The point cloud scene flow prediction method based on the attention mechanism according to claim 3 is characterized in that The step S22 is specifically as follows: obtaining a specified number of key points from the input point cloud by the farthest point sampling method, taking the key point as the center, finding the area points within a certain radius to construct a local point set, applying a multi-layer perceptron and maximum pooling to the point set to extract the geometric structure information of the local area and aggregate it to the center point to complete the extraction of one layer of features.
5. The point cloud scene flow prediction method based on the attention mechanism according to claim 4 is characterized in that: The step S23 is specifically as follows: taking each point of the current frame as the center point, searching for its domain points in the next frame according to a pre-set radius, and after obtaining the domain point set, calculating the position vector of each domain point of the second frame to the center point of the first frame to form a candidate scene flow.
6. The point cloud scene flow prediction method based on attention mechanism according to claim 5 is characterized in that The step S24 is specifically as follows: using the feature of the center point as the query, the feature of each domain point as the key, and the constructed candidate scene flow as the value, the attention mechanism is used to obtain the weight of each candidate scene flow, and the candidate scene flow is weighted summed to obtain the point scene flow. The calculation formula is: In the above formula, flow is the scene flow of the field point, i is the coordinate index of the field point, n is the number of field points.
7. The point cloud scene flow prediction method based on attention mechanism according to claim 1 is characterized in that Specifically, step S3 is as follows: L1 Loss of the predicted scene flow and the real scene flow is used as the loss to optimize the network. Except for the prediction and real scene flow of the last layer, L1 Loss of different resolutions is added to each layer of attention for supervision, which is conducive to faster convergence of the network. During training, the Adam optimizer is used to train the network end-to-end.