A real-time 3D target detection method for lidar point clouds suitable for unmanned vehicles

Abstract

The present invention discloses a real-time three-dimensional target detection method for laser radar point clouds suitable for unmanned vehicles, comprising the following steps: target detection data set preparation; model training; parameter update; simulation data preparation; model reasoning and result evaluation based on simulation data; model deployment and real vehicle data preparation; model reasoning and data preservation based on real vehicle data; data upload and annotation; incremental learning and parameter update. The three-dimensional target detection model proposed in the present invention specifically includes: a preprocessing module; a backbone network module; a neck network module; a feature selection module; and a detection head module. The three-dimensional target detection model proposed in the present invention mines richer spatial information by converting point cloud data into a variety of bird's-eye view images; uses a two-dimensional distance-aware sparse convolutional layer for feature extraction, and realizes multi-view feature fusion through a dual-branch network; combines a multi-scale feature pyramid network and a feature selection module to effectively improve the detection accuracy of distant targets and small targets; and by constructing an efficient neural network architecture and combining training, simulation testing and real data incremental learning strategies, effectively improves the robustness, accuracy and adaptability of the target detection model, providing powerful perception support for unmanned driving systems.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and in particular to a real-time three-dimensional target detection method using a laser radar point cloud suitable for unmanned vehicles. Background Art

[0002] 3D object detection technology is at the core of autonomous driving research. It enables vehicles to perceive their surroundings in three dimensions. By accurately identifying 3D information about surrounding objects, it enhances navigation accuracy, improves driving safety, and reduces accident risks. With the advancement of sensors and artificial intelligence, the continuous innovation of 3D object detection technology is also driving the widespread application of autonomous driving technology and the further development of intelligent transportation systems.

[0003] Among the majority of existing LiDAR point cloud-based 3D object detection methods, deep learning technology has become the dominant technology in the field due to its superior nonlinear modeling capabilities and automatic feature extraction capabilities. However, in complex traffic scenarios, existing object detection methods often suffer from low detection accuracy when processing distant or small targets. For distant targets, traditional point cloud processing methods often result in large detection errors due to the small target size and low resolution. In addition, small targets such as pedestrians and cyclists typically have low point cloud density and unclear geometric features, making it difficult for detection models to effectively extract their features, thus affecting detection accuracy. Summary of the Invention

[0004] In response to the shortcomings of existing technologies, the present invention aims to provide a real-time 3D target detection method for unmanned vehicles using LiDAR point clouds. This method primarily addresses the detection accuracy issues for long-range and smaller targets in 3D target detection. The present invention employs the following technical solutions:

[0005] A real-time 3D target detection method for lidar point clouds suitable for unmanned vehicles includes the following steps:

[0006] S1. Object detection dataset preparation: When the object detection model is not fully trained, the autonomous driving object detection dataset is used as the training, validation, and test set for the object detection model. The target categories include cars, pedestrians, cyclists, and other categories. Data augmentation is performed on the training set, including splicing point cloud data of various target objects randomly selected from other frames in the training set into the current frame, and randomly flipping and scaling the point cloud of the target objects in the horizontal plane.

[0007] S2. Model training: Train an end-to-end object detection model, which includes the following modules:

[0008] (1) Preprocessing module: preprocess the point cloud data in the target detection dataset;

[0009] (2) Backbone network module: connected to the preprocessing module to extract features from the preprocessed data;

[0010] (3) Neck network module: connected to the backbone network module, it fuses and enhances the multi-scale feature maps;

[0011] (4) Feature selection module: connected to the neck network module, it selects the feature map of the corresponding scale according to the size of the target;

[0012] (5) Detection head module: connected to the feature selection module, predicting the target category and bounding box based on the feature map after feature selection;

[0013] S3, Parameter Update: The model's weights and bias parameters are adjusted by calculating the gradient of the loss function. The Adam optimization algorithm is used, with the initial learning rate set to 0.001 and gradually decayed during training to ensure rapid model convergence while preventing overfitting. Each update round uses the backpropagation algorithm to obtain gradient information and combines L2 regularization to suppress overfitting.

[0014] S4. Simulation data preparation: To generate simulation data and annotations consistent with the dataset format, a virtual environment containing roads, vehicles, pedestrians, cyclists, and other objects is built in the simulation platform. The lidar sensor is configured, and parameters such as scanning angle and resolution are set to be consistent with the target detection dataset. The virtual vehicle equipped with the lidar moves in the scene to collect point cloud data transmitted by the lidar. The simulation platform automatically generates annotation information, which includes the coordinates of the center point of the 3D bounding box of the target object, the size of the 3D bounding box, the heading angle, and the category. The collected point cloud data is preprocessed, including adding appropriate noise and simulating missing data. The preprocessed point cloud data and annotations are converted to a format consistent with the target detection dataset, and the files are timestamp aligned and path normalized to ensure that they can be directly used by the target detection model.

[0015] S5. Model reasoning and result evaluation based on simulation data: In order to verify the model performance and optimize the data set to ensure that the detection effect meets expectations, the target detection model after training and parameter update in step S3 is deployed in a simulation test environment. The target detection model uses the simulation data output in step S4 as input and generates target detection results through model reasoning, including the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, the category and the detection score. The result evaluation adopts a series of key indicators, including average precision, false alarm rate, recall rate, and F1 score. Whether the model performance meets the expected effect is judged based on whether the indicators are higher than the set threshold. If it meets the expected effect, the next step is entered; otherwise, the process returns to step S1 and the target detection data set is further enhanced, including expanding the data of specific scenarios that lead to poor simulation results.

[0016] S6. Model deployment and real-vehicle data preparation: To implement model deployment and prepare compatible data, and ensure target detection accuracy in a real-vehicle environment, the target detection model trained in step S5 and meeting the expected results is deployed on an unmanned vehicle equipped with a lidar. When the unmanned vehicle is in driving mode, the raw point cloud data collected by the lidar on the unmanned vehicle is converted in real time into point cloud data in the same format as the target detection dataset. The point cloud data is preprocessed, including adding noise, filling missing values, time alignment, and coordinate conversion. Through feature selection and format conversion, the data is standardized to the structure of the target detection dataset, and outliers are screened and removed. The processed point cloud data is stored in the target detection dataset format to ensure that it can be directly used by the target detection model.

[0017] S7. Model inference and data storage based on real vehicle data: To achieve real-time inference, result display and storage of real vehicle data and support subsequent model optimization, the processed point cloud data obtained in step S6 is input into the target detection model deployed on the unmanned vehicle in real time, frame by frame. The target detection results are generated through model inference, including the coordinates of the center point of the 3D bounding box of the target object, the size of the 3D bounding box, the heading angle, the category, and the detection score. The 3D bounding box is projected onto the 2D image captured by the camera at the same time and visualized on the screen. The point cloud data, target detection results, and user feedback on the target detection results (including missed and false detections) are saved.

[0018] S8, Data upload and annotation: Manually set the data upload cycle, and upload data to the cloud once per cycle; when the unmanned vehicle is not in driving mode, if the data upload cycle has not arrived, return to execute steps S6 and S7; if the data upload cycle has arrived, upload the point cloud data, target detection results, and user feedback information saved in step S7 to the cloud, and filter and process them based on data quality, data relevance, and data diversity; use automated annotation tools to set the detection confidence threshold, and combine the target detection results above the detection confidence threshold to perform bounding box and category annotation on all targets to generate an incremental data set;

[0019] S9, incremental learning and parameter update: To achieve continuous optimization of the target detection model, use the incremental dataset generated in step S8 to fine-tune the parameters of the target detection model, and use regularization constraints to prevent new knowledge from overwriting old knowledge; push the target detection model with updated parameters back to the unmanned vehicle; return to step S6, and repeat S6 to S9 to continuously update and optimize the target detection model.

[0020] Furthermore, the preprocessing module described in step S2 is used to preprocess the point cloud data. The horizontal area of ​​interest is defined as [-60, 60] × [0, 80] meters, and a fixed height range is selected based on the operating range of the lidar sensor to filter out ground point clouds and high-altitude point clouds. The point cloud data is denoised and filtered. The denoised and filtered point cloud is projected from three-dimensional space to the bird's-eye view plane with a resolution of 0.1 meters to generate a two-dimensional image representation of the height map, intensity map, density map, and curvature map of the same size. The height map is represented as:

[0021]

[0022] Where z is the height of the laser point, P i,j is the set of all laser points projected on the pixel grid where the coordinates (i, j) of the bird's-eye view are located, H i,j Represents the maximum height of all laser points in the set. Through logarithmic transformation, the influence of extreme values ​​on the target detection model is reduced, thereby improving the stability and generalization ability of the target detection model. The intensity map is represented as:

[0023]

[0024] Where r is the intensity of the laser point, N i,j is the number of laser points projected into the pixel grid where the coordinates (i, j) of the bird’s-eye view are located, I i,j Represents the average intensity of all laser points within the grid. The density map is represented by:

[0025]

[0026] The curvature graph is represented as:

[0027]

[0028] where X i,j is the three-dimensional coordinate of the laser point with the largest intensity projected on the pixel grid where the coordinate (i, j) of the bird's-eye view is located, Represents the set of neighborhood sample points centered on the laser point, where the number of sample points is n, X k is the three-dimensional coordinate of the neighborhood laser point; the height map, intensity map, density map and curvature map are stacked in the depth dimension to generate a three-dimensional bird's-eye view of four channels.

[0029] Furthermore, the backbone network module described in step S2 adopts a dual-branch deep convolutional neural network structure, including an image processing branch and a point cloud processing branch. Among them, the image processing branch uses the three-dimensional bird's-eye view generated by the preprocessing module as input and uses ResNet-50 as a feature encoder. Due to the sparsity of point cloud data, the projected image on the two-dimensional plane is also sparse. It is not efficient to use traditional convolutional neural networks to process sparse images. Therefore, a sparse convolution module is used to replace the first stage of ResNet-50. Taking into account the irregular distribution characteristics of point clouds and the more obvious defects of point clouds on distant objects or small targets, a two-dimensional distance-aware focal sparse convolution module is designed to replace the first stage of ResNet-50 to solve the above problems by focusing on the importance differences of point cloud data. The second to fifth stages of ResNet-50 generate 4 multi-scale feature maps with resolutions reduced to 1 / 2 of the previous stage respectively. The point cloud processing branch uses the backbone network of PointNet++ as the feature extraction network and takes the denoised and filtered point cloud data as input; the point cloud processing branch contains 1 SA layer, each of which extracts local features through the farthest point sampling and neighborhood aggregation operations. and global features Aggregate each layer of global features again to obtain the final global features The global feature is combined with the features extracted by the image processing branch By adding element by element, we can get the fusion feature F fu At the same time, feature propagation (l FP layers) is used to restore the spatial resolution of global features layer by layer until the original point cloud resolution. In the whole process, l takes the value of 4.

[0030] Furthermore, the neck network module described in step S2 adopts a feature pyramid structure based on the feature maps generated in the second to fourth stages of ResNet-50. Fusion feature F fuAnd the feature maps of each resolution extracted by each FP layer of PointNet++ Perform feature fusion and enhancement to improve the model's detection performance for multi-scale targets. Feature map and F fu Through the top-down network of the feature pyramid network, the high-level feature map is sampled layer by layer by interpolation, and is fused with the current layer feature map by element-by-element addition to generate a pyramid feature. Multi-scale pyramid features Feature maps of each resolution extracted by the FP layer The corresponding feature fusion is performed element by element to obtain the final features of each level of the pyramid The value of l is 4 throughout the process.

[0031] Furthermore, the feature selection module described in step S2, in order to learn the importance of different scales, combines the multi-scale feature map generated by the neck network module By interpolation, the size of the largest feature map is adjusted to obtain a feature map with the same size. The soft weights are generated by the softmax operation, and the final output features are obtained by linear weighted summation. The specific formula is:

[0032]

[0033] Here, l represents the number of layers, which is 4. This results in the final output feature that integrates the advantages of each scale and is optimized.

[0034] Furthermore, the detection head module described in step S2 uses two sub-heads to jointly complete the detection task: a center point heatmap head and a regression detection head. The center point heatmap head predicts the heatmap and the probability that each pixel position in the bird's-eye view is the center point of a certain category of object; the regression detection head predicts the center point offset, size, and angular information of the target object.

[0035] The total loss function consists of heatmap loss and bounding box regression loss, which is expressed as:

[0036] L total =α·L heatmap +β·L bbox

[0037] Among them L heatmap represents the heat map loss, comparing the predicted heat map and the real heat map, using the Gaussian focus loss function; L bbox represents the bounding box regression loss, which compares the predicted and actual target object center point offset, size and orientation angle, and adopts the smooth L1 loss function; α and β represent the weights of the heat map loss and bounding box regression loss, respectively, and are both set to 1.

[0038] Furthermore, the two-dimensional distance-aware focal sparse convolution module used in the backbone network module consists of three parts:

[0039] (1) Predicting feature importance: Convert the 3D bird's-eye view image into a sparse image consisting of sparse tensors, and extract the sparse features of each non-zero pixel of the sparse image through a submanifold sparse convolution layer; Use the sigmoid function to predict the importance score of the sparse features, with the score between 0 and 1:

[0040] I p =σ((W*X) p )

[0041] Where σ(·) represents the sigmoid function, (W*X) p Represents the element value of the feature map obtained by the convolution operation at feature p;

[0042] (2) Screening the predicted importance map: Screen the top k most important input features using the Top-k method:

[0043]

[0044] Among them, Top k (·) represents the Top-k method, P in is the input feature space, k accounts for 50% of the total input features; considering the impact of the distance between the target and the sensor on the point cloud density, the importance score of the prediction is weighted based on the distance, represents the predicted importance score after distance weighting, which is calculated as follows:

[0045]

[0046] The weight w d (p) The Euclidean distance between the feature point and the vehicle sensor is used for calculation:

[0047] w d (p) = 1-exp(-λ|X p -X o ‖)

[0048] where X o Indicates the coordinates of the laser radar, X p Represents the coordinates of feature point p, λ is a scaling factor that controls the sensitivity of normalization. A larger λ will make the normalized result approach 1 faster, and λ is 0.1;

[0049] (3) Dynamic output shape generation: For the features with high importance selected in (2), the surrounding extended features are screened through the Top-k algorithm to dynamically obtain extended features of different shapes, where k accounts for 50% of the total extended features; the output positions of other unimportant input features are the same as the input, essentially performing regular submanifold sparse convolution. In this way, through distance perception and dynamic output shape generation mechanism, more effective and adaptive convolution processing can be performed on important features while maintaining high efficiency.

[0050] The beneficial effects of the present invention are as follows: the present invention proposes a real-time three-dimensional target detection method for lidar point clouds suitable for unmanned vehicles. The proposed target detection model mines richer spatial information by converting point cloud data into a variety of bird's-eye view images; by adopting a two-dimensional distance-aware focal sparse convolution layer, dual-branch feature extraction and fusion, a feature pyramid network structure, and a comprehensive application of feature selection modules, it can improve the detection accuracy of distant targets and smaller targets. The present invention effectively improves the robustness, accuracy, and adaptability of the target detection model by constructing an efficient neural network architecture and combining training, simulation testing, and real data incremental learning strategies, providing powerful perception support for unmanned driving systems.

[0051] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a real-time three-dimensional target detection method using a laser radar point cloud suitable for an unmanned vehicle according to the present invention;

[0053] Figure 2 This is a diagram showing the overall structure of a 3D target detection model for a real-time 3D target detection method using a LiDAR point cloud suitable for unmanned vehicles according to the present invention;

[0054] Figure 3 This is a schematic diagram of the two-dimensional distance-aware focal sparse convolution module of the real-time three-dimensional target detection method of lidar point clouds suitable for unmanned vehicles described in the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below. It should be noted that this embodiment is based on the technical solution and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to this embodiment.

[0056] This embodiment provides a real-time three-dimensional target detection method for laser radar point cloud applicable to unmanned vehicles. Figure 1 As shown, the specific steps include:

[0057] S1. Preparation of target detection data set: When the target detection model has not completed training, the autonomous driving target detection data set is used as the training, verification and test set of the target detection model. The target categories include cars, pedestrians, cyclists and other categories. Then, the training set is enhanced. Specifically, first, for each frame in the training set, a certain number of point cloud data of various target objects are randomly selected from other frames in the training set, and they are spliced ​​into the current frame. Preferably, in this embodiment, point cloud data of 15 cars, 20 pedestrians and 20 cyclists are added to each frame respectively. Secondly, the point cloud of the target object is randomly flipped and scaled along the X-axis and Y-axis in the horizontal plane.

[0058] S2. Model training: Train the end-to-end target detection model. The overall structure is as follows Figure 2 As shown. The target detection model includes the following modules:

[0059] (1) Preprocessing module: used to preprocess the point cloud data in the target detection dataset.

[0060] Specifically, first, define the horizontal region of interest as [-60, 60] × [0, 80] meters, and select a fixed height range based on the operating range of the lidar sensor to filter out ground point clouds and high-altitude point clouds. Preferably, this embodiment filters within the height range of [-2.5, 1] ​​meters.

[0061] Secondly, the point cloud data is denoised and filtered.

[0062] Then, the denoised and filtered point cloud is projected from the three-dimensional space to the bird's-eye view plane with a resolution of 0.1 meters to generate a two-dimensional image representation of the height map, intensity map, density map, and curvature map with a size of 1200×800. The height map is represented as:

[0063]

[0064] Where z is the height of the laser point, P i,j is the set of all laser points projected on the pixel grid where the coordinates (i, j) of the bird's-eye view are located, H i,j represents the maximum height of all laser points in the set. The intensity map is represented as:

[0065]

[0066] Where r is the intensity of the laser point, N i,j is the number of laser points projected into the pixel grid where the coordinates (i, j) of the bird’s-eye view are located, I i,j Represents the average intensity of all laser points within the grid. The density map is represented by:

[0067]

[0068] The curvature graph is represented as:

[0069]

[0070] Among them, X i,j is the three-dimensional coordinate of the laser point with the largest intensity projected on the pixel grid where the coordinate (i, j) of the bird's-eye view is located, Represents the set of neighborhood sample points centered on the laser point, where the number of sample points is n, X k is the 3D coordinate of the neighborhood laser point. Preferably, in this embodiment, the number of sample points is 10.

[0071] Finally, the height map, intensity map, density map, and curvature map are stacked in the depth dimension to generate a 4-channel 3D bird's-eye view image with the size of 1200×800×4.

[0072] (2) Backbone network module: connected to the preprocessing module, used to extract features from the preprocessed data.

[0073] Specifically, the backbone network module adopts a dual-branch deep convolutional neural network structure, including an image processing branch and a point cloud processing branch.

[0074] The image processing branch takes the 3D bird's-eye view image generated by the preprocessing module as input and uses ResNet-50 as the feature encoder.

[0075] The first stage of ResNet-50 is replaced by a two-dimensional distance-aware focal sparse convolution module, which is as follows Figure 3 Specifically, the two-dimensional distance-aware focal sparse convolution module consists of three parts:

[0076] a. Predicting feature importance: Convert the 3D bird's-eye view image into a sparse image consisting of sparse tensors. Use a submanifold sparse convolutional layer to extract sparse features for each non-zero pixel in the sparse image. Use the sigmoid function to predict the importance score of the sparse features, with the score between 0 and 1:

[0077] I p =σ((W*X) p )

[0078] Where σ(·) represents the sigmoid function, (W*X) p Represents the element value of the feature map obtained by the convolution operation at feature p;

[0079] b. Filter the predicted importance map: Filter the top k most important input features using the Top-k method:

[0080]

[0081] Among them, Top k (·) represents the Top-k method, P in is the input feature space, k accounts for 50% of the total input features; represents the predicted importance score after distance weighting, which is calculated as follows:

[0082]

[0083] The weight w d (p) The Euclidean distance between the feature point and the vehicle sensor is used for calculation:

[0084] w d (p) = 1-exp(-λ|X p -X o ‖)

[0085] where X o Indicates the coordinates of the laser radar, X p represents the coordinates of the feature point p, λ is a scaling factor that controls the normalization sensitivity, preferably, λ is 0.1;

[0086] c. Dynamic output shape generation: For the filtered features with high importance, the surrounding extended features are screened through the Top-k algorithm to dynamically obtain extended features of different shapes, where k accounts for 50% of the total extended features. The output positions of other unimportant input features are the same as the input, essentially performing regular submanifold sparse convolution.

[0087] The second to fifth stages of ResNet-50 generate four multi-scale feature maps with resolutions reduced to 1 / 2 of the previous stage respectively. In this embodiment, the second to fifth stages of ResNet-50 generate feature maps of 600×400, 300×200, 150×100, and 75×50, respectively.

[0088] The point cloud processing branch uses the backbone network of PointNet++ as the feature extraction network to directly process point cloud data. The network contains l SA layers, each of which extracts local features through farthest point sampling and neighborhood aggregation operations. and global features Then aggregate the global features of each layer again to obtain the final global features Then the global feature is combined with the features extracted by the image processing branch By adding element by element, we can get the fusion feature F fu, for use by the subsequent neck network module. At the same time, feature propagation (l FP layer) is used to restore the spatial resolution of the global features layer by layer until the original point cloud resolution.

[0089] (3) Neck network module: connected to the backbone network module, used to fuse and enhance multi-scale feature maps.

[0090] Specifically, the neck network module adopts a feature pyramid structure based on the feature maps generated in the second to fourth stages of ResNet-50. Fusion feature F fu And the feature maps of each resolution extracted by each FP layer of PointNet++ Perform feature fusion and enhancement to improve the model's detection performance for multi-scale targets. Feature map and F fu The high-level feature maps are sampled layer by layer through the top-down network of the feature pyramid network, and are fused with the current layer feature maps to generate pyramid features. Specifically, at each level, the resolution of the high-level feature map is first adjusted by interpolation to match the resolution of the current layer, and then added element by element to the original feature map of the current layer to complete the fusion of multi-scale features and form a multi-scale pyramid feature. Feature maps of each resolution extracted by the FP layer in the point cloud processing branch The corresponding feature fusion is performed element by element to obtain the final features of each level of the pyramid The value of l is 4 throughout the process.

[0091] (4) Feature selection module: connected to the neck network module, it selects the feature map of the corresponding scale according to the size of the target.

[0092] Specifically, for the multi-scale feature map generated by the neck network module First, each scale map is adjusted to the size of the largest feature map by interpolation to obtain a feature map of consistent size. Then, a softmax operation is performed to generate soft weights to represent the importance of each scale. Finally, the final output feature is obtained through linear weighted summation. The specific formula is:

[0093]

[0094] Here, l represents the number of layers, which is 4. This results in the final output feature that integrates the advantages of each scale and is optimized.

[0095] (5) Detection head module: connected to the feature selection module, predicts the category and bounding box of the target based on the feature map after feature selection.

[0096] Specifically, the detection head uses two sub-heads to complete the detection task: a center point heatmap head and a regression detection head. The center point heatmap head predicts the heatmap, estimating the probability that each pixel in the bird's-eye view is the center point of a certain category of object. The regression detection head is used to predict other attribute information of the target object, including the center point offset, size, and orientation angle.

[0097] The total loss function consists of heatmap loss and bounding box regression loss, which is expressed as:

[0098] L total =α·L heatmap +β·L bbox

[0099] Among them L heatmap represents the heat map loss, comparing the predicted heat map and the real heat map, using the Gaussian focus loss function; L bbox represents the bounding box regression loss, which compares the predicted and actual target object center point offset, size and orientation angle, and adopts a smooth L1 loss function; α and β represent the weights of the heat map loss and bounding box regression loss, respectively, and both α and β are set to 1.

[0100] S3. Parameter Update: The model's weights and bias parameters are adjusted by calculating the gradient of the loss function. The Adam optimization algorithm is used, with an initial learning rate set to 0.001 and gradually decaying during training. Each update round uses backpropagation to obtain gradient information and incorporates L2 regularization to mitigate overfitting.

[0101] S4. Simulation data preparation: used to generate simulation data and annotations that are consistent with the format of the target detection dataset. First, a virtual environment containing roads, vehicles, pedestrians, cyclists and other objects is built in the simulation platform, and the lidar sensor is configured, and the scanning angle, resolution, etc. are set to be consistent with the target detection dataset. Secondly, the virtual vehicle equipped with the lidar is moved in the scene to collect point cloud data transmitted by the lidar. Then, the simulation platform automatically generates annotation information corresponding to the target object, including the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, the category, etc. In addition, the collected point cloud data is preprocessed, and appropriate noise and simulated missing data are added to improve authenticity. Finally, the preprocessed point cloud data and annotations are converted to a format consistent with the target detection dataset, and the files are timestamp aligned and path normalized to facilitate direct use by the target detection model. Preferably, this embodiment uses Carla as the simulation platform.

[0102] S5. Model reasoning and result evaluation based on simulation data:

[0103] First, model inference is performed. The target detection model trained and parameter-updated in step S3 is deployed in a simulation test environment. The simulation data output in step S4 is used as input. The target detection results are generated through model inference, including the coordinates of the center point of the target object's 3D bounding box, the size of the 3D bounding box, the heading angle, the category, and the detection score.

[0104] The results are then evaluated using a series of key indicators, including average precision, false alarm rate, recall rate, and F1 score. Whether the model performance meets the expected results is determined based on whether the indicators are higher than the set threshold. If so, proceed to the next step; otherwise, return to step S1 and further enhance the target detection dataset, including expanding specific scenario data (such as complex traffic conditions and extreme weather conditions) that lead to poor simulation results.

[0105] S6. Model deployment and real vehicle data preparation: The target detection model trained in step S5 and meeting the expected results is deployed on an unmanned vehicle equipped with a lidar. When the unmanned vehicle is in driving mode, the raw point cloud data collected from the lidar on the unmanned vehicle is converted in real time into point cloud data consistent with the format of the target detection dataset. Then, the point cloud data is preprocessed, including adding noise, filling missing values, time alignment, and coordinate conversion. Then, through feature selection and format conversion, the data is standardized to the structure of the target detection dataset, and outliers are screened and eliminated. Finally, the processed point cloud data is stored in the format of the target detection dataset, supporting direct use of the target detection model and ensuring seamless compatibility between the real vehicle data and the target detection dataset.

[0106] S7. Model inference and data storage based on real-world vehicle data: First, the processed point cloud data obtained in step S6 is input into the target detection model deployed on the unmanned vehicle in real time, frame by frame. Model inference generates target detection results, including the coordinates of the target object's 3D bounding box center point, 3D bounding box size, heading angle, category, and detection score. The 3D bounding box is then projected onto the 2D image captured by the camera at the same moment and visualized on the screen. Finally, the point cloud data, target detection results, and user feedback on the target detection results (including missed and false detections) are saved.

[0107] S8. Data upload and labeling: Manually set the data upload cycle, and upload data to the cloud once per cycle. Preferably, this embodiment sets the data upload once a week. When the unmanned vehicle is not in driving mode, if the data upload cycle has not been reached, return to execute steps S6 and S7; if the data upload cycle has been reached, the point cloud data, target detection results and user feedback on the results saved in step S7 are uploaded to the cloud, and screened and processed according to data quality, data relevance and data diversity. Then, use the automated labeling tool to set the detection confidence threshold, and combine the target detection results above the detection confidence threshold to perform bounding box and category labeling on all targets to form an incremental data set. Preferably, this embodiment uses Xtreme1 as an automated labeling tool.

[0108] S9, Incremental Learning and Parameter Update: Fine-tune the parameters of the object detection model using the incremental dataset generated in step S8, applying regularization constraints to prevent new knowledge from overwriting old knowledge. The updated object detection model is then pushed back to the autonomous vehicle. Returning to step S6, S6-S9 are repeated to continuously update and optimize the object detection model.

[0109] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A real-time 3D target detection method for lidar point clouds suitable for unmanned vehicles, characterized by: The steps include: S1. Target detection dataset preparation; S2. Model training: Train an end-to-end object detection model, which includes the following modules: (1) Preprocessing module: preprocess the point cloud data in the target detection dataset; (2) Backbone network module: connected to the preprocessing module to extract features from the preprocessed data; (3) Neck network module: connected to the backbone network module, it fuses and enhances the multi-scale feature maps; (4) Feature selection module: connected to the neck network module, it selects the feature map of the corresponding scale according to the size of the target; (5) Detection head module: connected to the feature selection module, predicting the target category and bounding box based on the feature map after feature selection; S3, parameter update: adjust the model's weights and bias parameters by calculating the gradient of the loss function; S4. Simulation Data Preparation: Build a virtual environment containing roads, vehicles, pedestrians, cyclists, and other objects in the simulation platform; configure the LiDAR sensor, setting the scanning angle and resolution consistent with the target detection dataset; move the virtual vehicle equipped with the LiDAR in the scene to collect point cloud data transmitted by the LiDAR; the simulation platform automatically generates annotation information; S5. Model reasoning and result evaluation based on simulation data: The target detection model trained and parameter-updated in step S3 is deployed in a simulation test environment. The target detection model uses the simulation data generated in step S4 as input and generates target detection results through model reasoning. The result evaluation uses a series of key indicators to determine whether the model performance meets the expected results based on whether the indicators are higher than the set threshold. If so, the next step is performed. Otherwise, the process returns to step S1 and further enhances the target detection dataset, including expanding the data for specific scenarios that lead to poor simulation results. S6. Model deployment and real-vehicle data preparation: The target detection model trained in step S5 and meeting the expected results is deployed on an unmanned vehicle equipped with a lidar. When the unmanned vehicle is in driving mode, the point cloud data collected by the lidar on the unmanned vehicle is converted in real time into point cloud data consistent with the target detection dataset format; S7, model reasoning and data storage based on real vehicle data: The processed point cloud data obtained in step S6 is input to the target detection model deployed on the unmanned vehicle in real time frame by frame, and the target detection results are generated through model reasoning; The preprocessing module described in step S2 defines the horizontal area of ​​interest as [-60, 60] × [0, 80] meters, and selects a fixed height range based on the operating range of the lidar sensor to filter out ground point clouds and high-altitude point clouds; denoises and filters the point cloud data; and projects the denoised and filtered point cloud from three-dimensional space to the bird's-eye view plane with a resolution of 0.1 meters to generate a two-dimensional image representation of the height map, intensity map, density map, and curvature map of the same size. The height map is represented as: Where z is the height of the laser point, P i,j is the set of all laser points projected on the pixel grid where the coordinates (i, j) of the bird's-eye view are located, H i,j represents the maximum height of all laser points in the set; the intensity map is represented by: Where r is the intensity of the laser point, N i,j is the number of laser points projected into the pixel grid where the coordinates (i, j) of the bird’s-eye view are located, I i,j Represents the average intensity of all laser points in the grid; the density map is expressed as: The curvature graph is represented as: where X i,j is the three-dimensional coordinate of the laser point with the largest intensity projected on the pixel grid where the coordinate (i, j) of the bird's-eye view is located, Represents the set of neighborhood sample points centered on the laser point, where the number of sample points is n, X k is the three-dimensional coordinate of the neighborhood laser point; the height map, intensity map, density map and curvature map are stacked in the depth dimension to generate a three-dimensional bird's-eye view of four channels.

2. The real-time three-dimensional target detection method for laser radar point cloud applicable to unmanned vehicles according to claim 1 is characterized in that: The backbone network module described in step S2 adopts a dual-branch deep convolutional neural network structure, including an image processing branch and a point cloud processing branch; wherein the image processing branch uses the three-dimensional bird's-eye view generated by the preprocessing module as input and uses ResNet-50 as a feature encoder; a two-dimensional distance-aware focal sparse convolution module is used to replace the first stage of ResNet-50; the second to fifth stages of ResNet-50 respectively generate four multi-scale feature maps with a resolution reduced to 1 / 2 of the previous stage The point cloud processing branch uses the backbone network of PointNet++ as the feature extraction network and takes the denoised and filtered point cloud data as input; the point cloud processing branch contains 1 SA layer, each of which extracts local features through the farthest point sampling and neighborhood aggregation operations. and global features Aggregate each layer of global features again to obtain the final global features F SA Features extracted by the image processing branch By adding element by element, we can get the fusion feature F fu At the same time, feature propagation is used to restore the spatial resolution of global features layer by layer through l FP layers until the original point cloud resolution. l represents the number of layers, and l takes the value of 4 in the whole process.

3. The real-time three-dimensional target detection method for laser radar point cloud applicable to unmanned vehicles according to claim 2 is characterized in that: The neck network module described in step S2 adopts a feature pyramid structure based on the feature maps generated in the second to fourth stages of ResNet-50 Fusion feature F fu And the feature maps of each resolution extracted by each FP layer of PointNet++ Perform feature fusion and enhancement; among them, the feature map and F fu Through the top-down network of the feature pyramid network, the high-level feature map is sampled layer by layer by interpolation, and is fused with the current layer feature map by element-by-element addition to generate a pyramid feature. Multi-scale pyramid features Feature maps of each resolution extracted by the FP layer The corresponding feature fusion is performed element by element to obtain the final features of each level of the pyramid l represents the number of layers, and the value of l is 4 in the whole process.

4. The real-time 3D target detection method for laser radar point cloud applicable to unmanned vehicles according to claim 1, characterized in that: The feature selection module described in step S2 selects the multi-scale feature map generated by the neck network module By interpolation, the size of the largest feature map is adjusted to obtain a feature map with the same size. Generate soft weights through softmax operation; obtain the final output features through linear weighted summation; The specific formula is: Where l represents the number of layers and its value is 4.

5. The real-time 3D target detection method for laser radar point cloud applicable to unmanned vehicles according to claim 1, characterized in that: The detection head module described in step S2 includes two sub-heads: a center point heat map head and a regression detection head. The center point heat map head predicts the heat map and the probability that each pixel position in the bird's-eye view is the center point of a certain category of target. The regression detection head predicts the center point offset, size, and orientation angle of the target object. The total loss function consists of heatmap loss and bounding box regression loss, which is expressed as: L total =α·L heatmap +β·L bbox Among them L eatmap represents the heat map loss, comparing the predicted heat map and the real heat map, using the Gaussian focus loss function; L bbox represents the bounding box regression loss, which compares the predicted and actual target object center point offset, size and orientation angle, and adopts the smooth L1 loss function; α and β represent the weights of the heat map loss and bounding box regression loss, respectively, and are both set to 1.

6. The real-time three-dimensional target detection method for laser radar point cloud applicable to unmanned vehicles according to claim 2, characterized in that: The two-dimensional distance-aware focal sparse convolution module consists of three parts: (1) Predicting feature importance: Convert the 3D bird's-eye view image into a sparse image consisting of sparse tensors, and extract the sparse features of each non-zero pixel of the sparse image through a submanifold sparse convolution layer; Use the sigmoid function to predict the importance score of the sparse features, with the score between 0 and 1: I p =σ((W*X) p ) Where σ(·) represents the sigmoid function, (W*X) p Represents the element value of the feature map obtained by the convolution operation at feature p; (2) Filter the predicted importance map: Filter the top k most important input features using the Top-k method: Among them, Top k (·) represents the Top-k method, P in is the input feature space, k accounts for 50% of the total input features; represents the predicted importance score after distance weighting, which is calculated as follows: The weight w d (p) The Euclidean distance between the feature point and the vehicle sensor is used for calculation: w d (p)=1-exp(-λ||X p -X o ||) where X o Indicates the coordinates of the laser radar, X p Represents the coordinates of feature point p, λ is a scaling factor that controls the normalization sensitivity, and λ is 0.1; (3) Dynamic output shape generation: For the top k most important input features selected in (2), the surrounding extended features are screened through the Top-k algorithm to dynamically obtain extended features of different shapes, where k accounts for 50% of the total extended features; the output positions of other unimportant input features are the same as the input, essentially performing regular submanifold sparse convolution.

7. The method according to claim 1, characterized in that The specific process of step S1 is as follows: when the object detection model has not completed training, the autonomous driving object detection dataset is used as the training, verification, and test set of the object detection model. The target categories include cars, pedestrians, cyclists, and other categories. The training set is augmented with data, including splicing point cloud data of various target objects randomly selected from other frames in the training set into the current frame, and randomly flipping and scaling the point cloud of the target object in the horizontal plane.

8. The method according to claim 1, characterized in that In step S3, the Adam optimization algorithm is used, the initial learning rate is set to 0.001, and it is gradually decayed during the training process; each update round is based on the back propagation algorithm to obtain gradient information, and L2 regularization is combined to suppress overfitting; Step S4 also includes preprocessing the collected point cloud data, including adding appropriate noise and simulating missing data; converting the preprocessed point cloud data and annotations into a format consistent with the target detection dataset, and aligning the timestamps and normalizing the paths of the files to ensure that they can be directly used by the target detection model; In step S4, the annotation information automatically generated by the simulation platform includes the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, and the category; the target detection results in steps S5 and S7 both include the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, the category, and the detection score; in step S5, the key indicators used include average precision, false alarm rate, recall rate, and F1 score.

9. The method according to claim 1, characterized in that In step S6, the point cloud data collected by the laser radar on the unmanned vehicle is preprocessed, including adding noise, filling missing values, time alignment and coordinate conversion; Through feature selection and format conversion, the data is standardized into the structure of the target detection dataset, and outliers are screened and eliminated; The processed point cloud data is stored in the target detection dataset format to ensure that it can be directly used by the target detection model; Step S7 further includes: projecting the three-dimensional bounding box onto the two-dimensional image captured by the camera at the same moment, and visually displaying it on the screen; The point cloud data, target detection results and user feedback on the target detection results are saved, and the feedback information includes target missed detection and false detection.

10. The method according to claim 1, characterized in that The process also includes step S8, data upload and annotation: manually setting a data upload cycle, uploading data to the cloud once per cycle; when the unmanned vehicle is not in driving mode, if the data upload cycle has not arrived, returning to steps S6 and S7; If the data upload period has arrived, the point cloud data, target detection results, and user feedback on the results saved in step S7 are uploaded to the cloud and screened based on data quality, data relevance, and data diversity; Use automated annotation tools to set a detection confidence threshold, combine the target detection results above the detection confidence threshold, and annotate all targets with bounding boxes and categories to generate an incremental dataset. The method further includes step S9, incremental learning and parameter updating: fine-tuning the parameters of the target detection model using the incremental dataset generated in step S8, using regularization constraints to prevent new knowledge from overwriting old knowledge; and pushing the target detection model with updated parameters back to the unmanned vehicle. Return to step S6 and repeat steps S6-S9 to continuously update and optimize the target detection model.

Images