Laser radar point cloud real-time three-dimensional target detection method suitable for unmanned vehicle

Abstract

The invention discloses a laser radar point cloud real-time three-dimensional target detection method suitable for an unmanned vehicle. The method comprises the following steps: preparing a target detection data set; model training; updating parameters; simulation data preparation; performing model reasoning and result evaluation based on simulation data; model deployment and real vehicle data preparation; performing model reasoning and data storage based on real vehicle data; uploading and marking the data; and incremental learning and parameter updating. The three-dimensional target detection model provided by the invention specifically comprises a preprocessing module; a backbone network module; a neck network module; a feature selection module; and a detection head module. According to the three-dimensional target detection model provided by the invention, the point cloud data is converted into various aerial view images, and richer spatial information is mined; a two-dimensional distance sensing sparse convolutional layer is adopted to perform feature extraction, and multi-view feature fusion is realized through a double-branch network; a multi-scale feature pyramid network and a feature selection module are combined, so that the detection precision of a long-distance target and a small target is effectively improved; by constructing an efficient neural network architecture and combining training, simulation testing and real data incremental learning strategies, the robustness, precision and adaptive capacity of a target detection model are effectively improved, and powerful perception support is provided for an unmanned driving system.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and in particular to a laser radar point cloud real-time three-dimensional target detection method suitable for unmanned vehicles. Background Art

[0002] 3D object detection technology is the core of unmanned driving research. It enables vehicles to perceive the environment in three dimensions, and enhances navigation accuracy, improves driving safety, and reduces accident risks by accurately identifying the 3D information of surrounding objects. With the upgrade of sensors and the development of artificial intelligence, the continuous innovation of 3D object detection technology also promotes the widespread application of unmanned driving technology and the further development of intelligent transportation systems.

[0003] Among most existing 3D target detection methods based on LiDAR point clouds, deep learning technology has become the dominant technology in the field of 3D target detection due to its excellent nonlinear modeling capabilities and automatic feature extraction functions. However, in the face of complex traffic scenes, existing target detection methods often face low detection accuracy when dealing with distant targets or small targets. For distant targets, traditional point cloud processing methods often result in large detection errors due to small target size and low resolution. In addition, small targets such as pedestrians and cyclists usually have low point cloud density and unclear geometric features, making it difficult for the detection model to effectively extract their features, thus affecting the detection accuracy. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention aims to provide a real-time three-dimensional target detection method for laser radar point cloud suitable for unmanned vehicles. The method mainly solves the detection accuracy problem of long-distance targets and smaller targets in three-dimensional target detection. The present invention adopts the following technical solutions:

[0005] A real-time three-dimensional target detection method for laser radar point cloud applicable to unmanned vehicles comprises the following steps:

[0006] S1. Preparation of target detection dataset: When the target detection model has not been trained, the autonomous driving target detection dataset is used as the training, verification and test set of the target detection model. The target categories include cars, pedestrians, cyclists and other categories. The training set is enhanced, including splicing the point cloud data of various target objects randomly selected from other frames in the training set to the current frame, and randomly flipping and scaling the point cloud of the target object in the horizontal plane.

[0007] S2. Model training: Train an end-to-end object detection model, which includes the following modules:

[0008] (1) Preprocessing module: preprocess the point cloud data in the target detection dataset;

[0009] (2) Backbone network module: connected to the preprocessing module to extract features from the preprocessed data;

[0010] (3) Neck network module: connected to the backbone network module to fuse and enhance multi-scale feature maps;

[0011] (4) Feature selection module: connected to the neck network module, selects the feature map of the corresponding scale according to the size of the target;

[0012] (5) Detection head module: connected to the feature selection module, predicting the category and bounding box of the target based on the feature map after feature selection;

[0013] S3, parameter update: adjust the weight and bias parameters of the model by calculating the gradient of the loss function; use the Adam optimization algorithm, set the initial learning rate to 0.001, and gradually decay it during the training process to ensure rapid convergence of the model while preventing overfitting; each round of update is based on the back propagation algorithm to obtain gradient information, and combined with L2 regularization to suppress overfitting;

[0014] S4. Simulation data preparation: In order to generate simulation data and annotations in the same format as the dataset, a virtual environment including roads, vehicles, pedestrians, cyclists and other objects is built in the simulation platform; the lidar sensor is configured, and the scanning angle, resolution and other parameters are set to be consistent with the target detection dataset; the virtual vehicle equipped with the lidar moves in the scene to collect the point cloud data transmitted by the lidar; the simulation platform automatically generates annotation information, which includes the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, and the category; the collected point cloud data is preprocessed, including adding appropriate noise and simulating missing data; the preprocessed point cloud data and annotations are converted to a format consistent with the target detection dataset, and the files are timestamp aligned and path normalized to ensure that they can be directly used by the target detection model;

[0015] S5. Model reasoning and result evaluation based on simulation data: In order to verify the model performance and optimize the data set to ensure that the detection effect meets expectations, the target detection model trained and updated in step S3 is deployed in a simulation test environment. The target detection model uses the simulation data output in step S4 as input and generates target detection results through model reasoning, including the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, the category and the detection score; the result evaluation adopts a series of key indicators, including average precision, false alarm rate, recall rate, and F1 score; whether the model performance meets the expected effect is judged based on whether the indicators are higher than the set threshold. If it meets the expected effect, proceed to the next step, otherwise return to step S1, and further enhance the target detection data set, including expanding the specific scene data that causes poor simulation effect;

[0016] S6, model deployment and real vehicle data preparation: In order to achieve model deployment and prepare compatible data, and ensure the accuracy of target detection in the real vehicle environment, the target detection model trained in step S5 and meeting the expected effect is deployed on an unmanned vehicle equipped with a laser radar. When the unmanned vehicle is in driving mode, the original point cloud data collected by the laser radar on the unmanned vehicle is converted in real time into point cloud data consistent with the format of the target detection dataset; the point cloud data is preprocessed, including adding noise, filling missing values, time alignment and coordinate conversion; through feature selection and format conversion, the data is standardized to the structure of the target detection dataset, and outliers are screened and removed; the processed point cloud data is stored in the format of the target detection dataset to ensure that it supports direct use of the target detection model;

[0017] S7, model reasoning and data preservation based on real vehicle data: In order to realize real-time reasoning, result display and preservation of real vehicle data and support subsequent model optimization, the processed point cloud data obtained in step S6 is input into the target detection model deployed on the unmanned vehicle frame by frame in real time, and the target detection results are generated through model reasoning, including the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, the category and the detection score; the three-dimensional bounding box is projected into the two-dimensional image collected by the camera at the same time, and visualized on the screen; the point cloud data, the target detection results and the user's feedback information on the target detection results (including target missed detection and false detection) are saved;

[0018] S8, data upload and labeling: manually set the data upload cycle, and upload data to the cloud once every cycle; when the unmanned vehicle is not in driving mode, if the data upload cycle has not been reached, return to execute steps S6 and S7; if the data upload cycle has been reached, upload the point cloud data, target detection results and user feedback information on the results saved in step S7 to the cloud, and filter and process them according to data quality, data relevance and data diversity; use automated labeling tools, set the detection confidence threshold, and combine the target detection results above the detection confidence threshold to label all targets with bounding boxes and categories to generate incremental data sets;

[0019] S9, incremental learning and parameter update: In order to achieve continuous optimization of the target detection model, the incremental data set generated in step S8 is used to fine-tune the parameters of the target detection model, and regularization constraints are used to prevent new knowledge from overwriting old knowledge; the target detection model with updated parameters is pushed back to the unmanned vehicle; return to step S6, and repeat S6 to S9 to continuously update and optimize the target detection model.

[0020] Furthermore, the preprocessing module described in step S2 is used for preprocessing point cloud data. Define the horizontal area of ​​interest as [-60,60]×[0,80] meters, and select a fixed height range to filter out ground point clouds and high-altitude point clouds based on the working range of the lidar sensor. Perform denoising and filtering on the point cloud data. Use a resolution of 0.1 meters to project the denoised and filtered point cloud from three-dimensional space to the bird's-eye view plane to generate a two-dimensional image expression of the same size height map, intensity map, density map, and curvature map, where the height map is represented as:

[0021]

[0022] Where z is the height of the laser point, P i,j is the set of all laser points projected on the pixel grid where the coordinates (i, j) of the bird's-eye view are located, H i,j Represents the maximum height of all laser points in the set. Through logarithmic transformation, the influence of extreme values ​​on the target detection model is reduced, thereby improving the stability and generalization ability of the target detection model. The intensity map is represented as:

[0023]

[0024] Where r is the intensity of the laser spot, N i,j is the number of laser points projected into the pixel grid at the coordinate (i, j) of the bird’s-eye view, I i,j Represents the average intensity of all laser points in the grid. The density map is represented by:

[0025]

[0026] The curvature graph is represented as:

[0027]

[0028] Where X i,j is the three-dimensional coordinate of the laser point with the largest intensity projected in the pixel grid where the coordinate (i, j) of the bird's-eye view is located, represents the set of neighborhood sample points centered on the laser point, where the number of sample points is n, X k are the three-dimensional coordinates of the neighborhood laser points; the height map, intensity map, density map, and curvature map are stacked in the depth dimension to generate a three-dimensional bird's-eye view of four channels.

[0029] Furthermore, the backbone network module described in step S2 adopts a dual-branch deep convolutional neural network structure, including an image processing branch and a point cloud processing branch. Among them, the image processing branch uses the three-dimensional bird's-eye view generated by the preprocessing module as input, and uses ResNet-50 as a feature encoder. Since point cloud data is sparse, the projected image on the two-dimensional plane is also sparse. It is not efficient to use traditional convolutional neural networks to process sparse images. Therefore, a sparse convolution module is used to replace the first stage of ResNet-50. Taking into account the irregular distribution characteristics of point clouds and the more obvious defects of point clouds on distant objects or small targets, a two-dimensional distance-aware focal sparse convolution module is designed to replace the first stage of ResNet-50 to solve the above problems by focusing on the importance differences of point cloud data. The second to fifth stages of ResNet-50 generate 4 multi-scale feature maps with resolutions reduced to 1 / 2 of the previous stage respectively. The point cloud processing branch uses the backbone network of PointNet++ as the feature extraction network and takes the denoised and filtered point cloud data as input; the point cloud processing branch contains l SA layers, each of which extracts local features through farthest point sampling and neighborhood aggregation operations. and global features Aggregate each layer of global features again to get the final global features The global feature is combined with the feature extracted by the image processing branch By adding each element together, we can get the fusion feature F. fu At the same time, feature propagation (l FP layers) is used to restore the spatial resolution of global features layer by layer until the original point cloud resolution. In the whole process, l takes the value of 4.

[0030] Furthermore, the neck network module described in step S2 adopts a feature pyramid structure based on the feature maps generated in the second to fourth stages of ResNet-50 Fusion feature F fuAnd the feature maps of each resolution extracted by each FP layer of PointNet++ Perform feature fusion and enhancement to improve the model's detection performance for multi-scale targets. and F fu Through the top-down network of the feature pyramid network, the high-level feature map is upsampled layer by layer by interpolation, and fused with the current layer feature map by element-by-element addition to generate a pyramid feature. Multi-scale pyramid features Feature maps of each resolution extracted by the FP layer The corresponding feature fusion is performed element by element to obtain the final features of each level of the pyramid The value of l is 4 in the whole process.

[0031] Furthermore, the feature selection module described in step S2, in order to learn the importance of different scales, combines the multi-scale feature map generated by the neck network module Through interpolation, the size of the largest feature map is adjusted to obtain feature maps with the same size. The soft weights are generated through the softmax operation, and the final output features are obtained through linear weighted summation. The specific formula is:

[0032]

[0033] Among them, l represents the number of layers, which is 4; thus, the final output features that integrate the advantages of each scale and are optimized are obtained.

[0034] Furthermore, the detection head module described in step S2 uses two sub-heads to jointly complete the detection task: a center point heat map head and a regression detection head. The center point heat map head predicts the heat map and predicts the probability that each pixel position in the bird's-eye view is the center point of a certain category of target; the regression detection head predicts the center point offset, size, and angular information of the target object.

[0035] The total loss function consists of the heat map loss and the bounding box regression loss, expressed as:

[0036] L total =α·L heatmap +β·L bbox

[0037] Where L heatmap represents the heat map loss, comparing the predicted heat map with the real heat map, using the Gaussian focal loss function; L bbox represents the bounding box regression loss, which compares the predicted and actual target object’s center point offset, size, and orientation angle, and uses a smooth L1 loss function; α and β represent the weights of the heat map loss and bounding box regression loss, respectively, and are both set to 1.

[0038] Furthermore, the two-dimensional distance-aware focal sparse convolution module used in the backbone network module consists of three parts:

[0039] (1) Predicting feature importance: Convert the 3D bird's-eye view image into a sparse image consisting of sparse tensors, extract the sparse features of each non-zero pixel of the sparse image through a submanifold sparse convolutional layer, and use the sigmoid function to predict the importance score of the sparse features, with the score between 0 and 1:

[0040] I p =σ((W*X) p )

[0041] Where σ(·) represents the sigmoid function, (W*X) p Represents the element value of the feature map obtained by the convolution operation at feature p;

[0042] (2) Filter the predicted importance map: Filter the top k input features with higher importance using the Top-k method:

[0043]

[0044] Among them, Top k (·) indicates the Top-k method, P in is the input feature space, k accounts for 50% of the total input features; considering the impact of the distance between the target and the sensor on the point cloud density, the importance score of the prediction is weighted based on the distance, represents the predicted importance score after distance weighting, which is calculated as follows:

[0045]

[0046] The weight w d (p) The Euclidean distance between the feature point and the vehicle sensor is used for calculation:

[0047] w d (p) = 1-exp(-λ||X p -X o ‖)

[0048] Where X o Represents the coordinates of the laser radar, X p represents the coordinates of feature point p, λ is a scaling factor that controls the sensitivity of normalization. A larger λ will make the normalized result approach 1 faster, and λ is 0.1;

[0049] (3) Dynamic output shape generation: For the features with high importance selected in (2), the surrounding extended features are screened through the Top-k algorithm to dynamically obtain extended features of different shapes, where k accounts for 50% of the total extended features; the output positions of other unimportant input features are the same as the input, essentially performing regular submanifold sparse convolution. In this way, through distance perception and dynamic output shape generation mechanism, more effective and adaptive convolution processing can be performed on important features while maintaining high efficiency.

[0050] The beneficial effects of the present invention are as follows: the present invention proposes a real-time three-dimensional target detection method for laser radar point clouds suitable for unmanned vehicles. Among them, the proposed target detection model mines richer spatial information by converting point cloud data into a variety of bird's-eye views; by adopting a two-dimensional distance-aware focal sparse convolution layer, dual-branch feature extraction and fusion, feature pyramid network structure and feature selection module, the detection accuracy of distant targets and smaller targets can be improved. The present invention effectively improves the robustness, accuracy and adaptability of the target detection model by constructing an efficient neural network architecture and combining training, simulation testing and real data incremental learning strategies, providing powerful perception support for unmanned driving systems.

[0051] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a real-time three-dimensional target detection method for laser radar point cloud applicable to unmanned vehicles according to the present invention;

[0053] Figure 2 It is an overall structural diagram of a three-dimensional target detection model of a real-time three-dimensional target detection method of a laser radar point cloud applicable to an unmanned vehicle according to the present invention;

[0054] Figure 3 This is a schematic diagram of a two-dimensional distance-aware focal sparse convolution module of a real-time three-dimensional target detection method of a lidar point cloud suitable for an unmanned vehicle described in the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below. It should be noted that this embodiment is based on the technical solution and provides a detailed implementation method and a specific operation process, but the protection scope of the present invention is not limited to this embodiment.

[0056] This embodiment provides a real-time three-dimensional target detection method for laser radar point cloud applicable to unmanned vehicles, such as Figure 1 As shown, the specific steps include:

[0057] S1. Preparation of target detection data set: When the target detection model has not completed training, the autonomous driving target detection data set is used as the training, verification and test set of the target detection model. The target categories include cars, pedestrians, cyclists and other categories. Then, the training set is data enhanced. Specifically, first, for each frame in the training set, a certain number of point cloud data of various target objects are randomly selected from other frames in the training set, and they are spliced ​​into the current frame. Preferably, in this embodiment, point cloud data of 15 cars, 20 pedestrians and 20 cyclists are added to each frame respectively. Secondly, the point cloud of the target object is randomly flipped and scaled along the X-axis and Y-axis directions in the horizontal plane.

[0058] S2. Model training: Train the end-to-end target detection model. The overall structure is as follows Figure 2 As shown. The target detection model includes the following modules:

[0059] (1) Preprocessing module: used to preprocess the point cloud data in the target detection dataset.

[0060] Specifically, first, define the horizontal area of ​​interest as [-60, 60] × [0, 80] meters, and select a fixed height range to filter out ground point clouds and high-altitude point clouds according to the working range of the lidar sensor. Preferably, this embodiment performs filtering according to the height range of [-2.5, 1] ​​meters.

[0061] Secondly, the point cloud data is denoised and filtered.

[0062] Then, the denoised and filtered point cloud is projected from the three-dimensional space to the bird's-eye view plane with a resolution of 0.1 meters to generate a two-dimensional image expression of the height map, intensity map, density map, and curvature map with a size of 1200×800. The height map is represented as:

[0063]

[0064] Where z is the height of the laser point, P i,j is the set of all laser points projected on the pixel grid where the coordinates (i, j) of the bird's-eye view are located, H i,j represents the maximum height of all laser points in the set. The intensity map is represented by:

[0065]

[0066] Where r is the intensity of the laser spot, N i,j is the number of laser points projected into the pixel grid at the coordinate (i, j) of the bird’s-eye view, I i,j Represents the average intensity of all laser points in the grid. The density map is represented by:

[0067]

[0068] The curvature graph is represented as:

[0069]

[0070] Among them, X i,j is the three-dimensional coordinate of the laser point with the largest intensity projected in the pixel grid where the coordinate (i, j) of the bird's-eye view is located, represents the set of neighborhood sample points centered on the laser point, where the number of sample points is n, X k is the three-dimensional coordinate of the neighborhood laser point. Preferably, in this embodiment, the number of sample points selected is 10.

[0071] Finally, the height map, intensity map, density map, and curvature map are stacked in the depth dimension to generate a 4-channel 3D bird’s-eye view image with a size of 1200×800×4.

[0072] (2) Backbone network module: connected to the preprocessing module and used to extract features from the preprocessed data.

[0073] Specifically, the backbone network module adopts a dual-branch deep convolutional neural network structure, including an image processing branch and a point cloud processing branch.

[0074] The image processing branch takes the 3D bird's-eye view image generated by the preprocessing module as input and uses ResNet-50 as the feature encoder.

[0075] The first stage of ResNet-50 is replaced by a 2D distance-aware focal sparse convolution module, which is Figure 3 Specifically, the two-dimensional distance-aware focal sparse convolution module consists of three parts:

[0076] a. Predict the importance of features: convert the 3D bird's-eye view image into a sparse image composed of sparse tensors, extract the sparse features of each non-zero pixel of the sparse image through a submanifold sparse convolution layer; use the sigmoid function to predict the importance score of the sparse features, with the score between 0 and 1:

[0077] I p =σ((W*X) p )

[0078] Where σ(·) represents the sigmoid function, (W*X) p Represents the element value of the feature map obtained by the convolution operation at feature p;

[0079] b. Filter the predicted importance map: Filter the top k input features with higher importance through the Top-k method:

[0080]

[0081] Among them, Top k (·) indicates the Top-k method, P in is the input feature space, k accounts for 50% of the total input features; represents the predicted importance score after distance weighting, which is calculated as follows:

[0082]

[0083] The weight w d (p) The Euclidean distance between the feature point and the vehicle sensor is used for calculation:

[0084] w d (p) = 1-exp(-λ||X p -X o ‖)

[0085] Where X o Represents the coordinates of the laser radar, X p represents the coordinates of feature point p, λ is a scaling factor that controls the normalization sensitivity, preferably, λ is 0.1;

[0086] c. Dynamic output shape generation: For the filtered features with higher importance, the surrounding extended features are screened through the Top-k algorithm to dynamically obtain extended features of different shapes, where k accounts for 50% of the total extended features; the output positions of other unimportant input features are the same as the input, essentially performing regular submanifold sparse convolution.

[0087] The second to fifth stages of ResNet-50 generate four multi-scale feature maps with resolutions reduced to 1 / 2 of the previous stage. In this embodiment, the second to fifth stages of ResNet-50 generate feature maps of 600×400, 300×200, 150×100, and 75×50, respectively.

[0088] The point cloud processing branch uses the backbone network of PointNet++ as the feature extraction network to directly process point cloud data. The network contains l SA layers, each of which extracts local features through farthest point sampling and neighborhood aggregation operations. and global features Then the global features of each layer are aggregated again to obtain the final global features The global feature is then combined with the features extracted by the image processing branch By adding each element together, we can get the fusion feature F. fu, for use by the subsequent neck network module. At the same time, feature propagation (l FP layers) is used to restore the spatial resolution of the global features layer by layer until the original point cloud resolution.

[0089] (3) Neck network module: connected to the backbone network module, used to fuse and enhance multi-scale feature maps.

[0090] Specifically, the neck network module adopts a feature pyramid structure based on the feature maps generated in the second to fourth stages of ResNet-50. Fusion feature F fu And the feature maps of each resolution extracted by each FP layer of PointNet++ Perform feature fusion and enhancement to improve the model's detection performance for multi-scale targets. and F fu The high-level feature maps are sampled layer by layer through the top-down network of the feature pyramid network, and fused with the current layer feature map to generate pyramid features. Specifically, at each level, the resolution of the high-level feature map is first adjusted by interpolation to match the resolution of the current layer, and then it is added element by element to the original feature map of the current layer to complete the fusion of multi-scale features and form a multi-scale pyramid feature. Feature maps of each resolution extracted by the FP layer in the point cloud processing branch The corresponding feature fusion is performed element by element to obtain the final features of each level of the pyramid The value of l is 4 in the whole process.

[0091] (4) Feature selection module: connected to the neck network module, it selects the feature map of the corresponding scale according to the size of the target.

[0092] Specifically, for the multi-scale feature map generated by the neck network module First, each scale map is adjusted to the size of the largest feature map by interpolation to obtain a feature map of consistent size. Then, a softmax operation is used to generate soft weights to represent the importance of each scale. Finally, the final output feature is obtained through linear weighted summation. The specific formula is:

[0093]

[0094] Among them, l represents the number of layers, which is 4; thus, the final output features that integrate the advantages of each scale and are optimized are obtained.

[0095] (5) Detection head module: connected to the feature selection module, predicts the category and bounding box of the target based on the feature map after feature selection.

[0096] Specifically, the detection head uses two sub-heads to complete the detection task: the center point heat map head and the regression detection head. The center point heat map head predicts the heat map and the probability that each pixel position in the bird's-eye view is the center point of a certain category of target; the regression detection head is used to predict other attribute information of the target object, including the center point offset, size, and direction angle of the target object.

[0097] The total loss function consists of the heat map loss and the bounding box regression loss, expressed as:

[0098] L total =α·L heatmap +β·L bbox

[0099] Where L heatmap represents the heat map loss, comparing the predicted heat map with the real heat map, using the Gaussian focal loss function; L bbox represents the bounding box regression loss, which compares the predicted and actual target object’s center point offset, size, and orientation angle, and adopts a smooth L1 loss function; α and β represent the weights of the heat map loss and bounding box regression loss, respectively, and both α and β are set to 1.

[0100] S3, parameter update: adjust the weight and bias parameters of the model by calculating the gradient of the loss function. The Adam optimization algorithm is used, and the initial learning rate is set to 0.001, which is gradually decayed during the training process. Each round of update is based on the back propagation algorithm to obtain gradient information, and combined with L2 regularization to suppress overfitting.

[0101] S4, simulation data preparation: used to generate simulation data and annotations that are consistent with the format of the target detection data set. First, build a virtual environment containing roads, vehicles, pedestrians, cyclists and other objects in the simulation platform, configure the laser radar sensor, and set the scanning angle, resolution, etc. that are consistent with the target detection data set. Secondly, move the virtual vehicle equipped with the laser radar in the scene to collect the point cloud data transmitted by the laser radar. Then, the simulation platform automatically generates the annotation information corresponding to the target object, including the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, the category, etc. In addition, the collected point cloud data is preprocessed, and appropriate noise and simulated missing data are added to improve authenticity. Finally, the preprocessed point cloud data and annotations are converted to a format consistent with the target detection data set, and the files are timestamp aligned and path normalized to facilitate direct use of the target detection model. Preferably, this embodiment uses Carla as a simulation platform.

[0102] S5. Model reasoning and result evaluation based on simulation data:

[0103] First, model inference is performed. The target detection model trained and updated in step S3 is deployed in a simulation test environment. The simulation data output in step S4 is used as input. The target detection results are generated through model inference, including the coordinates of the center point of the 3D bounding box of the target object, the size of the 3D bounding box, the heading angle, the category and the detection score.

[0104] Then the results are evaluated using a series of key indicators, including average precision, false alarm rate, recall rate, and F1 score. Whether the model performance meets the expected results is determined based on whether the indicators are higher than the set thresholds. If so, proceed to the next step; otherwise, return to step S1 and further enhance the target detection data set, including expanding specific scenario data (such as complex traffic conditions and extreme weather conditions) that result in poor simulation results.

[0105] S6, model deployment and real vehicle data preparation: The target detection model trained in step S5 and meeting the expected results is deployed on an unmanned vehicle equipped with a laser radar. When the unmanned vehicle is in driving mode, the original point cloud data collected by the laser radar on the unmanned vehicle is converted in real time into point cloud data consistent with the format of the target detection dataset. Then, the point cloud data is preprocessed, including adding noise, filling missing values, time alignment, and coordinate conversion. Next, through feature selection and format conversion, the data is standardized to the structure of the target detection dataset, and outliers are screened and removed. Finally, the processed point cloud data is stored in the format of the target detection dataset, supporting the direct use of the target detection model and ensuring seamless compatibility between the real vehicle data and the target detection dataset.

[0106] S7, model reasoning and data preservation based on real vehicle data: First, the processed point cloud data obtained in step S6 is input to the target detection model deployed on the unmanned vehicle frame by frame in real time, and the target detection results are generated through model reasoning, including the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, the category and the detection score. Then, the three-dimensional bounding box is projected into the two-dimensional image collected by the camera at the same time, and visualized on the screen. Finally, the point cloud data, target detection results and user feedback on the target detection results (including target missed detection and false detection) are saved.

[0107] S8, data upload and labeling: manually set the data upload cycle, and upload data to the cloud once per cycle. Preferably, the present embodiment sets the data to be uploaded once a week. When the unmanned vehicle is not in driving mode, if the data upload cycle has not been reached, return to execute steps S6 and S7; if the data upload cycle has been reached, the point cloud data, target detection results and user feedback on the results saved in step S7 are uploaded to the cloud, and screened and processed according to data quality, data relevance and data diversity. Then, use an automated labeling tool to set a detection confidence threshold, and combine the target detection results above the detection confidence threshold to perform bounding box and category labeling on all targets to form an incremental data set. Preferably, the present embodiment uses Xtreme1 as an automated labeling tool.

[0108] S9, incremental learning and parameter update: Use the incremental data set generated in step S8 to fine-tune the parameters of the target detection model, and use regularization constraints to prevent new knowledge from overwriting old knowledge. Then, push the target detection model with updated parameters back to the unmanned vehicle; return to step S6, and repeat S6 to S9 to continuously update and optimize the target detection model.

[0109] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A real-time three-dimensional target detection method for laser radar point cloud suitable for unmanned vehicles, characterized in that: The steps include: S1. Preparation of target detection dataset: When the target detection model has not been trained, the autonomous driving target detection dataset is used as the training, verification and test set of the target detection model. The target categories include cars, pedestrians, cyclists and other categories. Perform data augmentation on the training set, including splicing point cloud data of various target objects randomly selected from other frames in the training set into the current frame, and randomly flipping and scaling the point cloud of the target object in the horizontal plane; S2. Model training: Train an end-to-end object detection model, which includes the following modules: (1) Preprocessing module: preprocess the point cloud data in the target detection dataset; (2) Backbone network module: connected to the preprocessing module to extract features from the preprocessed data; (3) Neck network module: connected to the backbone network module to fuse and enhance multi-scale feature maps; (4) Feature selection module: connected to the neck network module, selects the feature map of the corresponding scale according to the size of the target; (5) Detection head module: connected to the feature selection module, predicting the category and bounding box of the target based on the feature map after feature selection; S3, parameter update: adjust the weight and bias parameters of the model by calculating the gradient of the loss function; use the Adam optimization algorithm, set the initial learning rate to 0.001, and gradually decay it during the training process; each round of update is based on the back propagation algorithm to obtain gradient information, and combined with L2 regularization to suppress overfitting; S4. Simulation data preparation: Build a virtual environment including roads, vehicles, pedestrians, cyclists and other objects in the simulation platform; configure the lidar sensor and set the scanning angle and resolution consistent with the target detection data set; move the virtual vehicle equipped with lidar in the scene to collect point cloud data transmitted by the lidar; the simulation platform automatically generates annotation information, which includes the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, and the category; pre-process the collected point cloud data, including adding appropriate noise and simulating missing data; convert the pre-processed point cloud data and annotations into a format consistent with the target detection data set, and align the timestamps and normalize the paths of the files to ensure that they can be directly used by the target detection model; S5, model reasoning and result evaluation based on simulation data: deploy the target detection model after training and parameter update in step S3 in a simulation test environment, use the simulation data generated in step S4 as input, and generate target detection results through model reasoning, including the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, the category and the detection score; the result evaluation adopts a series of key indicators, including average precision, false alarm rate, recall rate, and F1 score; judge whether the model performance meets the expected effect according to whether the indicators are higher than the set thresholds, if it meets the expected effect, proceed to the next step, otherwise return to step S1, and further enhance the target detection data set, including expanding the specific scene data that causes poor simulation effect; S6, model deployment and real vehicle data preparation: deploy the target detection model trained in step S5 and meeting the expected effect on an unmanned vehicle equipped with a laser radar. When the unmanned vehicle is in driving mode, convert the point cloud data collected by the laser radar on the unmanned vehicle into point cloud data in the same format as the target detection dataset in real time; preprocess the point cloud data, including adding noise, filling missing values, time alignment and coordinate conversion; standardize the data into the structure of the target detection dataset through feature selection and format conversion, and screen and remove outliers; store the processed point cloud data in the format of the target detection dataset to ensure that the target detection model can be directly used; S7, model reasoning and data preservation based on real vehicle data: input the processed point cloud data obtained in step S6 to the target detection model deployed on the unmanned vehicle frame by frame in real time, generate target detection results through model reasoning, including the coordinates of the center point of the three-dimensional bounding box of the target object, the size of the three-dimensional bounding box, the heading angle, the category and the detection score; project the three-dimensional bounding box into the two-dimensional image collected by the camera at the same time, and display it visually on the screen; save the point cloud data, the target detection results and the user's feedback information on the target detection results (including target missed detection and false detection); S8, data upload and labeling: manually set the data upload cycle, and upload data to the cloud once every cycle; when the unmanned vehicle is not in driving mode, if the data upload cycle has not been reached, return to execute steps S6 and S7; if the data upload cycle has been reached, upload the point cloud data, target detection results and user feedback information on the results saved in step S7 to the cloud, and filter and process them according to data quality, data relevance and data diversity; use automated labeling tools, set the detection confidence threshold, and combine the target detection results above the detection confidence threshold to label all targets with bounding boxes and categories to generate incremental data sets; S9, incremental learning and parameter update: Use the incremental data set generated in step S8 to fine-tune the parameters of the target detection model, and use regularization constraints to prevent new knowledge from overwriting old knowledge; push the target detection model with updated parameters back to the unmanned vehicle; return to step S6, and repeat S6 to S9 to continuously update and optimize the target detection model.

2. The real-time three-dimensional target detection method of laser radar point cloud applicable to unmanned vehicles according to claim 1 is characterized in that: The preprocessing module described in step S2 defines the horizontal area of ​​interest as [-60,60]×[0,80] meters, and selects a fixed height range according to the working range of the laser radar sensor to filter out the ground point cloud and the high-altitude point cloud; denoises and filters the point cloud data; projects the denoised and filtered point cloud from the three-dimensional space to the bird's-eye view plane with a resolution of 0.1 meters, and generates a two-dimensional image expression of the height map, intensity map, density map, and curvature map of the same size; wherein the height map is represented as: Where z is the height of the laser point, P i,j is the set of all laser points projected on the pixel grid where the coordinates (i, j) of the bird's-eye view are located, H i,j represents the maximum height of all laser points in the set; the intensity map is expressed as: Where r is the intensity of the laser spot, N i,j is the number of laser points projected into the pixel grid at the coordinate (i, j) of the bird’s-eye view, I i,j Represents the average intensity of all laser points in the grid; the density map is expressed as: The curvature graph is represented as: Where X i,j is the three-dimensional coordinate of the laser point with the largest intensity projected in the pixel grid where the coordinate (i, j) of the bird's-eye view is located, represents the set of neighborhood sample points centered on the laser point, where the number of sample points is n, X k are the three-dimensional coordinates of the neighborhood laser points; the height map, intensity map, density map, and curvature map are stacked in the depth dimension to generate a three-dimensional bird's-eye view of four channels.

3. The real-time three-dimensional target detection method of laser radar point cloud applicable to unmanned vehicles according to claim 1 is characterized in that: The backbone network module described in step S2 adopts a dual-branch deep convolutional neural network structure, including an image processing branch and a point cloud processing branch; wherein the image processing branch uses the three-dimensional bird's-eye view generated by the preprocessing module as input and uses ResNet-50 as a feature encoder; a two-dimensional distance-aware focal sparse convolution module is used to replace the first stage of ResNet-50; the second to fifth stages of ResNet-50 respectively generate 4 multi-scale feature maps whose resolutions are successively reduced to 1 / 2 of the previous stage The point cloud processing branch uses the backbone network of PointNet++ as the feature extraction network and takes the denoised and filtered point cloud data as input; the point cloud processing branch contains l SA layers, each of which extracts local features through farthest point sampling and neighborhood aggregation operations. and global features Aggregate each layer of global features again to get the final global features F SA Features extracted by the image processing branch By adding each element together, we can get the fusion feature F. fu ; At the same time, feature propagation (l FP layers) is used to restore the spatial resolution of global features layer by layer until the original point cloud resolution; l represents the number of layers, and l takes the value of 4 in the whole process.

4. The real-time three-dimensional target detection method of laser radar point cloud applicable to unmanned vehicles according to claim 1, characterized in that: The neck network module described in step S2 adopts a feature pyramid structure based on the feature maps generated in the second to fourth stages of ResNet-50 described in claim 2. Fusion feature F fu And the feature maps of each resolution extracted by each FP layer of PointNet++ Perform feature fusion and enhancement; among them, the feature map and F fu Through the top-down network of the feature pyramid network, the high-level feature map is upsampled layer by layer by interpolation, and fused with the current layer feature map by element-by-element addition to generate a pyramid feature. Multi-scale pyramid features Feature maps of each resolution extracted by the FP layer The corresponding feature fusion is performed element by element to obtain the final features of each level of the pyramid l represents the number of layers, and the value of l is 4 in the whole process.

5. The real-time three-dimensional target detection method of laser radar point cloud applicable to unmanned vehicles according to claim 1, characterized in that: The feature selection module described in step S2 selects the multi-scale feature map generated by the neck network module Through interpolation, the size of the largest feature map is adjusted to obtain feature maps with the same size. Generate soft weights through softmax operation; get the final output features through linear weighted summation; The specific formula is: Among them, l represents the number of layers, and its value is 4.

6. The real-time three-dimensional target detection method of laser radar point cloud applicable to unmanned vehicles according to claim 1, characterized in that: The detection head module described in step S2 includes two sub-heads: a center point heat map head and a regression detection head; wherein the center point heat map head predicts the heat map and predicts the probability that each pixel position in the bird's-eye view is the center point of a certain category of target; the regression detection head predicts the center point offset, size, and direction angle of the target object; The total loss function consists of the heat map loss and the bounding box regression loss, expressed as: L total =α·L heatmap +β·L bbox Where L heatmap represents the heat map loss, comparing the predicted heat map with the real heat map, using the Gaussian focal loss function; L bbox represents the bounding box regression loss, which compares the predicted and actual target object’s center point offset, size, and orientation angle, and uses a smooth L1 loss function; α and β represent the weights of the heat map loss and bounding box regression loss, respectively, and are both set to 1.

7. The backbone network module according to claim 3, characterized in that: The two-dimensional distance-aware focal sparse convolution module consists of three parts: (1) Predicting feature importance: Convert the 3D bird's-eye view image into a sparse image consisting of sparse tensors, extract the sparse features of each non-zero pixel of the sparse image through a submanifold sparse convolutional layer, and use the sigmoid function to predict the importance score of the sparse features, with the score between 0 and 1: I p =σ((W*X) p ) Where σ(·) represents the sigmoid function, (W*X) p Represents the element value of the feature map obtained by the convolution operation at feature p; (2) Filter the predicted importance map: Filter the top k input features with higher importance using the Top-k method: Among them, Top k (·) indicates the Top-k method, P in is the input feature space, k accounts for 50% of the total input features; represents the predicted importance score after distance weighting, which is calculated as follows: The weight w d (p) The Euclidean distance between the feature point and the vehicle sensor is used for calculation: w d (p)=1-exp(-λ‖X p -X o ‖) Where X o Represents the coordinates of the laser radar, X p represents the coordinates of feature point p, λ is a scaling factor that controls the normalization sensitivity, and λ is 0.1; (3) Dynamic output shape generation: For the features with high importance selected in (2), the surrounding extended features are screened through the Top-k algorithm to dynamically obtain extended features of different shapes, where k accounts for 50% of the total extended features; the output positions of other unimportant input features are the same as the input, essentially performing regular submanifold sparse convolution.

Images